Because of a 56-second TikTok video by a teenager, thousands of scientific studies had to discard weeks of data.
The video for July 23rd is brief and straightforward. It begins with Sarah Frank, a recent Florida high school graduate and self-described “teen author,” sitting in her bedroom and smiling at the camera.
“Welcome to side hustles I recommend trying — part one,” she says in the video, directing viewers to Prolific.co. “Basically, it’s a bunch of surveys for varying amounts of money and time.”
That video received 4.1 million views in the month following its release, bringing tens of thousands of new users to the Prolific platform. Prolific, a tool for behavioral researchers, lacked screening tools to ensure that it delivered representative population samples to each study. Scientists used to getting a diverse range of subjects for their Prolific studies were suddenly inundated with responses from young women around Frank’s age.
That demographic shift was a major problem for researchers who rely on representative samples of the US population, with no obvious cause and no immediately obvious solution.
Using the Internet for research
Though it is not widely known, Prolific is part of a small group of online tools that have changed the way corporations and scientists study how people think and act. The first and largest of these research platforms is Amazon’s Mechanical Turk, which launched in 2005 as a general-purpose platform for crowdsourcing work on repetitive tasks. Soon after its release, behavioral scientists recognized its potential value for their research, and it quickly revolutionized several fields of study.
“Before Mechanical Turk, all social science research had to take place in a laboratory. “You’d need to bring in college sophomores and put them through questionnaires and surveys and the like,” said Nicholas Hall, director of the Stanford School of Business’s Behavioral Lab.
“That is a massively time-consuming and labor-intensive undertaking. It is much easier to conduct research online. “You create a survey, put it online, and within a day, you have 1,000 responses,” Hall explained to The Verge. “That altered the course of social science.”
According to Hall, the Behavioral Lab at Stanford now primarily uses the newer, smaller Prolific platform for online studies. While many Mechanical Turk customers are large corporations conducting corporate research, Prolific’s product is geared toward scientists.
The smaller platform promises more transparency, more ethical treatment of survey participants, and higher-quality research subjects than alternative platforms such as Mechanical Turk.
In the United States, scientists conducting this type of research generally want a pool of subjects who speak English as a first language, aren’t too accustomed to taking psychological surveys, and make up a reasonably representative demographic sample of the American population.
Most people agreed that Prolific did a good job of providing high-quality subjects. The platform’s reputation was threatened by a sudden shift in demographics.
Researchers scrambled to figure out what was going on with their studies in the days and weeks following Frank’s video.
“We have noticed a huge jump in the number of participants on the platform in the US Pool, from 40k to 80k,” a member of the Stanford Behavioral Laboratory wrote on a Prolific forum. Which is great; however, many of our studies now have a gender skew, with perhaps 85 percent of participants being women. Furthermore, the average age has been around 21.”
Hannah Schechter, a psychologist at Wayne State University, appears to have been the first to solve the case.
“This may seem far-fetched,” she tweeted, referring to Frank’s video, “but given the timing, virality of the video, and the user’s follower demographics…”
Long-time Prolific survey takers complained on Reddit that Frank had made it difficult to find paid surveys on the overrun platform.
“Now it’s just another shithole site to waste hours and make pennies on,” wrote one user, who claimed to have previously earned $30 per week on the platform.
Frank, who “guessed” she’d made about $80 taking surveys on Prolific before her video, told The Verge she noticed a difference on the platform as well.
“There have been fewer studies available for me and everyone else,” she told The Verge. “I’ve received some really nasty comments accusing me of ruining the site and being selfish — despite the fact that I received no compensation for that video.”
She also expressed hope that Prolific would be able to implement a system to deal with its changing demographics.
“I also predict that many people who signed up after seeing my video will forget about it, and the surge will subside,” she added.
According to prolific co-founder and CTO Phelim Bradley, many of the new users appear to be dropping off.
“Prior to Tiktok, approximately 50% of responses on our platform came from women,” he wrote in an email. “The surge pushed this up to as high as 75% for a few days, but since then, the number has been trending down, and we’re now back to 60% of responses coming from women.”
According to Bradley, Frank’s TikTok disrupted approximately 4,600 studies, accounting for roughly one-third of the total number of studies active on the platform during the surge. According to him, the vast majority of those should be salvageable.
Prolific has refunded researchers whose studies were adversely affected by the increase in female survey takers and introduced a new suite of demographic screening tools. A month after Frank posted her video, the company announced these changes. The company has now reorganized, putting a team in charge of demographic balancing in order to recognize and respond to similar problems more quickly in the future.
“To be honest, we were caught off guard, and we had no idea how big the impact would be,” Bradley said.
The uptick isn’t all bad. According to Vlad Chituk, a Yale graduate student in psychology who was running several pilot studies on Prolific when the surge hit, refreshing the pool of survey takers is likely to have long-term benefits. When subjects complete a large number of psychological surveys, they learn the tricks that scientists use to collect data, which can affect how they respond to future survey questions. Higher-quality data is provided by subjects who are new to the study.
“Young women who enjoy TikTok are also people,” he said.
According to Frank, her side hustle video is now the most popular TikTok she’s ever posted.
“It never occurred to me that the video would go viral. “I only posted it for my friends and followers, not for the amount of attention it received,” she explained. “I believe it exploded because the site is genuinely cool, and people love easy ways to make money.”
For the time being, Frank has put most of her side hustles on hold as she begins her freshman year at Brown.