What Is Crowdsourced Public Opinion?

For decades, researchers have gathered data on public opinion using expensive, time-intensive approaches such as in-person interviews, survey mailings, and phone calls. Crowdsourcing platforms like Amazon’s Mechanical Turk and ClickWorker dramatically change this dynamic, making the gathering of public opinion far cheaper and easier.

We call this approach “crowdsourced public opinion,” and several studies have shown that the technique provides high-quality research results at far lower costs than traditional methods.

In many cases, the results from crowdsourced public opinion methods are of higher quality than more traditional approaches. One University of Oxford study stated that MTurk resulted in better data than traditional resources for research.

Traditional methods to collect opinion data take a lot of time and money. A high-quality, national representative poll of 1,000 people typically costs at least $30,000. Focus groups of ten individuals run around $1,000 to -$1,200 per person in a major city.

What’s more, traditional approaches to surveys make it expensive to target subpopulations like groups of educators or medical professionals. The RAND Corporation, for example, has a panel of teachers and school leaders. But the cost is as much as $60,000 per survey.

Opinion crowdsourcing offers a low-cost alternative, and using crowdsourcing tools like Amazon Mechanical Turk, experts can gather rigorous, quality data in a far more cost-effective way.

​Among other things, the technologies allow for:

• National polls. National polls are the bread and butter of the industry, and they attempt to sample a cross-section of the public in order to predict results or gather public attitudes or opinions on particular topics. Online tools can now be used for such polls at much lower costs; our team conducted a survey around attitudes and perceptions of critical thinking, completed for the Reboot Foundation, which gained wide attention.
• Diary studies. Nielsen TV ratings are an example of a diary study, where participants record their behavior over a designated period of time, from days and weeks, to months or years. Again, online tools can make this innovative approach far less expensive, and our team worked with the Center for American Progress to utilize this type of approach creating a report with impressive results that garnered wide attention.
• Product reviews. Want to get feedback on a website? Get a review of a new tool or app? Crowdsourced opinions are a powerful way to gain these insights, and we, at the Learning Agency, have used online platforms to review everything from software to websites.
• Focus groups. Online platforms make it easy to run a focus group and results are 80 percent cheaper. Plus, the results come back within hours, not weeks or months.​

Yes. Last year alone, thousands of experiments or research studies were run on MTurk, and many researchers believe that the approach is often more effective than traditional approaches.

Within academia, tools like MTurk have dramatically changed research. In May 2018, the online magazine JSTOR Daily published an article with this bold headline: “Amazon’s Mechanical Turk has Reinvented Research.” The author, Alexander Samuel, went so far as to say that Mechanical Turk and other similar services have ushered in a “golden age in survey research.”

Several studies have shown that MTurk produces high-quality research results. A study led by Tara Behrend of George Washington University raised the specific question of whether or not survey data collected from an online pool would be as reliable as more traditional university panel survey pools.

In addition to identifying the fact that “crowdsourcing respondents were older, were more ethnically diverse, and had more work experience,” the “reliability of the data from the crowdsourcing sample was as good as or better than the corresponding university sample,” she concluded.

Another study found similar results. Lead by Michael Buhrmester from the University of Texas at Austin, the team found that MTurk data was found to be “at least as reliable as those obtained via traditional methods.” Additionally, the study concluded that MTurk could be used to achieve “high-quality data inexpensively and rapidly.”

Many large companies have been using these tools, and in a review by the research team at Yahoo, the authors compared MTurk to large market research studies, as well as another online survey platform. The conclusion was that online platforms, including MTurk, could be relied upon to achieve reliable results in just hours and at a fraction of the cost of more formal survey settings.​

Panels are groups of people who render their opinion or perspective on products, services, or research data. Experts have used panels for decades to get focused research from specific sub-populations.

At the Learning Agency, we have created or partnered to create different types of panels.

​Educator Panel. The Learning Agency has gathered more than 800 educators who are actively teaching in the field. These panel participants are involved in surveys, interviews, and focus groups. 

Most of these educators are teachers. Some are leaders and support staff. They are somewhat younger and whiter than the population as a whole.​

National Representative Panel. The Learning Agency has partnered with Matthew Sisco, a behavioral researcher at Columbia University, who has created a nationally representative panel of online participants.

This system allows highly representative estimates to be generated from online samples with two main features. First, the system maintains a pool of adult participants that mirror the population of the United States on key demographics including age, gender, and location. This allows any survey to be completed by a sample that closely represents the national population.

Second, the system employs state-of-the-art statistical corrections to the survey responses that account for any remaining misalignment. This correction is known as “Multi-level Regression and Poststratification” or MRP, and it enables accurate predictions of public opinion in fine-grained geographic areas such as states or counties. More simply, the system distributes and analyzes a survey in a way that a) generates estimates of opinions highly representative of national public opinion and b) estimates how opinions vary across different states and/or counties.​

Cost. A traditional survey using polls and a nationally representative sample typically costs around $30,000. Using a platform like MTurk, costs can be significantly reduced, and a 15 minute, 15 question survey of 1,500 participants costs around $5,000.

Speed. Crowdsourcing public opinion results can be collected within hours, days or weeks, in comparison to many weeks and sometimes months from more traditional means. Online platforms can very quickly be collected and tabulated, making evaluation much quicker and easier, as well.

Innovation. In a recent study published in the journal, Psychological Science, researchers examined the issue of collective narcissism, or where individuals “excessively (demonstrate) high regard for their own group.” The data was gathered using Amazon’s MTurk, with more than 2,800 Americans from all 50 states.

The study had an innovative design, asking Americans from each state what they thought their state’s contribution was to the overall history of the United States. The results received widespread press attention including in the Washington Post.

Anonymity. There may also be a “hidden” advantage to online research platforms like those found on MTurk, as people are more likely to give authentic answers because they’re online, versus a less anonymous setting.

In one study in the journal Clinical Psychological Science, researchers examined the use of MTurk to study clinical populations. Researchers were looking for the prevalence of severe psychiatric disorders within crowdsourcing populations, including Amazon’s MTurk.

Their findings suggested that crowdsourcing participants may be more comfortable revealing sensitive personal information than they would be in more traditional interviews and surveys. The anonymity of online forums can carry a distinct advantage here, and in this way, MTurk can offer a higher quality result.​

There are some concerns about MTurk as a research tool.

Fairness. Some, understandably, argue that many MTurk requesters don’t pay people enough. We strongly believe that people should be paid fairly, and we don’t publish any projects that pay less than $15/hr, the minimum wage in New York City, and higher than Washington D.C.’s $14/hr minimum wage. For our teacher and other panels, we typically pay closer to $20 an hour.

Focus. Another concern about online, crowdsourcing tools is that participants are not as focused as more traditional participants would be. In other words, an online survey can be completed while watching television, or talking with someone else.

However, researchers have tested this idea, and it seems that people do, in fact, pay attention. In one study in the Journal of Experimental Political Science, for example, few differences were found in the quality of results between laboratory administered questionnaires and those administered online. In other words, surveys taken at home, or in an uncontrolled setting, were just as reliable as more traditional settings.

Indeed, several studies have shown that MTurk respondents paid better attention than their counterparts; one study concluded that, of three different types of tasks, “MTurkers were more attentive to the instructions than were college students”  when it came to the appropriate completion of attention-check questions.

At the Learning Agency, we also include “attention checks,” these are questions or instructions designed to ensure that the person taking the survey is paying attention. They might consist of an instruction to write in, “I am paying attention,” on a particular survey answer, or answers that use reverse wording on two different, but similarly worded questions, so that a positive response for one question would mean a negative response for the other question.

Demographics. MTurk presents some issues regarding the demographics of participants. Americans on the platform are generally whiter and more highly educated than the general population. MTurk workers also have a higher rate of unemployment, retired or homemaker status, and more than 10 percent are college students, according to a University of Washington study.

Most differences in demographics can be factored in when final results are evaluated.

Mechanical Turk, or MTurk, is Amazon’s crowdsourcing marketplace. It’s named after an 18th-Century chess playing “machine” that defeated several human challengers. This “machine” hid, within its workings, a human chess-master who controlled the actions of the machine.

Amazon’s MTurk was launched in November of 2005, and the site soon became home to tens of thousands of “jobs” on the platform. By the spring of 2007, 100,000 workers had registered, from more than 100 countries. Just a few years later, there were half a million from almost 200 countries.

Most estimates indicate that 80% of “Turkers,” as MTurk workers are colloquially known, are located in the United States, and at any time there are 2,000-5,000 workers online at any given time, which is the estimated equivalent of 10,000-25,000 active full-time workers.

At the Learning Agency, we rely on MTurk because it is the largest crowdsourced platform and has proven itself to be an effective way to conduct experiments online, offering a researcher’s trifecta of a wide range of respondents, at a relatively low cost, and within a short duration of time. Most every type of traditional opinion collection method can be replicated, or at least modified using MTurk’s platform.