According to the Pew Research Centre, the number of bad respondents for a given online panel ranged between 4-7%. In most industries, this would be considered abysmal, but considering falling engagement, weak quality checks and the rapid technology adoption by respondents, this makes a lot of sense. However, this research only omitted the most obvious poor responses i.e. duplicate accounts, those outside the country of the research, gibberish or contradictory answers.
The reality is, when Grey Matter carried out a more intensive quality assessment, disregarding the obvious conflict of interest in panel companies investigating their own quality control, they found that a shocking 46% of respondents had to be omitted for low quality. This was if a respondent committed four or more issues, which identifies just how much panel suppliers are under-analysing their output for clients.
The purpose of this article is not to defame the online panel suppliers, nor is to take a superior higher ground as a new entrant. Quality control is incredibly difficult, it requires a huge investment in technology and a meticulous strategy for pursuing data that you can trust. This is not helped by the fact that panels have become commodised, with no incentive to deliver higher quality.
That being said, researchers are simply not doing enough to address these challenges, ignorance is certainly not bliss. Using over 40,000 interviews per month through a series of online panels, Harmon Research found that:
Every insights methodology has challenges and quality issues. Good researchers recognize this and do everything they can to reduce the problems and mitigate the challenges without throwing out the methodology. Yet, online panels have dropped the ball, and it’s time to accept the fact that there is a deep rooted problem in the industry.
Unfortunately, these ‘uncaught’ respondents have damaging knock-on effects for the end-client. They fundamentally distort key business decisions, misinform strategic choices, and can lead to the demise of many companies across the world. The only thing worse than waste is being misled, and there has never been a greater need for transparency in the online panel industry.
Do you want to rely on a brand awareness study showing familiarity that’s three times as high as it should be? Or a channel study showing a social media platform is 188% more popular than it really is? How about a consumer spending study that reveals spending to be 16X higher than in the real world. Unfortunately, this is the truth behind online panels around the world.
The solution is to make the best of a bad situation, and start implementing the following:
Ultimately, what you don’t know can most certainly hurt you. As an industry, it is essential that we tackle these problems head on. We must work together to demand more of both panel suppliers and research buyers to incentivise the time, money and effort necessary to solve these widespread issues.