In data we trust – steps to ensure data quality

Why should we trust the data? Are these respondents real people? Why do the results not match our internal data?

As market researchers, we get asked these questions regularly, especially after running an online survey. Researchers dig into the data and justify the responses or reiterate their quality checking process. This is a reactive approach to dealing with the issue of data quality – not a proactive one – and the industry can and should do better. The aim of this article is to help you ask the right questions to your research suppliers and give you complete trust in any online survey you run.

At Sapio Research, we believe data quality can be remedied at three points:

• The questionnaire – If you want the right answer, you’ve got to ask the right question
• Panel providers – We’ve conducted our own research on the quality of different online panels, helping us to make the best decisions on which to use for the best results
•   Data cleaning – Ensuring that poor quality responses are replaced with good quality ones

Ask a silly question and you’ll get a silly answer

A poorly designed questionnaire results in substandard data, and this leads to results that are ineffective, invalid, uninterpretable, and point to incorrect conclusions. Here are our top tips for ensuring good quality data:

1. Keep your questionnaire below 30 questions or 10 minutes: respondents lose focus in long surveys, and their answers will not be as well thought out or comprehensive.
2. Survey the right people: The big-dog C-level in a multi-million pound company may be your sales target, but are they going to be able to answer technical questions about your organisations IT infrastructure?
3. Don’t ask the unanswerable: Put yourself in the shoes of the respondent and ask whether you would be able to answer the question.
4. Be specific in your questioning: Respondents need to have an exact understand of what a question is asking. Be as specific as possible and include definitions for terms used that may not be universally understood.
5. Get the questions in the right order: Part of what contributes to a pleasant questionnaire experience is the narrative flow of questions.

The Good, The Bad and the Poor Quality Respondents

There are a variety of panel providers that market researchers can access, but how do we know which ones are well managed, and prioritise quality respondents?

At Sapio, we wanted to test the quality of the panels we use. We ran a survey with two different audiences across four different providers, and over 50 different individual panels.

• 2,036 consumers – representative by age and gender of the UK population aged 18+.
• 469 IT decision makers (ITDMs) – employees working within an IT department with authority for IT decisions.

In order to measure quality for both audiences we ran a speeder check (where respondents answering too quickly were removed) and we also removed those that has straight-lined through grid questions. Specific to the consumer audience, a response contradiction check was also included as well as selection of unlikely scenarios. For the ITDM audience, other checks included a grid contradiction check and knowledge of IT specific metrics and acronyms.

When comparing the representativeness of the consumer audience with the UK population, we found results were closely matched both in vaccination figures as well as results of the 2019 general election. When looking at quality differences within the audience, we found:

1. Answers from younger audiences were slightly lower in quality

2. Only a slight difference was observed between panel providers

3. Similarly, responses through mobile were slightly lower quality than those through desktop.

From this research, we can learn how to better improve the quality of data in our surveys. Here are the key actions we’re taking:

1. Introduce stricter quality procedures for younger respondents
2. Only select online panel providers that provide consistently high quality respondent
3. Improve the usability of mobile surveys

Clean up on aisle “data”

Once responses to your survey have been collected, it’s important your research partner cleans the data. They should clean out….

• Bad quality open-end responses: When a respondent leaves an answer in a text box, it’s a great indicator of whether or not they are a good quality respondent. Keyboard bashers must be removed
• Responses that don’t make sense: If a respondent has contradicted themselves through multiple questions, they’re probably not answering legitimately.
• Speeders: Those who fill out a survey too quickly to have been paying real attention to the questions.
• Straight liners: Respondents who consistently go through grid questions and select the same response for each row

Taking action at questionnaire design, fieldwork, and analysis is paramount in ensuring data quality. However, it’s not genuine to declare that survey data (or any other type of data for that matter) is void of data quality issues once these procedures have been implemented. There is always room for improvement, and the industry must constantly strive for new ways of preventing poor quality data.

Why can you trust Sapio to deliver research projects with excellent quality data?

With Sapio Research, our three levels of quality assurance mean you always get quality insights that you can rely on to make informed decisions. Our quality checking process sees to it that:

1. The questionnaire design and checking process ensures quality of insight.
2. Panel suppliers are selected for quality and reliability.
3. Extensive quality checks are carried out post-fieldwork to verify quality on the data gathered.

At Sapio, we understand how critical quality is in market research, and we apply it to everything we do. We’re committed to getting you the quality market research data you need to help your business. Contact us to start your research project.

> For more resources on this topic, read our full report Quality in Market Research Panels and our blog Detecting Dodgy Research Data.