What does nationally representative actually mean?
A national representative (nat rep) survey is what I would call a heritage survey. It goes back to the origins of research, which started out in a field. A literal field.
Market research was born out of the ability to ask people questions in an online format, the answers to which seemed to reflect the trends that were happening in the real world. Therefore, it negates the need to carry out all research in the field, or rather, in person.
However, the accuracy and representativeness of the insights we get out of our research are only as good as the accuracy and representativeness of the population we survey.
For data to continue to be a key part of business strategy and future thinking, accurate, representative data is extremely important.
Having worked in market research for the past seven years, nat rep surveys have become brick and mortar. However, while I have read and heard this term used countless times, I find it has rarely been discussed beyond its name until more recently.
So, what does nat rep mean? Well, that depends on who you ask.
A nationally representative sample means recruiting a sample of the population made up of accurately proportioned sub populations carefully balanced by key firmographics, demographics and other characteristics that reflect the make-up of the population in the ‘real world’.
Essentially, you survey a miniature version of the ‘real world’, online.
While nat rep is a widely used term in the research industry, when asking our team of researchers what nat rep means to them, I received an array of different answers.
Most commonly, in practice, nat rep surveys will, at the very minimum, be representative of a population by age and gender. Perhaps for some more specific studies, this definition may be expanded to include religion or ethnicity. However, there are many more factors and characteristics that make up a population’s dynamic than just their age and gender. For instance, other attributes you might consider are religion, ethnicity, first language, social grade, disability, and mental health, among others.
In a nat rep survey where representation is key, is surveying based on age and gender alone, truly representative?
I believe the answer is ‘it depends’.
When researchers gave their perspective, alongside age and gender, religion and ethnicity came out top, followed by sexual orientation and social grade as attributes that were important to consider including in a nat rep survey.
‘Which of the following do you think should be included in a nat rep survey?’
Responses ranked in order:
- Age, Gender, Religion
- Sexual Orientation, Social Grade
- Physical Disability, First Language
- Mental Health
When should you deploy a nat rep survey, and what to keep in mind?
While all demographic subsets deserve representation in surveys, we need to tackle this one step at a time. So, for now, let’s focus on which population groups are crucial for your research to be considered nationally representative.
To start to navigate the answer to this question, there are first some potential barriers to consider:
- Overengineering – we want our data to be as representative as possible, and this requires carefully capturing, balancing and interlocking a variety of the population’s demographics and characteristics. The more you layer and interlock these, the smaller the sub-populations in your survey become, and the number of different sub-populations you have increases. This adds time and complexity to managing what some might have considered a ‘straightforward’ survey. And the degree to which this improves your research accuracy starts to diminish.
Therefore, there is a ‘sweet spot’ to find between to the number of population variables you decide to include without ‘overengineering’ the sample. Every survey has its margin of error, and its results should be analysed with that in mind.
- Limitations or guidance on permitted data on the market you are surveying – different markets may have different limitations and/or guidance on what type of population can be collected. For example, in France and Germany, data on ethnicity is not collected at all, particularly in France, where this is against the law, except under special circumstances.
- Sensitive information – some demographic data is deemed sensitive, such as religious belief or political opinion. Asking for certain types of information, such as asking about a person’s mental health, may feel invasive or intrusive. Much care needs to be taken in how this information is collected. It also means not all participants may be willing to give us this information.
- Market relevance and breakdown of this data – different markets may break down or view aspects of population data differently. For example, ethnicity and Social Grade (a standardised way of grouping people based on their social and financial situation) classifications can vary by market.
- Respondents’ biased answers – while governing bodies often define demographic groupings, we cannot control how a respondent interprets and answers our question. For example, the interpretation of whether or not a respondent is a skilled or unskilled worker can often be interpreted differently. By our definition, examples of unskilled workers may include roles in construction and agriculture, but those within that profession may personally define and see themselves as skilled worker.
- Translation of demographic terms and classifications across markets – care needs to be taken in the translation of the categories across defining questions to ensure they are appropriately localised (rather than direct or literal) so that all respondents have the best chance of interpreting the question in the same way.
- Changing social environment – as with the nature of ‘nat rep’, the questions and categorisations often used to determine this, e.g. social class, have remained the same since I started my career. Therefore, given the changing populations, original classifications may no longer be entirely applicable and should perhaps be reviewed and reflected upon and potentially adjusted or updated for the purposes of your survey.
Having acknowledged the above, answering these questions will help you determine the best approach for your nat rep survey:
- Time – how time-sensitive is your project?
The management of interlocking quotas and/or trying to reach harder, more niche audiences requires time and effort. Therefore, there is potentially a balance to be struck between the margin of error you are willing to accept and the time you have to do the research. However, if these audiences are required, do not exclude them because they present a challenge. Instead, consider a mixed methodologies approach, including Computer Assisted Telephone Interviewing (CATI), to top up populations less represented or able to take part online.
- Patterns or trends – are you already aware of any particular patterns or trends in the way different populations tend to answer?
For example, do different generations consistently differ in their views? If so, then age is certainly a demographic to manage and include.
- Purpose of research – what is the purpose of your research?
The purpose of your research may determine how deep you need to go into and include different population subsets. Research for media and content offers requires breadth, a broad set of opinions from a number of different populations or stakeholders, whereas research for more strategic purposes, such as product testing, may require a narrower, more specific type of ‘nat rep’ targeting.
Remember -> While a single term is used, there is not one type of nat rep.
So, regardless of how straightforward it sounds, it is a good idea to have this conversation upfront and gain clarity on the questions above to ensure everyone is on the same page when it comes to their definition of nat rep.
While the goal is a perfectly representative survey, we should get comfortable with the idea of it not always being perfect. This quote from Bicentennial Man (1999) describes it best, ‘Imperfection is the key. Imperfections make us individuals, that’s what makes us unique.’
Therefore, be aware that there are and always will be limitations to your nat rep and keep them in mind when analysing your results.