Using Statistics In Communications - A Best Practice Guide

Reputable organisations including the MRS, RSS, BPC and CIPR have published guidance to how to report on statistics. Here we summarise the key takeaways for you.

Firstly, why do we need a best practice guide to statistics?

Over the last few years, and particularly so since Brexit there’s been a rise in ‘fake news’, as well as using statistics and numbers to emphasise a point or a view, or make a convincing message. Unfortunately, the way that statistics and numbers are presented can be misleading, and as reputable organisations we want to ensure that what is being reported is not taken out of context.

Statistics are incredibly powerful

As the stated in the MRS/CIPR/RSS guide – “Good and accurate use of statistics can help to establish credibility and to increase influence. Poor use of statistics can lead to loss of trust and reduced authority.“

Think about an argument or research without statistics, compared to one with. Being able to say “80% of UK adults prefer Cadbury’s to Galaxy” has much more impact than “Cadbury’s is generally preferred to Galaxy”. Statistics have the ability to substantiate (or refute) claims, influence thinking and help understanding.

However, statistics are notoriously easy to manipulate

“Lies, damned lies and statistics” is a famous quote (origin unknown, thought to maybe be Mark Twain or Benjamin Disraeli). This phrase is used in reference to the ability for statistics to bolster weak arguments. There are agencies dedicated to fact checking.

In the UK we have Full fact – an independent fact checking agency. They spend their time checking out what people are publishing and saying, often with a focus on political speeches and news stories. Full Fact then publish their findings so that the public can easily find out whether the claims that have been made are fully substantiated, or if they have been taken out of context.

However, the agencies don’t cover everything so it is our responsibility, as researchers, and your responsibility as professionals (whether PR, marketing or journalism), to care about reporting on what the statistics are really saying.

What can you do about it?

Follow the below guidance and you’ll be in the best position, knowing you’ve done your due diligence and understanding what’s behind the stat.

How to make sure audiences know your claims are reliable

It’s important when using statistics that the audience is able to make an informed decision about how reliable they are. To do this we need to include certain pieces of information:

Who commissioned the research
Who conducted the research
Objectives of the research
Sample size (how many people were interviewed)
Audience being represented (e.g. UK adults, cat owners etc.)
How data was collected (e.g. online, by phone, face-to-face)
What data collection method was used (e.g. questionnaire, discussion guide etc.)
When data was collected (fieldwork period)
Sample demographics (e.g. age, region, gender etc.)
Whether data is weighted
Survey results, including confidence levels, margin of error or statistical reliability of the results
Best practice would be to include a link to the full details of the research or poll, and the full wording of each question

If you don’t know any of the above, just ask your research agency – they will be able to provide all of this for you, and often do alongside any deliverables.

Common pitfalls to avoid

As mentioned above, statistics can easily be misunderstood or misrepresented, and not necessarily intentionally. Below are common pitfalls that the MRS and BPC suggest you look out for:

Averages – make sure you know which average has been reported (mean, median or mode) and how these differ. Some are better suited for certain measurements than others.
Sample representation – be really careful over who has been interviewed. If the sample is nationally representative then the make up of the sample should match the population split on certain demographics (such as age and gender). Don’t fall into the trap of thinking that a large group of people (e.g. parents) represent the whole population.
Sample size – if you have a sample of 1,000 UK adults then you have a robust sample to represent this group. However, if you then start breaking it down into age bands, regions, genders etc. you need to be careful – 100 is the suggested minimum for a subset, although for some samples 50 may be sufficient. Even so, it is important to make sure any results and comparisons made are statistically significant.
Leading questions – make sure your questionnaire or discussion guide doesn’t include any leading questions. People can be more inclined to agree, or to be biased by the wording of the question. For example a leading question might be “do you agree that cats are better than dogs?”. A better way to ask this (if a closed question) is “Which animal do you think is better? Cats, Dogs, Neither”
Double barrelled questions – only ask one think in a question, otherwise you don’t know what the respondent is referring to. For example, if people disagree with the statement “running is bad for you as it can cause knee problems later in life” we don’t know if they disagree that running is bad for you, or that it causes knee problems later in life, or both parts of the question.
Calculating trend data – say last year 40% of people said that they preferred cats to dogs, and this year 48% of people said that they preferred cats to dogs. This is an increase of 8 percentage points, not an increase of 8% (the actual increase is 20%).
Correlation vs. causation – just because more people now prefer cats to last year, and more people now drink Coke to last year does not mean that drinking Coke causes people to prefer cats.
Self-selection – if a poll was optional, beware! Often you don’t know the make up of the sample (e.g. demographics etc) but it’s also common for people who have strong opinions to respond to this type of poll or survey.
Don’t know? Don’t ignore! – If you’re removing don’t knows from your data, it’s important to make that clear to the audience. If 60% agree is that 60% of the total sample, or just of those that expressed an opinion? The difference can be huge (see page 12 of the BPC guide for a detailed example).

The BPC says: “Anyone using or reporting the results of a poll should look carefully at the questions that were asked – and decide for themselves whether they think they were worded clearly and fairly or not.”

We completely agree, and will work with you to think about the questions before we conduct the research. The old adage of ‘rubbish in, rubbish out’ fits here.

Resources

The Market Research Society (MRS) has teamed up with CIPR and the Royal Statistical Society (RSS) to provide a comprehensive guide for those working in the PR industry on how to use statistics in communications. For the full guide, follow the link here.

The British Polling Council has also provided an easy to read guide for journalists.

The RSS has free online training courses including ‘Statistics for journalists’, ‘Introduction to data’ and ‘Presenting data’.