Ranking questions in PR Research – Not my cup of tea

“Can we get respondents to put the options in rank order?”. If you’ve ever asked me this question, you’ve likely received a polite “I don’t think you should, because…”. Secretly, I’ve taken my Voodoo Doll that represents the PR industry out from under my desk and stuck another pin in it (don’t worry, I don’t really do that, but you get my point!). 

When creating a PR or Marketing campaign that uses research, you need stats that give you snappy headlines. Ranking questions don’t achieve this – here’s why:

1. Making sense of the data you get back is a struggle

The data you get back from ranking questions will show the percentage of people that have selected each option at each position (see the example table below).  Say you ask people to rank five options from one to five, that’s 25 data points, and if you break that down by five age groups that’s 125 data points – Wowza!

This data table shows the results for just one option.  You can start to imagine how complicated this could get when you have a table for each option, with age and gender breaks as well.

There is an overflow of data that you’re simply not going to use. Have you ever seen a headline saying “40% of people said X was their third favourite thing!!”? No, of course you haven’t! You’ll find yourself sifting through potentially hundreds of data points before finding the key ones.

Some people try to calculate an “average ranking position” – mathematically this is inaccurate, and again you’ll end up with non-sensical stats for your content. E.g. “On average people ranked this as their 1.546th favourite thing” Eh?

You’re better off using a different solution to get those snappy headlines (which I’ll show you later!)

2. Respondents find it difficult to put options in rank order

If I give you a list of 10 films you’ve seen, and ask you to put them in order from most favourite to least favourite, can you think of the answer in less than 30 seconds, and not regret your response afterwards? Comparing each film against each other, and then ranking from best to worst when so many combinations are possible is a tough ask! Here we can see how ranking questions differ from rating questions in the amount of mental effort required:

Ranking questions make surveys longer and more difficult, which means respondents are more likely to lose focus and give you rubbish answers (and you’ll have to pay extra for an extended survey!)

3. Respondents are forced to differentiate between options that they think are similar

Ranking questions “force” respondents to put options in an order. It’s possible that they think two or more options should hold the same ranking position, but they won’t be able to select this in the survey. Choosing which similar option you prefer can be a brain scrambler – for example, “Would you rather fight a horse-sized duck or a hundred duck-sized horses?” – I’ll give you a few minutes with that one…

If respondents don’t know what order to put their options in, they’re likely to select random boxes just to get through the question – this means the results are less accurate!

What should I use instead of a ranking question?

Let’s take a simple example …You want to find out what factors are important to someone when purchasing a mobile phone. E.g. 

• Price
• Design
• Usability
• Etc.

If you want a single stat to show most preferred e.g. 30% say that price is the most important factor when purchasing a mobile phone, more than design (20%) and usability (15%) then use:

Which of the following is most important when purchasing a mobile phone? Please select one

This is a single choice question

If you want a single stat to show order of importance e.g. Price is the key factor when purchasing a mobile phone. 60% said it was important then use:

Which of the following are important when purchasing a mobile phone? Please select all that apply

This is a multiple-choice question. This will also yield a larger % as you are asking respondents to select multiple options. A higher % can look better in a headline

When you want more depth and flexibility with the data use:

Please rate on the following scale how important each of the following are when purchasing a mobile phone. i. Very important ii. Quite Important iii. Not important

This is a scaling question. With this question we have collected more depth in the data, because we know the extent to which each of the options are important.

Here are a few more tips for thought leadership questionnaire design.

If you would like advice on your questionnaire, drop us an email team@sapioresearch.com or give us a call on +44 (0) 207 2361 604