The sample. Is the sample size sufficient? For example, something claiming to represent the UK with only 40 responses probably won’t help you a great deal. 40 responses to an internal survey within a department of 50 people, however, could be useful.
The method. How has the data been collected? An online survey that shows everyone uses the internet, for instance, would only account for people that have the internet to take the survey. Check that the method for data collection gives an accurate representation.
Doing your own research
If there’s nothing out there or what does exist isn’t enough to build an accurate picture, you might need to do some research of your own. This means choosing the right method.
The good news is that there are only two main methods to consider: quantitative and qualitative.
Quantitative research is all about the numbers. It aims to quantify things. The cold, hard facts. If you’ve ever completed a survey filled with multiple choice questions and rating scales, that’s quantitative research. With the right questions and sample, it’ll produce data that helps you see the big picture.
Qualitative research brings depth and detail in a way that quantitative research often doesn’t. You can explore what people really think and follow up on important points there and then. You’ll have seen this in practice if you’ve been at a focus group and or taken part in a depth interview.
Which should you use?
If you’re looking to quantify something, such as how many of my customers would recommend my services to a colleague or which branding option do they like most, quantitative research is your best bet. You want numbers that are representative of your customers.
If you’re looking to get some detailed feedback on a specific service or branding option and encourage open discussion, qualitative research — conducting in-depth interviews or focus groups for example — will probably give you better results.
But it’s not always a one or the other deal. Like tea and biscuits, qualitative and quantitative often work best when paired together. We sometimes call this mixed methods.
Say you hosted an event, you might start with an evaluation survey at the end to ask all your attendees how they rated things like the speakers, location, accessibility, food and drink.
You have a look at the results when you get back and find people didn’t really like your speakers. But you have no idea why. So how are you going to know what to do for the next event? That’s when you might want to conduct a few depth interviews with a sample of attendees to find out what they didn’t like about the speakers, what would work better for them, that kind of thing.
Or if you were thinking ahead, stick some open questions in your survey where you can ask them to comment on or explain something.
For a bit of further reading on this, Global Web Index has listed the pros and cons of qualitative and quantitative.
And the survey says…
I’m more of a quantitative researcher so that’s what I’m focusing on for the rest of this post. If you want to know more about qualitative research methods, check out the Association for Qualitative Research.
A survey is one of the quickest and easiest ways to gather information from a large number of people.
This is where all that thinking you did at the start comes to the fore.
Start with the questions
If you’re lucky, people might give your survey a go. But if the questions aren’t engaging or relevant, they’ll soon find something better to do.
You should always be focused on gathering only the data you need to keep your survey as short and simple as possible. You’re wasting everyone’s time if the question isn’t useful.
Ask the right questions in the right way.
- Keep questions simple. Use language your audience can relate to and make them easy to understand. Just because you know what a transient diaphragmatic spasm is, doesn’t mean everyone will. It means getting the wind knocked out of you, if you were wondering.
- Keep questions neutral. Saying “you enjoyed our event, didn’t you? DIDN’T YOU?” isn’t exactly balanced.
- Keep response options neutral. A balanced question can be let down by the response options you give. “Did you enjoy the event? (Absolutely, yes, sure did, no)”
- Avoid double-barrelled questions. Questions like “How would you rate the food and location?” are different topics with different answers. The location might have been great, but the choice of biscuits with your brew was poor. Stick to one topic at a time.
- Consider including “Don’t know” or “Does not apply” as a response option if there’s a chance your question might not apply to everyone. For example, I would struggle to answer “how would you rate our museum?” if I’ve never been.
The design is also important here. Not in a bells and whistles kind of way. It needs to look presentable, but you don’t need flashing banners or pop-up GIFs.
I mean from a user experience point of view. Use routing and logic in your survey so people only get asked the questions that are relevant. Consider the flow of your survey too. Is there a logical progression through the topics you’re covering? You might want to break your survey into sections to help with this.
It’s worth checking out this article on what people like and dislike about doing surveys.
Dealing with the data
Yep. It’s the GDPR bit.
If you’re asking for any kind of personal data in a survey — name, postcode, email address, etc — you need consent from the respondent. You also need to make it clear how their data will be used, how long it will be retained and how they can access, rectify and erase data.
Here are some ways you can stay on the right side of GDPR.
- Explain why you’re collecting data from them and how you’ll use it
- Explain how you process their data and keep it secure in a Privacy Notice and link to this in the survey intro
- Check any third-party apps or software being used to make sure they comply with GDPR.
How should you run your survey?
In most circumstances, online is the way to go. Unless you need to reach people who might not have access for whatever reason. There’s still a small percentage of the population who don’t go online, remember.
Survey software is quick to set up and easy to use for you and your audience.
Most platforms will have free account options (Google Forms is completely free) and can be set up in minutes without technical expertise.
Everyone has their own favourites. I pay to use Survey Gizmo at the moment. I've also used Qualtrics, Snap Surveys and Smart Survey in the past. I’d recommend them all.
But I’ve also heard some good things about Typeform, Survey Monkey and Google Forms depending on what you need.
Free accounts are limited in terms of what you can do, but there’s enough features with each to be able to set up and run a basic survey. If you do need to access the additional features or response options, monthly subscriptions are available. So you can always pick one, use it and end your subscription when you’re done.
You’re spoilt for choice so my advice is play about with a few and find one that you like. And if you think you’ll be running surveys regularly, consider signing up for a premium level as you can do a lot more with these.
Getting people to respond
Have you ever seen the movie Field of Dreams? It stars Kevin Costner as a farmer who hears a mysterious voice in his cornfield saying, “if you build it, he will come.” So he builds a baseball field on his land and low and behold, ghosts of great players turn up and play ball.
He built it. They came.
Unfortunately, you can’t Field of Dreams a survey.
You can have the best questions, laid out in the best way, using the best software, but it doesn’t guarantee the best response. In fact, it doesn’t guarantee any response at all.
You need to give people a reason to respond. I wrote a post on this and how consultation and engaging your audience relates to letting the dogs out for a wee. You can read it here.
Engagement only matters if you have the right audience in the first place though. Think about your sample. You need to be sure that you’re reaching the right people.
Survey data can be influenced by what’s known as the “Angels and Demons” effect (not the movie with Tom Hanks). For example, if you’re trying to get feedback on your services:
- You’ll have some loyal customers who won’t have a bad word said about you – your angels
- But you might also have some disgruntled customers or arch enemies (maybe not) who are likely to be more negative – your demons
Keep this in mind when sharing your survey and choosing your sampling method. Too many of one or the other could skew your results. Ideally, your sample will be representative of your target audience or population.
You scratch my back, I’ll scratch yours
There’s nothing like a good incentive to get people interested in a survey. Brands do this all the time by offering to enter you into a prize draw in exchange for feedback on a recent purchase.
If you’ve got something of value to offer that’s not going to sway feedback, go for it. However, you should read through the Market Research Society Code of Conduct first. There are rules around what you can and can’t do with incentives — goods or services, or vouchers to purchase client goods or services being one.
If you’re still struggling to get people involved, it’s possible to run your survey through a representative panel such as Cint, Populus and Dynata, or use recruiters to increase distribution. You can set quotas to make sure you get a minimum number of responses from particular audience groups, for example by age, gender or customer type. These methods will cost you, though.
Making sense of it all
We’ll fast forward a bit now and imagine that, despite your lack of a budget, your research has unearthed a ton of useful information and your survey has been a roaring success.
The next step is to take all of that good stuff and present it in a way that makes sense, not just to you but to everyone that’s going to see it.
There are all kinds of tools and techniques to help you do this.
Online survey platforms come with built-in tools that neatly and automatically organise data so that you can easily identify patterns. They also let you pull all of the data into a spreadsheet to get a bit more hands on. It’s worth looking at this when you decide which platform to use.
Excel might not be the prettiest of tools, but it’s brilliant for cutting up and restructuring data. Use it to run lookups on things like postcodes or demographics and create pivot tables and charts that display data clearly. If you’re reading ‘lookups’ and ‘pivot tables’ and wondering what on earth I’m talking about, here’s a guide on how to use Excel’s VLOOKUP feature. And here’s another guide on creating pivot tables and charts.
Coding comments helps to create categories for your open questions (or qualitative data if you’ve run a focus group or depth interviews) so that it’s easier to see similarities, differences and frequencies. Codes are usually words or short phrases that you select from text and give a code name to. Then, the next time you come across text with the same meaning, you give it the same code name.
For example, let’s say you were surveying people about what they liked best about your event and this was the feedback from the first two respondents:
Respondent A: I really enjoyed the speakers.
Respondent B: My favourite part of the event was the talks on social media and SEO.
You could code both sentences “speakers”. You may also decide to code “social media” and “SEO” as “topics.”
It depends on what the question is and what you’re looking to get out of the data, but coding is a great way to identify common themes. If you want a better understanding of how it all works, have a read of this.
SPSS software can help you get more out of your data analysis. It’s designed for all skill levels, but in my experience it helps if you’ve used analysis tools before. If you’re a student, you can pick up the software for free. You can also try it for free for 30 days before the monthly fee kicks in. If you’d rather not pay, there’s a free SPSS alternative called PSPP (see what they did there?) that works in much the same way. Q Research Software is also a great alternative (I’ve used it myself) but there’s an annual cost for this.
Free tools like BatchGeo and OpenAudience can help you map or understand postcode data. Just paste in your postcodes and sit back while the tools go about their work.
QGIS is an open-source system for creating, editing, visualising and publishing geospatial data. It’ll help if you’ve done mapping and worked with geographic information system (GIS) applications before. If not, QGIS Tutorials and Tips is filled with…er, tutorials and tips.
R and Python – I’ve not used these, but increasingly analysts are using them to manipulate and analyse heavy statistical data. Both are free but, by all accounts, will take some time to learn.
Presenting the data
Start with The Chartmaker Directory by Visualising Data. It’s a free tool that answers the most commonly asked question of all “which tool do you need to make a chart?”. Use it to find out which charts can be made using which tools. If you’re a geek and like charts, you can lose hours in here.
Tableau is a software platform that lets you import your data to build interactive dashboards and data visualisations, which can be a really useful way to present complex data to an audience in a simple, visual way. It’s not exactly cheap though if you want to keep your data to yourself. But there’s a free version if you’re happy to make your data public. And if you’re a student or an instructor at an accredited academic institution, you can pick up a licence for free, There are other data visualisation platforms out there too, such as Power BI and Qlik.
Infographic tools are easy to use and can be a nice way to show off your data. If the visual presentation is important to you, it’s probably best to find a graphic designer to help. They’ll have the skills and the software to create something professional. But if your budget won’t stretch that far, any one of these will let you knock up some visuals for free:
Make the data matter
You’ve done the thinking, found some data, gathered some more data, organised it and turned it into something that’s easy to understand.
Now you need to return to the reason for doing the research in the first place: to make a difference.
Research is about finding problems and discovering opportunities. It’s the thing that informs decisions and change. Put your hard work to good use by showing your data to the right people and using it as a way to strengthen your argument. Did you know that 95% of CEOs feel more confident about a decision if they have the data to back it up? (I made this up, but you get the idea)
If you’ve got an audience, show your research to them. Publish it for the world to see. Heck, you’ve spent long enough on the project, having as many people see it as possible is the least you deserve.
Use your data to prompt discussions and generate ideas. Then use what comes out of that as the basis for your next research project. But maybe have a cup of tea first.
I send out an email once a month on stuff I think you might find interesting. Fancy signing up? Go on, it’ll make my day.
And if there’s a research project you need some help with, you know where I am.