many of you are starting to use surveymonkey or other online survey tools to collect some exploratory data about your topics. This is a great start to your inquiry. However, not all surveys or questionnaires are created equal! There are some simple, but effective things you can do to make any questions you ask, but particularly questionnaires better.
Survey research is often lumped in with quantitative research. It is an incorrect assumption to say that the only data you can gather from surveys is about numbers and percentages. Surveys are one of the most used and important forms of data collection instrument because they allow you to collect a lot of data from people in a relatively short period of time at a lower cost.
The survey is the package that presents the questions and also contains the record of responses for each respondent. The first section of the survey generally introduces the study to the respondents. The middle section contains the items and scales to measure the survey topics in a logical sequence. The final section usually has questions to measure the respondent’s demographic characteristics, like age, gender and income, so they can be grouped and compared.
The survey usually has other components, which help in data preparation and analysis. This includes spaces to record the data and codes that identify particular respondents. The survey may also have interviewer instructions to make sure that each respondent gets asked the same questions in the same way. This ensures answers that can be compared, collated and analysed in the most effective way. Interviewer instructions are a standard part of survey design. These instructions do not suggest that interviewers do not know what they are doing, but are designed to assist interviewers through sometimes confusing and complex surveys.
How many times have you asked what you thought was a clear question only to get an answer that doesn’t make sense to you or to realise that the person has interpreted the question differently to how you intended? For example, imagine asking a friend this question: ‘How is that cool new iPod you bought last Saturday?’ and your friend replies: ‘Oh no, I didn’t buy it on Saturday, I just went to visit my grandmother on Saturday’. Obviously, you wanted to know about the new iPod, regardless of when it was bought. A good survey is only as good as the questions it asks.
There are a number of very important guidelines to keep in mind when designing a survey. Most of these ensure that the reliability and validity of the survey is kept intact and you get the information you set out to get in the first place! No researcher wants to spend weeks of time and thousands of dollars on a research project only to find that they can throw out the results because of a bad question.
The problem with survey design is a simple one – there is no one right way. The questions you ask, the order you ask them in all will change according to the situation. Survey design is never black and white; it always has shades of grey.
Some key things to think about as you write your survey instrument…
1. What questions should be asked?
2. How should questions be phrased?
3. How to ask good questions
4. What is the best question sequence?
Let’s talk about questions specifically. Writing questions is one of the most important aspects of survey design. Here are some of the most common considerations in writing questions for a survey…
1. Clarity: The questions you ask must be clear and without ambiguity. Your objective is to make sure your respondent interpret the question differently. You want to make sure each respondent answers the question in the same way. Keep your language simple, your questions clear and easy to understand and don’t get caught up in jargon or complexity if possible.
For example, if you are asking people about how often they do something, using a scale such the one below is open to interpretation
(i) Very Often
It is better to make the choices more concrete, for example;
(vi) At least once a day
(vii) 2-5 Times a Week
(viii) About Once a Week
(ix) About Once a Month
2. Embarrassing Questions: Embarrassing questions dealing with personal or private matters should be avoided, unless necessary. Questions involving personal or private data should have those cleared through the ethics approval process.
3. Hypothetical Questions Hypothetical questions are based on the respondents opinions without having them based in actual experience. Sure, they make fun games at pubs, but are not really indicators that if put in a situation in real life that the respondent would behave in the same way.
If you were the director of theatre, what would you do to stop declining attendances?
4. Prestige Bias: This is one of the hardest issues to avoid. Prestige bias is the tendency for respondents to answer in a way that makes them feel better. For example, asking people if they donate to children’s charities will often result in the person answering yes simply so they don’t look bad in the eyes of the person interviewing them. Think carefully about whether you are getting a true response at all time.
5. Double-barreled questions
This is when you as the respondent to comment on two or more actually quite different concepts. The result of this type of question is that you may not get a true response on either of your questions.
Example: How do you rate the quality and taste of the food you have just eaten in my restaurant?
The quality may have been good, but the taste was like old boots. How would you answer this question?
6. Loaded questions
Loaded questions are when you attempt to influence the answer the respondent gives by introducing bias into the question.
Example: How much did you enjoy the performance?
7. Mutually exclusive scales
Sometimes, you ask respondents to put information on a scale or a tick box. Mutually exclusive scales ensure that the person ticks the most appropriate answer
What age are you currently (please tick)?
18 or less
45 or over
What happens if a person is 35? Which box do they tick?
8. Open ended questions
Open ended questions are when you want more than a yes/no answer to your questions. Open ended questions are designed to draw out detail, observations, and feelings – all qualitative data. Open ended questions often ask the question why? Some examples include;
Can you describe the emotions you felt at the completion of the performance?
In what ways do you feel you could improve the service to the customer?
9. Closed questions
This is where you want to limit the range and scope of answers that arise from specific questions. Closed questions allow the researcher to tabulate (bring data together from different respondents) by using percentages.
Example: What is your sex?
Did you purchase your ticket for the performance?
On the phone
Other (please specify)
10. Asking questions that you don’t really need to know the answer to
Respondents have a limited amount of time generally and you need to be aware of the issues surrounding asking too many questions. Make sure the questions you ask help solve your research problem. If they are irrelevant or confusing, rewrite them or get rid of them altogether.
The funnel technique
A good way to start writing your survey is to think carefully about the broader issues you want to investigate, then with each successive question you get down to the nitty gritty. This is called the funnel technique because your questions start at a very broad level and gradually funnel inwards.
• What do you feel are the three main issues for dance educators in improving the skills level of primary school children?
• Now thinking specifically about issue number 1, in what was does it impact on your curriculum development?
• What strategies have you used to keep up-to-date with current thinking around issue number 1?
Do I have to all of this on paper?
No way! There are a number of online ways to both design and administer a survey. The simplest to use of all of these is survey monkey (http://www.surveymonkey.com). It is free to use for a small number of surveys.
Here is a link to a little survey I have put up as an illustration. Have a go at doing the survey and see what it can do. When you have done the survey, when we start to talk about data collection and analysis I will use the data to illustrate my talk.
Here is the link to the survey, it is only short (around 6 questions)
Here is a youtube video on using survey monkey
Some resources to have a read through