Getting the best out of your survey

many of you are starting to use surveymonkey or other online survey tools to collect some exploratory data about your topics.  This is a great start to your inquiry.  However, not all surveys or questionnaires are created equal!  There are some simple, but effective things you can do to make any questions you ask, but particularly questionnaires better.

Survey research is often lumped in with quantitative research. It is an incorrect assumption to say that the only data you can gather from surveys is about numbers and percentages. Surveys are one of the most used and important forms of data collection instrument because they allow you to collect a lot of data from people in a relatively short period of time at a lower cost.

The survey is the package that presents the questions and also contains the record of responses for each respondent. The first section of the survey generally introduces the study to the respondents. The middle section contains the items and scales to measure the survey topics in a logical sequence. The final section usually has questions to measure the respondent’s demographic characteristics, like age, gender and income, so they can be grouped and compared.

The survey usually has other components, which help in data preparation and analysis. This includes spaces to record the data and codes that identify particular respondents. The survey may also have interviewer instructions to make sure that each respondent gets asked the same questions in the same way. This ensures answers that can be compared, collated and analysed in the most effective way. Interviewer instructions are a standard part of survey design. These instructions do not suggest that interviewers do not know what they are doing, but are designed to assist interviewers through sometimes confusing and complex surveys.

How many times have you asked what you thought was a clear question only to get an answer that doesn’t make sense to you or to realise that the person has interpreted the question differently to how you intended? For example, imagine asking a friend this question: ‘How is that cool new iPod you bought last Saturday?’ and your friend replies: ‘Oh no, I didn’t buy it on Saturday, I just went to visit my grandmother on Saturday’. Obviously, you wanted to know about the new iPod, regardless of when it was bought. A good survey is only as good as the questions it asks.

There are a number of very important guidelines to keep in mind when designing a survey. Most of these ensure that the reliability and validity of the survey is kept intact and you get the information you set out to get in the first place! No researcher wants to spend weeks of time and thousands of dollars on a research project only to find that they can throw out the results because of a bad question.

The problem with survey design is a simple one – there is no one right way. The questions you ask, the order you ask them in all will change according to the situation. Survey design is never black and white; it always has shades of grey.

Some key things to think about as you write your survey instrument…

1. What questions should be asked?
2. How should questions be phrased?
3. How to ask good questions
4. What is the best question sequence?

Let’s talk about questions specifically. Writing questions is one of the most important aspects of survey design. Here are some of the most common considerations in writing questions for a survey…

1. Clarity: The questions you ask must be clear and without ambiguity. Your objective is to make sure your respondent interpret the question differently. You want to make sure each respondent answers the question in the same way. Keep your language simple, your questions clear and easy to understand and don’t get caught up in jargon or complexity if possible.
For example, if you are asking people about how often they do something, using a scale such the one below is open to interpretation
(i) Very Often
(ii) Often
(iii) Sometimes
(iv) Rarely
(v) Never
It is better to make the choices more concrete, for example;
(vi) At least once a day
(vii) 2-5 Times a Week
(viii) About Once a Week
(ix) About Once a Month
(x) Never

2. Embarrassing Questions: Embarrassing questions dealing with personal or private matters should be avoided, unless necessary. Questions involving personal or private data should have those cleared through the ethics approval process.

3. Hypothetical Questions Hypothetical questions are based on the respondents opinions without having them based in actual experience. Sure, they make fun games at pubs, but are not really indicators that if put in a situation in real life that the respondent would behave in the same way.
If you were the director of theatre, what would you do to stop declining attendances?
4. Prestige Bias: This is one of the hardest issues to avoid. Prestige bias is the tendency for respondents to answer in a way that makes them feel better. For example, asking people if they donate to children’s charities will often result in the person answering yes simply so they don’t look bad in the eyes of the person interviewing them. Think carefully about whether you are getting a true response at all time.

5. Double-barreled questions
This is when you as the respondent to comment on two or more actually quite different concepts. The result of this type of question is that you may not get a true response on either of your questions.

Example: How do you rate the quality and taste of the food you have just eaten in my restaurant?

The quality may have been good, but the taste was like old boots. How would you answer this question?

6. Loaded questions
Loaded questions are when you attempt to influence the answer the respondent gives by introducing bias into the question.

Example: How much did you enjoy the performance?

7. Mutually exclusive scales
Sometimes, you ask respondents to put information on a scale or a tick box. Mutually exclusive scales ensure that the person ticks the most appropriate answer

What age are you currently (please tick)?
18 or less
18-25
25-35
35-45
45 or over

What happens if a person is 35? Which box do they tick?

8. Open ended questions
Open ended questions are when you want more than a yes/no answer to your questions. Open ended questions are designed to draw out detail, observations, and feelings – all qualitative data. Open ended questions often ask the question why? Some examples include;

Can you describe the emotions you felt at the completion of the performance?
In what ways do you feel you could improve the service to the customer?

9. Closed questions
This is where you want to limit the range and scope of answers that arise from specific questions. Closed questions allow the researcher to tabulate (bring data together from different respondents) by using percentages.

Example: What is your sex?
Did you purchase your ticket for the performance?
Online
In person
On the phone
Other (please specify)

10. Asking questions that you don’t really need to know the answer to

Respondents have a limited amount of time generally and you need to be aware of the issues surrounding asking too many questions. Make sure the questions you ask help solve your research problem. If they are irrelevant or confusing, rewrite them or get rid of them altogether.

The funnel technique
A good way to start writing your survey is to think carefully about the broader issues you want to investigate, then with each successive question you get down to the nitty gritty. This is called the funnel technique because your questions start at a very broad level and gradually funnel inwards.

For example:

• What do you feel are the three main issues for dance educators in improving the skills level of primary school children?
• Now thinking specifically about issue number 1, in what was does it impact on your curriculum development?
• What strategies have you used to keep up-to-date with current thinking around issue number 1?

Do I have to all of this on paper?
No way! There are a number of online ways to both design and administer a survey. The simplest to use of all of these is survey monkey (http://www.surveymonkey.com). It is free to use for a small number of surveys.

Here is a link to a little survey I have put up as an illustration. Have a go at doing the survey and see what it can do. When you have done the survey, when we start to talk about data collection and analysis I will use the data to illustrate my talk.

Here is the link to the survey, it is only short (around 6 questions)

http://www.surveymonkey.com/s/PBD9LHN

Here is a youtube video on using survey monkey

Some resources to have a read through

http://www.lboro.ac.uk/library/skills/Advice/QuestionnaireDesign.pdf

http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires/5

Topic setting in a vague and indefinite world

Problem definition is often considered the most important part of the research process. If we do not look for the reasons behind the problems our research may be misdirected and not solve the real problem. It may only provide temporary relief of the symptoms surrounding the problem – just like your headache tablet.

Proper problem definition ensures that you are asking the right questions. Asking irrelevant questions will provide meaningless information. The initial problem that you identify in a research project is likely to be a symptom of the problem rather than the actual problem. The symptoms alert us to the fact that the problem exists. As researchers we need to investigate further to determine what the problem actually is. How do I start defining my research problem/topic

1. Ask questions about what you know about the topic and what you don’t know
Example: I want to find out why pedestrian crossings are painted black and white? What do I know: they are black and white. The road is black and the paint is white. What I don’t know: Why white paint? Why not blue or pink or yellow? When did they start being painted in white? Were they always white? (In Australia, they used to be yellow – true story)

2. Define the problem further from your questions
What is the benefit of pedestrian crossings being black and white? Is it the most effective combination of colours?

3. Locate some sources of data (literature, theory, other research) about your topic
JOE MORAN (2006). CROSSING THE ROAD IN BRITAIN, 1931–1976. The Historical Journal, 49 , pp 477-496
Moran (2006) discusses the history of pedestrian crossings in the UK, with a focus on why they have evolved into the form, colour and usage we have today. You may need to do further research on your topic before you even start to get a final version of your topic. This process is called exploratory research. To define the problem correctly, you may need to conduct some exploratory research. This is research that helps you to discover the problem and what is involved in that problem. Some activities that might be considered exploratory research include;
* Experience survey – discussion with decision makers and interviews with industry experts
* Case study – examining what has happened in similar situations
* Secondary data analysis – including historical data, government reports

The purpose of exploratory research is to gain an understanding of the major components of the problem in order to define the problem correctly. Exploratory research will help you:
* Identify (and eliminate) symptoms
* identify the underlying problems
* develop research questions.

This will allow you to break the research problem into its key components; i.e. the key types of information that needs to be gathered. These key components can then be refined into research questions, which need to be answered in order to find a solution/s to the problem. A properly formulated research question will make the rest of the process flow more easily, because the objectives of the research project will be clear and you will understand exactly what information you need from your research. A question always needs an answer. Finally, a good little test about your research problem is whether it is;
Feasible – can you do it in the time allowed, do you have the resources and skills to complete the research
Interesting – Is it interesting enough to keep you going for the year? Is it interesting for anyone to else to read and use?
Novel – Is it unique and new? Has it been done before?
Ethical – Are you dealing with children? Privacy? Is your research ethical in that it has been collected without breaking laws and ensuring that the way you have collected the information complies with ethical principles
Relevant -Does the research meaning something? Can it improve practice? Can other people use your research to investigate further?

Check out this little youtube video on research topics…

and

4 different ways to ask the same question

Inquiry is a very powerful tool for the exploration and understanding of practice. It is positioned in the notion of being curious, of finding things out that interest us and others. Inquiry should also have some sense of significance, in that it should mean something to you, to your industry, to your peers.

Inquiry is not simply a process of finding out what people think. Sure, that may be part of it, but inquiry can also be about comparing what people think to what experts in the field have written. It may be testing your own opinions. It might be finding out that if I change one thing what impact will that have on the rest of the world, like pushing down one domino and setting off a chain reaction.

Inquiry can also get you out in the field, talking to people and interacting with a wider professional network. This can also take the form of working with peers in impacting directly on your practice or profession through testing and implementing a change in your workplace and then evaluating the impact of that change.

Let’s start this discussion with an example. I would like to know about why people use YouTube. It has been around 5 years, I know it’s popular. I use it a lot both personally and as part of my practice as a teacher. I know a lot of other people but use it, but I am curious about why they engage it. My own perceptions are that there is a lot of junk there, things get put up and taken down a lot. Often the sound and picture quality aren’t great, yet I use I still use it!

Researching YouTube part 1 – why?

This YouTube video was made by a university student in the US. It asks a simple question of a diverse range of people. That question is…’why we tube?’

So, what did you get from watching that? I saw a whole heap of observations about the motivations behind why people use YouTube. Those observations vary from it being voyeuristic to it simply being fun. The people interviewed were real people, speaking hopefully openly and honestly about the internet usage? What did this tell us about YouTube?

Researching YouTube part 2 – how?

The next part of this is a video from a guy who is showing a practical example of how YouTube is used by people to interact with lawmakers. This type of research is different. He isn’t asking people for their opinions. He is using a case study or example to make an observation on how YouTube can be used, and perhaps making an inference or suggestion that the experiences of these kids from West Virginia might be applicable or useful to other situations which require a conversation or dialogue to be started.

Researching YouTube part 3 – how many?

These two links have been collected in different ways. One is research conducted by a private market research company for the purpose of identifying how many hours’ people spend watching TV (and as part of this, watching online video). This research would be used by marketers and advertisers as a way of measuring the value of using TV advertising. The second link comes from YouTube itself. It describes the number of people who are using YouTube currently. Now, have a think about these two sources of information. What issues arise for you?

http://blog.nielsen.com/nielsenwire/nielsen-news/americans-watching-more-tv-than-ever/

http://techcrunch.com/2010/05/16/five-years-in-youtube-is-now-streaming-two-billion-videos-per-day/

What type of trust would you place in each of these sets of data? Would you trust them equally, or in different ways? In what ways does the reliability of the information vary from article 1 to article 2? How might you use this information to help in your understanding of why people use YouTube?

Research YouTube part 4 – Making connections
The final link is for an academic journal article on the connection between YouTube and the formation of social networks.

http://www.cs.uwaterloo.ca/~apidduck/CS432/Assignments/YouTube.pdf

It argues that in some ways YouTube alters the public and private behaviours of the users, and those changed behaviours are used by youth to ‘carve out’ private space amongst a highly observed and recorded society, and that these users are fully aware of the practices required to maintain their desired level of privacy or anonymity.

So, all four resources are different types of inquiry. They are generally researching the same thing, the motivations that underpin or inform people’s usage of youtube. They tell us different things about that phenomena, and they have different implications for our practice. I can also this information in different ways. I might cite or quote the academic article. I might show the video in a class I am teaching about social networking, or I might apply the West Virginia example to my own practice (could students submit questions to a class discussion via YouTube). So, as you are thinking about your inquiry, dig deep into what people have already found out about your area. Use search engines like Google, or the library catalogue and databases to find out what has already been done. It doesn’t mean that you should stop if you think that your area has already been researched. Maybe, you have a different insight, or a different situation that might reinforce or challenge the other stuff. Either way, look at each piece of information with a critical eye. Can you trust it? How reliable is it? How was the data collected? Who collected the data?