Survey vs. Questionnaire: What’s the Difference?

Last Updated on: 29th August 2022, 08:09 am

Surveys are very common and effective (mostly) in scholarly research. They are an excellent way to collect data related to human behavior and opinions. And, a survey can support both qualitative and quantitative research and analysis.

In this article I explain the differences between surveys and questionnaires, discuss sampling, and talk about considerations for each of these concepts.

Survey vs. Questionnaire 

Yes, there is a difference! A questionnaire is an instrument (like an interview protocol, an observation plan, or an experiment)—a written set of questions. 

Survey is a broader term that encompasses both the instrument (questionnaire) and the process of employing the instrument—collecting and analyzing the responses from those questions. 

So, you might say, the questionnaire is one component of the survey. Planning a survey is a different task from constructing a questionnaire. The potential errors and bias, and their impacts on reliability are different.

• In surveys, we’re concerned with coverage error (ensuring all prospective groups or characteristics have an opportunity to participate to avoid selection bias), and nonresponse error (low response rate). 
• For questionnaires, we’re concerned with clarity, length, and construct validity, which relates to measurement error (accurately measuring the constructs the questionnaire purports to measure).

Why Is This Difference Important?

Terms matter, and using them properly contributes to your credibility. And, from a practical perspective, understanding the tasks ensures that your research is rigorous, unbiased, and valid. The reliability of your research depends on how you handle both the questionnaire and the survey, and the reliability issues are different for each.

So, let’s delve into each in a little more detail.

Survey Research

The survey is the overall process of using a questionnaire to collect data. 

There are some very important considerations when choosing the survey method. Any choice about your research methodology should fit the purpose and the research objectives. These questions should take you in the right direction:

• What is your research problem? 
• What’s the gap in the current research?
• What must you learn to address that research question? 
• What kind of data do you need?
• What’s the population about which you wish to infer some attribute? 
• What method best generates those data? Quantitative or qualitative research? Survey, interviews, observation, experiment?

Once you choose the survey method, then the next set of questions help scope your effort. Needless to say, especially for busy, starving, stressed students, there are real-world constraints in terms of costs, time, and effort (which is why we sample). 

• How much time do you have?
• How much money do you have to invest in it?
• How feasible is your plan? Can you reach members of the population? Are there any insurmountable hurdles to reaching your population?

You have options: 

• Self-administered survey using mail or hand-delivery?
• Internet-based?
• Access your population through a professional or social group, association, or online social media platform (LinkedIn, Facebook)?
• Use a survey service (e.g., SurveyMonkey)? 

There are advantages and disadvantages to each option. While self-administered surveys provide control and flexibility, they suffer from low response rates and potentially high costs. A service may guarantee a specified sample size of respondents who meet your criteria, and may help construct the questionnaire. But, they also incur some costs to the researcher. Using an association leads to convenience sampling (discussed a bit later).

Once you decide on the survey mode, many of your considerations relate to choosing or designing an instrument, or with sampling. Let’s tackle sampling now.

Sampling

Sampling is a very important aspect of survey research (and, for that matter, most scholarly research). Some simple definitions:

• Sampling: It would be great to obtain data for the entire population. But, due to constraints on resources, you may need to sample and infer characteristics of the entire population.
• Population: The entire set of all objects (or participants) sharing characteristics or qualifications (e.g., all undergraduate students in the U.S.); and to which the researcher intends to infer something of interest.
• Target population: A subset of the population, delimited by some additional characteristics (e.g., undergraduate students in public universities); or, feasibility or access issue.
• Sample frame: The subset of the target population from which an actual sample will be drawn, to which the researcher has access (e.g., undergraduate students in California state universities). The sample frame may be the same as the target population.
• Sample: A subset of the sample frame—those who meet the participant criteria and are contacted to complete the questionnaire; selection is based on a sampling technique (e.g., random sample of candidates within the sample frame).
• Participant criteria: The people who comprise your sample must meet the criteria (characteristics and qualifications) you establish.

Sampling Techniques 

• The purest form of sampling is a completely random sample from the sample frame. This requires the researcher to develop some mechanism for randomly selecting participants (or rely on a service to do it). 
• Convenience sampling is using a mechanism to contact qualified candidates in the sample frame, such as LinkedIn; or people attending a conference. 
• Stratified sampling divides the target population and sample frame into distinct cells: combinations of attributes (e.g., race, gender) proportionate to the population; then samples randomly within the cells.

A final consideration, no matter what sampling technique you use: How you will find and contact participants?

Sample Size 

In quantitative studies for which you will test hypotheses and make inferences about population attributes, there are online tools to compute minimum sample size, including G*Power. Here’s an example:

Say you’re comparing GRE scores by race and gender, using ANOVA, medium effect size, α = .05, power = .90. Using G*Power, you compute a minimum sample size of n_min = 270.

Overcoming Response Error

With a minimum sample size (n_min) calculated, you must ensure that either the survey service obtains the minimum sample size, or you must send out a sufficient number of questionnaires to account for the response rate. AND, you should consider the likelihood of incomplete, invalid, or corrupted questionnaires.

Continuing with the example:

Survey response rate (from a similar study documented in a journal article) is 30%. Rate of corrupted, incomplete, invalid questionnaires is 10%.

n_min = 270 (min required sample size of valid questionnaires)

270 = .90 × n₂ ⇒ n₂ = 300 (n₂ is the number of questionnaire returned)

300 = .30 × n₃ ⇒ n₃ = 1000 (n₃ is the number of questionnaires sent out)

So, to ensure you have the minimum number of valid, complete questionnaires (270), you would need to send out 1000 questionnaires to prospective participants.

Minimum Sample Size

Some survey services may guarantee your minimum sample size. But, (huge point!), be sure to consider the rate of questionnaire validity, and call that the minimum sample size for the survey service.

There’s little you can do after data collection to obtain more data if later you find that some of it is corrupt. And, it’s tragic to get to data analysis only to find out your sample is too small.

Let’s turn now to the instrument used in the survey method.

Questionnaires

The style of a questionnaire should fit the purpose and the research objectives. That means, using the kind of vocabulary that your target population is comfortable with. And, choosing a format that serves your data collection needs.

Here are some considerations:

• Questionnaires can sometimes be obtained off-the-shelf, having been used in previous research. These may need permission obtained from the developer. Or, they can be self-developed. 
  • In either case, you must provide evidence of instrument validity, and the details of development and purpose and prior use. If previously used, this information should be available in a citable source. 
  • One measurement of internal construct validity is Cronbach’s alpha. For off-the-shelf questionnaires, did the author report Cronbach’s alpha? If self-developed, you as the researcher must report on Cronbach’s alpha.
  • For self-developed questionnaires, perform a pilot study of your instrument to ensure it is understandable, with no confusing questions or potential bias; that the length is appropriate; and to compute Cronbach’s alpha.
• Provide an introduction to your study, which is clear and professional, and addresses your purpose.
• Consider return envelopes and postage, anonymity of the participants, deadlines, and incentives. (Another important point! Consider the Institutional Review Board [IRB] policies for any agency with a stake in the research, such as a university or an organization targeted for a survey.)
• Will it be cross-sectional (one point in time across sample frame) or longitudinal (data collected over time)?
• Depending on the research question, will the questionnaire have structured questions using a scale, semi-structured short-answer questions, or open-ended questions?
  • If a structured questionnaire, what kind of responses are needed, which determines what kind of scale?
  • Will the responses be categorical/nominal? Dichotomous (e.g., yes or no)? Ordinal (such as a Likert scale)? Or numerical (a measured or counted quantity such as age or test scores)? 
• When writing the questions, consider objectivity and avoid language or questions that might be perceived to be biased.
• Consider complexity and length of time to complete.
• Use consistent ordinal responses (positive is high and negative is low); or reverse scoring will be needed.

Survey vs. Questionnaire: What’s It All Mean?

Surveys and questionnaires are different animals. A questionnaire is a component of a survey. And, there are different considerations for each, different sources of error and bias, impacting reliability and validity. 

But, why is this all important?

The survey method, with a properly designed questionnaire and rigorous sampling, is very useful in scholarly research.

Performing good survey research boils down to these three principles:

• The purpose of the research, which drives research methodology, the survey process, and the questionnaire as a data collection instrument.
• Rigorous planning for each component of that research, to meticulously address each potential source of bias and error.
• Disciplined execution to obtain, through sampling, a valid data set and to perform analysis to make inferences about the population.

Happy surveying!