20.4 – Conducting surveys

Introduction

What is a survey? A survey is a method of collecting information from a sample of individuals from a study or source population —  drawn from a reference population — the benchmark group used to compare statistics. The number of individuals in the sample is the sample population. By information we may refer to ad hoc or targeted, where ad hoc refers to a survey designed to get fast updates on an emerging issue rather than a focused or targeted approach which are based on a specific population.

Targeted surveys are intended to be representative of a larger population while responses from ad hoc surveys would be limited to the specific context in which the survey was conducted. The CDC’s CDC WONDER is described as an ad-hoc query system for analyzing existing public health data; the National Health Interview Survey (NHIS), used to “monitor the health of people across the country,” is an example of a targeted survey system because it focuses specifically on “the civilian population of the United States, excluding active-duty military, people in correctional facilities or long-term care, and the homeless.” For example, consider telehealth: ad hoc questions include “How satisfied were you with the length of time you waited in the virtual waiting room today?” compared to questions directed at targeted population, “How well does your provider create a safe, nonjudgmental environment during virtual visits?” (see Xu et al 2022 for telehealth targeted survey example; for ad hoc survey example on telehealth, see White et al 2020).

Surveys are implemented in many fields including biomedical work (see Chapter 5.4), but also in environmental (see People’s Climate Vote, 2024), and conservation studies (see Griffith et al 1989). We can design a cross-sectional study, which gathers information at one time or the study can be longitudinal whereby information is gathered over a period of time. If the purpose of the survey is to determine association between two (or more) variables, then either cross-sectional or longitudinal approaches will do. However, if cause-effect hypotheses are the purpose, then longitudinal (eg, prospective cohort) approaches would be better (cf discussion in Chapter 5 – Experimental design).

Survey basics

Steps to conduct a survey include

1. Identify and clarify the purpose of the survey. If the purpose is to find out how common something is, then this is descriptive. If we are interested in why something has occurred, then this is an analytic survey.
2. Define the reference population. It is essential that you know which group the survey applies to. For example, if one wishes to study the opinions of undergraduates at your university, then postgraduate students cannot be included in the sample as they are not part of the reference population.
3. Design sampling method and determine sample size. Sampling needs to be done to obtain unbiased sample from representative population. If the size of the population is known, then a target of 10% might be the relevant sample size, and a procedure should be taken to obtain a simple random sample. A measure of the success of a survey sample design is the size of the response rate, defined as the ratio of surveys returned divided by the number of surveys distributed.
4. What information is needed? Care needs to be taken to make sure that the questions asked actually yield the desired information. for example, if the questionnaire is long, there may be a tendency for some to skim or skip questions. If too much information is requested, this may lower the response rate.
5. How will the information be collected? Types and format of questions (closed or open ended). A phone survey? Written survey dropped off as a mailer? Interview?
6. In thinking about the data to be collected from the questions, you also need some scoring system. Will a dichotomous response (eg, True/False, or Yes/No) be adequate, or would a Likert-like scale be more appropriate? When scaling responses, one needs to also be concerned with floor and ceiling effects.
7. Collect the data. What protocol will be employed to achieve a high response rate with unbiased responses?
8. Analyze the data. Often chi-square contingency table or the related logistic regression would be appropriate.

Bias sources in survey research

When a statistical estimator consistently under or overestimates the true value, this is called statistical bias, which was introduced and discussed in Chapter 5.3. The potential for bias responses to survey questions is an important constraint on the applicability of the survey.  An example, it is well-known that adults tend to overestimate their height, but underestimate their weight.

In survey research, different classes of bias have been defined.

• Information bias occurs when trends are present in the measurement of the response, (1) recall bias and (2) observer bias.
• Recall bias is when a difference occurs because some people are much more likely to remember and event than others.
• Observer bias can be as a result of differences between different observers. If different persons conduct interviews, then it is important that all observers use a standardized method of collecting data.
• A reality of survey is that not all targets of the survey will answer the questions.  This may result in bias.  Non-response bias is the situation when those who respond to a questionnaire, the responders, differ in some way from those who don’t, the non-responders.
• Selection bias results when the sample group you have chosen is not representative of the population you want to generalize your results to. Random sampling can help to minimize this from happening in your survey, but a stratified sampling approach is needed to avoid missing representation, eg, economic groups, ethnicity.

How to ask questions?

The goal is to maximize the number of people who respond to the survey while maintaining accuracy and relevance of the responses. This is accomplished by asking the right questions in the right manner, but also by how the questionnaire is presented and administered. To get accurate answers, one should include additional questions to check the consistency of the responses provided by a person. For example, if the study is about smoking, you could ask either of two questions (or both…?):

Question 1. Do you smoke tobacco cigarettes? Yes / No

Question 2. How many cigarettes did you smoke yesterday?
0 …. 1 – 10 …. 11 – 20 …. 21 +

Note that these questions are CLOSED — the responder must answer using the answers provided rather than making something up. This has the advantage of restricting the possible answers and allows you to test specific hypotheses. An OPEN question might be

Question 3. Do you smoke a lot of cigarettes in a day?

As you can imagine, you would expect to get a variety of interpretations of this question, which limits your ability to analyze test the hypothesis. It would be a poor question to use.

Closed or forced format questions can take on a variety of styles.

Question 4. What is your favorite soft drink? (select one answer only)
__ water
__ cola
__ ice tea
__ fruit juice
__ no preference

Note that Question 4 gives the responder a choice among categories. For another example,

Question 5. Biostatistics is my favorite subject
___Strongly disagree ___Disagree ___Undecided ___Agree ___Strongly Agree

And still another example might employ ranking, requesting the responder to rank from 1 to 8 their favorite subjects from a list of academic topics.

All of these closed format responses can be easily converted for analyses by contingency table or other nonparametric statistics.

Some other general tips in writing a survey.

• Keep sentences simple and short.
• Ask for only one piece of information at a time.
• Ask precise questions.
• Start the survey with the question(s) most relevant to the subject of the study.
• Avoid asking personal questions at the start.
• Write some questions then conduct a pilot study to test for the efficacy of the survey

Questions

1. Write up three learning outcomes for this page. Hint: Point your favorite generative AI to this page and ask for help.

2. Be able to define and distinguish with examples, reference population, sample population, source population, and study population.

Quiz Chapter 20.4

Conducting surveys

Start Quiz

Question

Confirm

✕

✓

Your answer:

Correct answer:

Next

Please wait..

↺

You got {{SCORE_CORRECT}} out of {{SCORE_TOTAL}}

Your Answers

Suggested readings

Statistics Canada: a site with lots of material about survey methods.

United Nations Development Programme (UNDP) and University of Oxford. The Peoples’ Climate Vote 2024. Published June 2024.

White, B. A. A., Johnson, J., Arroliga, A. C., & Couchman, G. (2020). Ad hoc teams and telemedicine during COVID-19. Proceedings (Baylor University. Medical Center), 33(4), 696–698.

Wikipedia: Statistical survey

Xu, Y., Feeney, M. P., Surface, M., Novak, D., Troche, M. S., Beck, J. C., & Alcalay, R. N. (2022). Attitudes Toward Telehealth Services Among People Living With Parkinson’s Disease: A Survey Study. Movement Disorders, 37(6), 1289–1294.

Chapter 20 contents

• Additional topics
• Area under the curve
• Peak detection
• Baseline correction
• Surveys
• Time series
• Dimensional analysis
• Estimating population size
• Diversity indexes
• Survival analysis
• Growth equations and dose response calculations
• Plot a Newick tree
• Phylogenetically independent contrasts
• How to get the distances from a distance tree
• Binary classification
• Meta-analysis

/MD

5.1 – Experiments

How to put an experiment together

With the background behind us, we outline a designed experiment. Readers may wish to review concepts presented previously, in

• Chapter 2.4: Experimental design and rise of statistics in medical research.
• Chapter 3.4 – Estimating parameters
• Chapter 3.5 – Statistics of error

Experiments have the following elements

1. Define a reference population.
   1. All patients with similar angina symptoms — shortness of breath, chest pain, dizziness — but who have not suffered a heart attack, or myocardial infarction (Wu et al 2017);
   2. All adult frogs in a pond near industrial waste runoff (eg, Sowers et al 2009).
2. Define sample unit from the reference population (eg, as many patients as can be screened by the physician are recruited; all frogs visible to the biologist are captured).
3. Design sampling scheme from population (eg, random sampling vs. convenience/haphazard sampling).
4. Agree on the primary outcome or endpoint to be measured and whether additional secondary outcomes are to be observed and collected.
   1. For the patients with similar symptoms, we follow
   2. For the frogs, we count adults with developmental malformations.
5. Separate the sample into groups so that comparisons can be made
   1. We may be tempted to provide the same treatment and follow the group for improvement — that is the clinically valid approach; to be an experiment, we would use random assignment of individuals to one of two groups — half will receive a placebo, half will receive a beta blocker;,
   2. The frog example doesn’t follow this scheme — the event has already occured, it’s an observational study.
6. Devise a way to exclude or distinguish from the possible explanations, or alternative hypotheses.
   1. eg, for the physician, she will record how many of her patients fail to respond to treatment;
   2. for the biologist, noting presence or absence of parasites between the normal and abnormal frog defines the groups).

A basic experimental design looks like this.

1. One treatment, with two levels (eg, a control group and an experimental group)
2. A collection of individuals recruited from a defined population, esp. by random sampling.
3. Random assignment of multiple individuals to one of the two treatment levels.
4. The treatments are applied to each individual in the study.
5. A measure of response (the primary outcome) and additional features (secondary outcomes) are recorded for each individual.

Of course, there are other considerations, and by now, students reading this text should be able to identify what is missing from the list. (See quiz questions.)

Contrast with design of observational studies

Importantly, for observational studies, no matter how sophisticated the equipment used to measure the outcome variable(s), key steps outlined above are missing. Researchers conducting observational studies do not control allocation of subjects to treatment groups (steps 4 and 3 in the two above lists, respectively). Instead, they may use case-control approaches, where individuals with (case) and without (control) the condition (eg, lung cancer) are compared with respect to some candidate risk factor (eg, smoking). Both case and control groups likely have members who smoke, but if there is an association between smoking and lung cancer, then more smokers will be in the case group compared to the control group.

A cohort study, also a form of observational study, is similar to case-control except that the outcome status is not known. A cohort study includes, in our example, smokers and non smokers who share other characteristics: age, medical history, etc. The other kind of design you will see is the cross-sectional study. Cross-sectional studies are descriptive studies, and, therefore also observational. Primary outcomes along with additional characteristics and outcomes are measured for a representative subsample, or perhaps even an entire population. Cross-sectional studies are used to absolute and relative risk rates. In ecology and evolutionary biology cross sectional studies are common, eg, comparisons of species for metabolic rate (Darveau et al 2005) or life history traits (Jennings et al 1998), and specialized statistical approaches that incorporate phylogenetic information about the species are now the hallmark of these kinds of studies.

A word on outcomes of an experiment. Experiments should be designed to address an important question. The outcomes the researcher measures should be directly related to the important question. Thus, in the design of clinical trials, researchers distinguish between primary and secondary outcomes or endpoints. In educational research the primary question is whether or not students exposed to different teaching styles (eg, lecture-style or active-learning approaches) score higher on an knowledge-content exams. The primary outcome would be the scores on the exams; many possible secondary outcomes might be collected including student’s attitudes towards the subject or perceptions on how much they have learned.

About Primary and Secondary Outcomes.

Consider a evolutionary physiology project on species differences and effects of body size on burst running speed of terrestrial lizards (eg, Bauwens et al 1995; Dohm et al 1998; Labonte et al 2024). Our independent variables, the things we believe influence running performance — and therefore we wish to control for — species, body size, core body temperature, body condition, race quality (see cited papers for these and other examples).

What do we measure? What are the likely primary vs. secondary outcomes for a this kind of project?

Primary outcomes are the direct, definitive measures of maximal locomotor performance. Examples may include

• Maximum Burst Speed: The absolute highest velocity achieved by the lizard over a short distance or time interval (eg, across a 0.5-meter interval).
• Maximum Acceleration: The highest rate of speed change during the start of the burst or during a sudden sprint increase.
• Stride Length: The physical distance covered during a single full stride cycle at maximum speed.
• Stride Frequency: The number of strides taken per second at maximum speed.

And what other measures may be of interest — secondary outcomes — those that may help explain why or how the primary performance metrics change across the independent variables. To list a few,

• Stance and Swing Duration: The individual time durations spent with feet on the ground (stance) vs. in the air (swing).
• Gait Transition Thresholds: The speed at which a lizard shifts from a walk/trot to a high-speed gallop or bipedal running (if applicable to the species).
• Fatigue Rate: The decline in burst speed across successive trials or over a sustained treadmill run.
• Trial-to-Trial Repeatability: The statistical consistency (intraclass correlation coefficient) of a lizard’s performance across multiple bursts.

In a prospective cohort study, we may track participants over time to observe how baseline factors predict the development or worsening of type 2 diabetes (eg, Gao et al 2026). Progression over a two year followup would be measured by incident clinical events or the longitudinal rate of change in biomarkers. Possible primary outcomes are

• Glycated Hemoglobin (HbA1c): A biomarker reflecting average blood glucose levels over the past 2 to 3 months (glycemic control).
• Fasting Plasma Glucose (FPG): Glucose concentration in the blood after an overnight fast (acute baseline glycemic regulation).
• T2D Complication Stage: A clinical score or categorization indicating the presence and severity of microvascular or macrovascular damage.

Secondary outcomes may include

• Fasting Lipid Profile
• C-Reactive Protein
• Body Mass Index
• Estimated Glomerular Filtration Rate

If one or more primary outcomes are significant, this reveals whether an intervention or exposure (like a vegan diet or income level) changes the clinical course of Type 2 Diabetes. Secondary outcomes are critical for explaining how and why it happens. Secondary outcomes provide mechanistic context.

Some caution is warranted — imagine if, after conducting the study, we find diet, age, or income did not statistically affect your primary outcome, but did significantly alter a secondary outcome (eg, improved lipid profiles). What to report? We state in your Abstract and Discussion that the study failed to meet its primary objective, and that any significant secondary findings are exploratory. We cannot claim that a diet “successfully manages diabetes progression” based solely on a secondary marker (cf discussion and suggsted guidelines in Butcher et al 2022).

Example of an experiment.

We’ll work through a familiar example. The researcher is given the task to design a study to test the efficacy in reducing tension headache symptoms by a new pain reliever. There are many possible outcomes: blurry vision, duration, frequency, nausea, need for and response to pain medication, and level of severity (Mayo Clinic). The drug is packaged in a capsule and a placebo is designed that contains all ingredients except the new drug. Forty subjects with headaches are randomly selected from a population of headache sufferers. All forty subjects sign the consent form and agree to be part of the study. The subjects also are informed that while they are participating in the research study on a new pain reliever, each subject has a 50-50 chance of receiving a placebo and not the new drug. The researcher then randomly assigns twenty of the subjects to receive the drug treatment and twenty to receive the placebo, places either a placebo pill of the treatment pill into a numbered envelope and gives the envelopes to a research partner. The partner then gives the envelopes to the patients. Both patients and the research partner are kept ignorant of the assignment to treatment.

We can summarize this most basic experimental design in a table, Table 1.

Table 1. Simple formulation of a 2X2 experiment (aka 2×2 contingency table).

where N is the total number of subjects, a is number of subjects who DID receive the treatment AND got better, c is number of subjects who DID NOT receive the treatment, but DID get better, b is number of subjects who DID receive the treatment and DID NOT get better, and d is number of subjects who DID NOT receive the treatment and DID NOT get better.

And our basic expectation is that we are testing whether or not treatment levels were associated with subjects “getting better” as measured on some scale.

One possible result of the experiment, although unlikely, all of the negative outcomes are found in the group that did not receive the treatment (Table 2).

Table 2. One possible outcome of our 2X2 experiment.

Results of an experiment probably won’t be as clear as in Table 2. Treatments may be effective, but not everyone benefits. Thus, results like Table 3 may be more typical.

Table 3. A more likely outcome of our 2X2 experiment.

Must a control group always be included in an experiment?

To be an experiment, yes. However, it is not true that the control treatment must only be applied to separate, individual subjects. For example, a common experimental design would be repeated measures of an outcome, like behavior (eg, feeding response) before and after an intervention is applied. Thus, the individual is its own control or reference point. For example, Dohm et al (2008) measured feeding and locomotor behavior of adult toads (Rhinella marina, formerly Bufo marinus) prior to and after ozone exposure. We concluded that a single 4-h exposure to O₃ depressed toad feeding behavior, but had little effect on voluntary locomotor behavior. Typically, the design randomly assigns individuals to the  treatments — some individuals get the exposure first, then the control exposure second; others, again selected at random, receive the control exposure first, then the active exposure.

The within subjects design is an important method in biological research; these designs offer control for individual differences as confounding variables and require fewer participants compared to separate control group designs. This design is effective provided there is little risk of carryover effects; if the effects of a previous condition linger, then it may influence a participant’s response in subsequent conditions, confounding the results. For our study (Dohm et al 2008) we used a modified within-subjects design by employing a split plot design. We return to within-subjects designs in Chapter 14.4 – Randomized block design.

Questions

1. When I updated this page in 2020, we were in our fifth month of the coronavirus pandemic of 2019-2020. Daily it seemed, we heard news about “promising coronavirus treatments,” but no study had at the time been published that would meets our considerations for a proper experiment. However, on May 1, 2020, the FDA issued an emergency use authorization for use of the antiviral drug remdesivir (Gilead), based on early results of clinical trials reported 29 April in The Lancet (Wang et al 2020). Briefly, their study included 158 treated with does of the antiviral drug and 79 provided with a placebo control. Both groups were treated otherwise the same. Improvement over 28 days was recorded: 62 improved with Placebo and 133 improved receiving dose of remdesivir.
   1. Using the background described on this page, list the information needed to design a proper experiment. Using your list, review the work described in The Lancet article and check for evidence that the trial meets these requirements.
   2. Create a 2×2 table with the described results from The Lancet article.
2. Consider our hazel tea and copper solution experiment described in Chapter 3. The outcome variables (Tail length, tail percentage, Olive moment) are quantitative, not categorical. Create a table to display the experimental design.
3. For the migraine example, identify elements of the design that conform to the randomized control trial design described in Chapter 2.4.

Quiz Chapter 5.1

Experiments

Start Quiz

Question

✕

✓

Your answer:

Correct answer:

Next

Please wait..

↺

You got {{SCORE_CORRECT}} out of {{SCORE_TOTAL}}

Your Answers

Chapter 5 contents

• The basics explained
• Experiments
• Experimental and Sampling units
• Replication, Bias, and Nuisance Variables
• Clinical trials
• Importance of randomization in experimental design
• Sampling from Populations
• References and suggested readings