Fundamentals of Survey Design and Analysis
From ICE Primer: A Tobacco Control Research Methodology Primer
In the field of tobacco control research, surveys are undertaken for purposes of understanding behaviour and testing and evaluating interventions. The research questions of the survey will dictate the target population, and whether the survey will be cross-sectional one-time, cross-sectional but periodic, or longitudinal. The sampling frame and logistic considerations will determine the study population. The funding available for the survey and the conditions for field operations will determine the sample size and sampling design. In designing the survey, it is important to consider these aspects from the beginning.
The target population is the population to which we would like to be able to generalize the results of the survey. The study population is the one from which the sample will actually be drawn. The sampling frame is a list or map which covers the units of the study population.
The first steps
In the initial part of planning, the research questions are formulated broadly, and contact is made with collaborators, stakeholders and sponsors to estimate the available funding. A suitable sampling frame is found: [examples]. A rough budget is prepared, and a sample size justification analysis is carried out to determine whether the sample size which can be afforded will suffice to answer the key questions. This sample size justification must take into account the kind of sampling design proposed. If it turns out that the sample size is insufficient, and further resources are not available, the survey should be abandoned.
Sampling plan and protocols
If it is decided to go ahead, a detailed protocol and sampling plan should be worked out and supplied to several fieldwork organizations for quotes. It will be necessary at this stage to have a sense of how long an interview or self-administration of the questionnaire will take, so that the bidders can estimate the effort required to obtain responses.
Questionnaires
At the same time, the researchers and stakeholders should develop the questionnaires or other instruments. A reasonable general rule is that a questionnaire or interview script should be as short and simple as possible, while asking enough of the right kinds of questions to allow the research issues to be addressed. If the study population consists of several groups, such as current smokers, never smokers, and former smokers, then several questionnaires and a routing screener will have to be developed.
The best way to develop the instrument is through a series of meetings, either face to face or by telephone. Between meetings, participants propose additional questions, deletions and edits.
Testing
If the questionnaires are programmed, the programming should be checked carefully through a systematic process in which the tester pretends to be various kinds of respondent. For a survey being conducted for the first time, the protocols and instruments should be tested in a pilot survey. Interviewers should be debriefed carefully. Cognitive testing may be used to assess the face validity of the instruments.
Data collection and monitoring
Quality control measures are necessary throughout the data collection process. The fieldwork organization should provide weekly reports on the progress of fieldwork, and should provide the first week’s worth of records so that the logic of the program can again be checked.
