Questionnaire design and testing

Questionnaires are a potential source of bias and must be formulated very carefully to guarantee the quality of the data collected. From survey design to questionnaire testing, it is important to avoid errors in respondent comprehension and interpretation and to avoid influencing or offending respondents.

Questionnaire design

Designing a questionnaire requires detailed knowledge not only of the survey topic and the target population, but also of the technical potential of the chosen data collection method. Researchers need to be familiar with the existing scientific literature in their field and with any other surveys conducted in the same area. Reliable knowledge of the language, the ordinary or lay terms used to designate aspects of the situation under investigation, as well as of the cultural and cognitive skills needed to answer the questions, is also indispensable.

That knowledge is especially important for surveys on sensitive subjects (sexuality, violence, drug use) or involving specific populations (of particular cultural origins or social backgrounds). It is also needed for general population surveys which, by definition, encompass all categories of persons.

Generally speaking, questions are most effective when they are precise and clearly contextualized, short and formulated in simple language, and when the terms used cannot be misinterpreted. It is also extremely important not to ask leading questions, i.e. questions that may encourage respondents to choose a particular response or to answer in a particular way. These constraints may be difficult to reconcile, which is why questions must be tested and researchers should plan to speak with interviewers—respondents too, if possible—throughout the questionnaire preparation phase.

Questionnaire testing and pilot surveys

Survey protocol, data collection and questionnaire testing are fundamental components of the data production process. Testing provides a means to check whether the questionnaire is valid in itself (form; question formulation, including translation if relevant; overall structure and transitions between questions and/or question sets), and to validate the data collection protocol (interviewer recruitment and training; interpreter training where relevant; the tasks of contacting respondents and presenting the survey to them; tools used for follow-up in the field, etc.). This is an indispensable stage that must be planned for and adequately budgeted.

During the preliminary stages of questionnaire construction, it is extremely useful for members of the research project team to test the questionnaire by administering it among themselves.

Further along in the questionnaire formalization process, the interviewers must be brought in. There are various testing and validation methods, and several tests may be necessary, depending on the types of modification made (elimination or reformulation of questions, changes to the list of response categories or question order, etc.).

The tests are also an opportunity to develop, clarify or improve interviewer training, another crucial factor in data quality. The pilot survey, a sort of “dress rehearsal,” the last step before the full-scale survey is launched, is generally conducted at a larger scale. It serves to finalize the protocol and resolve any remaining problems.

While pilot and even full-scale surveys may be outsourced, the different testing phases are usually handled internally at INED in order to develop and fine-tune the overall survey protocol. These preliminary phases take time. In fact, they are the most time-consuming component of survey preparation. They may take from one to two years, depending on the project.