Can we trust the results?
No-one can guarantee that every answer given by every respondent in a survey is completely accurate and honest. However, we can go a long way to improving our confidence in the figures by taking care over each aspect of the process:
- Questionnaire design
We strive to ask questions in a clear and unbiased way, concentrating on what has actually happened in pupils' and students' lives. So, we ask questions like 'what did you eat or drink before lessons this morning?', and not 'what do you usually eat for breakfast?' (which, in just seven words, makes three different assumptions that might not be correct!). Questions are scrutinised before use, not just by SHEU staff, but also by expert groups from local authorities, such as teachers, advisers and public health professionals.
- Piloting
Questions are tested with smaller groups before use across our services. Once we have evidence that the answers from small-scale work are meaningful and accurate, we can support more wide-scale use.
- Post-questionnaire interviews
We have regularly conducted interviews with pupils and students following completion of a pilot questionnaire. These interviews can let us know that pupils and students are willing and able to find and choose an answer that properly reflects their situation.
- Introduction to the pupils and students
Pupils and students must feel that the exercise has some value if they are to cooperate. We produce extensive guidance for schools explaining ways of introducing the purpose and nature of the survey to pupils and students.
- Feedback from schools about data collection
We ask for feedback from every single class that completes one of our surveys. Staff who supervise data collection are alert to problems that arise and provide important feedback on questions that could be worded better.
- Response checking
Staff at SHEU inspect every set of responses coming into the Unit. Our experienced data preparation staff are sensitive to any odd-looking responses and we will respond, perhaps by excluding an individual's questionnaire (if they are not taking it seriously) or even by excluding everyone's answers if the responses from one question are not consistent with other trusted information.
- Internal reliability
Questions in one part of the questionnaire can be related to another; for example, a question on recent spending might report spending on sweets, which can be compared with a question many pages away about eating habits. When we do these comparisons, the match is always very good.
- External reliability
Figures from one survey can be compared with results from another survey. For example, we know a lot about patterns of cigarette smoking among young people over the last 30 years from Government surveys. The questions they use have been double-checked, not just using the approaches described above, but also by saliva tests that show if someone is smoking. SHEU figures show very much the same levels and also the same rises and falls over the years as are seen in the Government reports.
- Feedback from schools and authorities
Once results are circulated, we invite feedback from schools and authorities by asking directly if they have confidence that the figures accurately reflect their young people. They may have other supporting sources of information that we aren't aware of, or a sense that the picture they see is about right.
For more information, see the SHEU publications from the Young People into… series, or contact Dr.David Regis, the Research Manager at SHEU.