Can we trust the results?

No-one can guarantee that every answer given by every respondent in a survey is completely accurate and honest.  However, we can go a long way to improving our confidence in the figures by taking care over each aspect of the process:

  • Questionnaire design

We strive to ask questions in a clear and unbiased way, concentrating on what has actually happened in pupils' and students' lives.  So, we ask questions like 'what did you eat or drink before lessons this morning?', and not 'what do you usually eat for breakfast?' (which, in just seven words, makes three different assumptions that might not be correct!).  Questions are scrutinised before use, not just by SHEU staff, but also by expert groups from local authorities, such as teachers, advisers and public health professionals.

  • Piloting

Questions are tested with smaller groups before use across our services.  Once we have evidence that the answers from small-scale work are meaningful and accurate, we can support more wide-scale use.

  • Post-questionnaire interviews

We have regularly conducted interviews with pupils and students following completion of a pilot questionnaire.  These interviews can let us know that pupils and students are willing and able to find and choose an answer that properly reflects their situation.

  • Introduction to the pupils and students

Pupils and students must feel that the exercise has some value if they are to cooperate.  We produce extensive guidance for schools explaining ways of introducing the purpose and nature of the survey to pupils and students. 

  • Feedback from schools about data collection

We ask for feedback from every single class that completes one of our surveys.  Staff who supervise data collection are alert to problems that arise and provide important feedback on questions that could be worded better. 

  • Response checking

Staff at SHEU inspect every set of responses coming into the Unit.  Our experienced data preparation staff are sensitive to any odd-looking responses and we will respond, perhaps by excluding an individual's questionnaire (if they are not taking it seriously) or even by excluding everyone's answers if the responses from one question are not consistent with other trusted information.

  • Internal reliability

Questions in one part of the questionnaire can be related to another; for example, a question on recent spending might report spending on sweets, which can be compared with a question many pages away about eating habits.  When we do these comparisons, the match is always very good.

  • External reliability

Figures from one survey can be compared with results from another survey.  For example, we know a lot about patterns of cigarette smoking among young people over the last 30 years from Government surveys.  The questions they use have been double-checked, not just using the approaches described above, but also by saliva tests that show if someone is smoking.  SHEU figures show very much the same levels and also the same rises and falls over the years as are seen in the Government reports.

  • Feedback from schools and authorities

Once results are circulated, we invite feedback from schools and authorities by asking directly if they have confidence that the figures accurately reflect their young people.  They may have other supporting sources of information that we aren't aware of, or a sense that the picture they see is about right.

 

For more information, see the SHEU publications from the Young People into… series, or contact Dr.David Regis, the Research Manager at SHEU.

Topics: 

Comments about SHEU

"Thank you from my staff to you and all your staff. The speed of 'turn-around' of the questionnaires is outstanding in anyone's terms." Headteacher

Headteacher

"I would like to say that this survey was very useful and made me realise things about PE and health that I had never realised before......Food at school is groovy, especially if your school does Jamie Olivers School Dinners. Viva apples and thanks for the survey." Female pupil, 13 yrs old

Female pupil, 13yrs

"...the only question to cause a problem was 'has everyone got a pen?" Supervisor's notes following a school survey

Supervisor's Notes

"Just to say a huge thank you for all your efforts in helping us with the Health survey amongst pupils. It has provided us with significant data which will be used across the school to help us improve. It helped us to obtain a healthy schools standard as well. I hope we can make this an annual feature as we can track the changing health of our pupils." Headteacher

Headteacher

“(The SHEU survey) was very, very useful. It gave us reassurance we weren’t missing a trick. For example not many pupils in the sample year groups were taking illegal drugs, which re-enforced our opinions. But the survey also raised issues and flagged some things up. We discovered that some of our girls weren’t eating enough – the percentage of girls in our school not eating lunch the day before the survey was higher than the county average. There were other concerns too, specifically around cigarettes, alcohol and attendance.
The school used this data and took a number of actions to address it. More female peer mentors were put in place and the school asked NEXUS (the Extended Schools service) for help, so they developed a programme for girls which addressed their eating patterns, healthy eating, sex education and self-esteem issues.
We ran an anti-bullying group for Year 9 as a preventative measure, based upon data provided by our current Year 10 students.
The travel data revealed that a high number of pupils took the car to school so we involved the BIKE-IT scheme who ran assemblies, brought in their bikes (including one with a pedal-powered smoothie maker!), and raised awareness of health and green issues.
The information about how happy the students were with their lives raised some concerns as far fewer girls were as happy as the boys, so work was done around developing aspirations, role-models and self-esteem."        
 

Deputy Head, Secondary School

"Thanks for presenting the survey to local schools this morning, I just wanted to thank you for such interesting and thought-provoking information.  
I’m really glad we were able to take part - the information (particularly headline data and differences) will support us to have some really interesting questions with the Year group as a whole about the sense they’re making of this; what they think it might mean in terms of changes they might make, and what they need to support them in this."

Deputy Headteacher

"Please send an additional copy of our report - it is the most requested and borrowed item in the whole library." Health Promotion Unit

Health Promotion Unit

"The system works and I find quite a lot of it useful in my work. I've also recommended it to others."

Teenage Pregnancy Manager

"The Schools Health Education Unit is the jewel in the crown of the Health Education Authority." Major General Sir John Acland 1928 - 2006

Major General Sir John Acland

"SHEU data proved the best source of the kind of information we were looking for (...) to provide research support to the National Healthy Schools Programme." 

Department of Health