SHEU paper and online questionnaires for young people

 

Paper and online surveys completed by pupils in schools.
Do the same questions perform differently?

 

School and College paper-based surveys  

The Schools Health Education Unit [SHEU] is probably the oldest survey service in England dedicated to collecting data about young people’s healthy lifestyles. Millions of questionnaires have been completed by pupils.

     
SHEU school and college online surveys  

SHEU started school-based paper surveys in 1977 and launched its online survey service in 2004. At that time we had the idea that nearly all the survey work would be online within a few years.

While the proportion of work online has increased, some schools and colleges find that paper surveys are still more convenient for them. We have found that our surveys give robust data via online or paper-based questionnaires. Often the final decision about which survey method to use can depend on the availability and ease of access of computers in schools and colleges

Do the same questions perform differently?

The following is from a small study we carried out some time ago. We had just completed the second wave of survey work in an Authority. The idea was originally that the survey be conducted on our online survey website but in the end about half of the schools preferred to complete the exercise on paper.  This arrangement of 17 schools working on paper and 16 online allowed us to perform a ‘natural experiment’ to compare whether the same questions performed differently in the different media. We generated a whole set of tables that showed the paper and online results side by side.  The figures showed good agreement, we thought: then, we decided, very good... actually, these are spookily similar!

 

Paper

Online

All

No

72%

71%

72%

Don't know

3%

4%

4%

Yes

24%

25%

25%

Valid Responses

%

100%

100%

100%

Count

1720

1779

3499

Total Sample (Count)

1760

1867

3627
 
Not all the tables are that close but you get the idea. This gives us a lot of confidence that the questionnaire is robust across the different methods of delivery.  Whatever headaches it gives our programming colleagues in explaining to the different sets of computers how to combine the information, the pupils seem to take the questionnaire in their stride.
In 2014 Dr David Regis wrote, “Ten years ago, I thought we would be doing everything online by now! But I fairly often get told something like:
"There's 2 year groups to get through, 6 classes each, so we need to book the IT suite for 12 slots, and it's already booked up, and the classes are 30 but the IT suite has just 25 computers, 24 of which work... Just send us a parcel of questionnaire booklets, and we can do it all in one morning!"
“Also, the best-laid plans of a school can oft gang agley because of power failure, sulky networks, flooding, winter vomiting, or ...”
“SHEU will continue to offer both paper questionnaires and our online survey service, and in case of events will be able to switch to the other approach for a school at short notice.”

One example of research about school-based health questionnaires:

Web-based questionnaires and paper-based questionnaires, used as part of a school-based study of young people's health-related behavior, are compared in terms of (a) completion rates and (b) data contents. It is concluded that, on the basis of the quantitative data from this survey, there is little evidence of a mode effect linked to web-based questionnaires.
Web-Based Questionnaires and the Mode Effect: An Evaluation Based on Completion Rates and Data Contents of Near-Identical Questionnaires Delivered in Different Modes. Social Science Computer Review Summer 2006 vol. 24 no. 2 246-254
http://jwolf-ra.wdfiles.com/local--files/mode-effects/WebBasesQuestandModeEffect2006.pdf