Daily Research News Online no. 2520 - Do Online Surveys Get More Extreme Answers?

Do Online Surveys Get More Extreme Answers?

August 1 2003

Responses to a survey conducted online may suggest sharper opinions than a paper equivalent, with more high and low ratings and strong agreements / disagreements, as opposed to fence-sitting or moderate answers, according to a paper at the recent ORC International 8th annual employee research conference.

Mary Dalgleish from the Department of Work and Pensions explained to the conference how giving staff the option to complete the annual employee survey on-line has resulted in more extreme responses. The paper questionnaire was distributed to all 145,000 staff in more than 2,000 locations. In addition employee research specialist ORC International developed an on-line questionnaire solution to give staff the opportunity to provide their feedback on-line. This also enabled DWP to assess whether the survey methodology used - ie, paper versus on-line - influenced the way respondents completed their questionnaire.

The survey contained 117 questions, and in 50 of these there were significant differences between answers from the on-line and the paper group. Respondents to the on-line questionnaire appeared to be more likely to tick the 'strongly agree' response option than the 'agree' option, while respondents to paper-based questionnaires were more likely to respond using the 'agree' option than the 'strongly agree' option. There was no significant difference in employees with 'neutral' responses - the significant difference was only apparent when examining the 'positive' and 'negative' responses.

The likelihood of entering a comment and the length of each comment were also compared between the two survey methodologies. It was found that respondents were just as likely to add comments in a paper questionnaire as they were in an on-line questionnaire, but in on-line questionnaires the average comment was longer. Two possible explanations are suggested for this:

there is a larger amount of space for comments in an on-line questionnaire, compared with the limiting box given in the paper version
people may find it easier to type on a keyboard than to write with pen and paper.

DWP are embarking on their next employee survey, which employees will be able to complete on-line. Mary Dalgleish foresees many advantages in moving the staff survey on-line: 'speed and simplicity of completion, reduced costs for printing, distribution and data entry, and improved response rates through monitoring, in addition to speedier production and distribution of results. A paper-based survey will still be available, but we will definitely be encouraging staff to complete the next staff survey on-line'.

All articles 2006-23 written and edited by Mel Crowther and/or Nick Thomas unless otherwise stated.