Handbookof WebSurveys
SecondEdition
SilviaBiffignandi
UniversityofBergamo
Italy
JelkeBethlehem
FacultyofSocialandBehavioralSciences, InstituteofPoliticalScience
LeidenUniversity TheNetherlands
contact: biffisil@teletu.it j.g.bethlehem@fsw.leidenuniv.nl
website: www.web-survey-handbook.com/
Thissecondeditionfirstpublished2021 ©2021byJohnWiley&Sons,Inc.
EditionHistory
JohnWileyandSons,Inc.(1e,2012)
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recording,orotherwise, exceptaspermittedbylaw.Adviceonhowtoobtainpermissiontoreusematerialfromthistitleisavailableat http://www.wiley.com/go/permissions.
TherightofSilviaBiffignandiandJelkeBethlehemtobeidentifiedastheauthorsofthisworkhasbeen assertedinaccordancewithlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice 111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproductsvisit usatwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthat appearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakeno representationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisworkand specificallydisclaimallwarranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantabilityor fitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysalesrepresentatives,writtensales materialsorpromotionalstatementsforthiswork.Thefactthatanorganization,website,orproductis referredtointhisworkasacitationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthe publisherandauthorsendorsetheinformationorservicestheorganization,website,orproductmayprovide orrecommendationsitmaymake.Thisworkissoldwiththeunderstandingthatthepublisherisnot engagedinrenderingprofessionalservices.Theadviceandstrategiescontainedhereinmaynotbesuitablefor yoursituation.Youshouldconsultwithaspecialistwhereappropriate.Further,readersshouldbeawarethat websiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthisworkwaswrittenandwhen itisread.Neitherthepublishernorauthorsshallbeliableforanylossofprofitoranyothercommercial damages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationData
Names:Biffignandi,Silvia,author.|Bethlehem,JelkeG.,author.|John Wiley&Sons,Inc.,publisher.
Title:Handbookofwebsurveys/SilviaBiffignandi, UniversityofBergamo,Italy [and]JelkeBethlehem,LeidenUniversity,FacultyofSocialand BehavioralSciences,InstituteofPoliticalScience,TheNetherlands.
Description:Secondedition.|Hoboken,NJ:Wiley,2021.| Includesbibliographicalreferencesandindex.
Identifiers:LCCN2020028687(print)|LCCN2020028688(ebook)|ISBN 9781119371687(hardback)|ISBN9781119371694(adobepdf)|ISBN 9781119764496(epub)
Subjects:LCSH:Internetsurveys.|Surveys–Methodology.
Classification:LCCHM538.B552021(print)|LCCHM538(ebook)|DDC 001.4/33–dc23
LCrecordavailableathttps://lccn.loc.gov/2020028687
LCebookrecordavailableathttps://lccn.loc.gov/2020028688
CoverDesign:Wiley
CoverImages:©Adisa/GettyImages
Setin10.5/12ptAGaramondbySPiGlobal,Pondicherry,India
10987654321
1THEROADTOWEBSURVEYS1
1.1 Introduction,1
1.2 Theory,2
1.2.1 TheEverlastingDemandforStatisticalInformation,2
1.2.2 TraditionalDataCollection,8
1.2.3 TheEraofComputer-AssistedInterviewing,11
1.2.4 TheConquestoftheWeb,13
1.2.5 WebSurveysandOtherSources,23
1.2.6 HistoricSummary,28
1.2.7 Present-DayChallengesandOpportunities,28
1.2.8 ConclusionsfromModern-DayChallenges,30
1.2.9 ThrivingintheModern-DaySurveyWorld,30
1.3 Application,31
1.3.1 Blaise,31
1.4 Summary,39 KeyTerms,41 Exercises,42 References,44
2ABOUTWEBSURVEYS47
2.1 Introduction,47
2.2 Theory,50
2.2.1 TypicalSurveySituations,51
2.2.2 WhyOnlineDataCollection?,56
2.2.3 AreasofApplication,60
2.2.4 TrendsinWebSurveys,62
2.3 Application,64
2.4 Summary,68
KeyTerms,68 Exercises,69 References,71
3AFRAMEWORKFORSTEPSANDERRORS INWEBSURVEYS73
3.1 Introduction,73
3.2 Theory,75
3.3 Application,88
3.4 Summary,89 KeyTerms,90 Exercises,90 References,91
4SAMPLINGFORWEBSURVEYS93
4.1 Introduction,93
4.2 Theory,95
4.2.1 TargetPopulation,95
4.2.2 SamplingFrames,98
4.2.3 BasicConceptsofSampling,103
4.2.4 SimpleRandomSampling,106
4.2.5 DeterminingtheSampleSize,109
4.2.6 SomeOtherSamplingDesigns,112
4.2.7 EstimationProcedures,118
4.3 Application,123
4.4 Summary,128 KeyTerms,129 Exercises,130 References,131
5ERRORSINWEBSURVEYS133
5.1 Introduction,133
5.2 Theory,142
5.2.1 MeasurementErrors,142
5.2.2 Nonresponse,164
5.3 Application,174
5.3.1 TheSafetyMonitor,174
5.3.2 MeasurementErrors,175
5.3.3 Nonresponse,177
5.4 Summary,179
KeyTerms,180 Exercises,182 References,185
6WEBSURVEYSANDOTHERMODESOF DATACOLLECTION189
6.1 Introduction,189
6.1.1 ModesofDataCollection,189
6.1.2 TheChoiceoftheModesofDataCollection,190
6.2 Theory,194
6.2.1 Face-to-FaceSurveys,194
6.2.2 TelephoneSurveys,200
6.2.3 MailSurveys,206
6.2.4 WebSurveys,211
6.2.5 MobileWebSurveys,215
6.3 Application,222
6.4 Summary,230
KeyTerms,231
Exercises,233 References,235
7DESIGNINGAWEBSURVEY QUESTIONNAIRE237
7.1 Introduction,237
7.2 Theory,240
7.2.1 TheRoadMapTowardaWebQuestionnaire,240
7.2.2 TheLanguageofQuestions,249
7.2.3 BasicConceptsofVisualization,252
7.2.4 AnswersTypes(ResponseFormat),258
7.2.5 WebQuestionnairesandParadata,271
7.2.6 TrendsinWebQuestionnaireDesignand Visualization,278
7.3 Application,281
7.4 Summary,282
KeyTerms,283
Exercises,284 References,286
8ADAPTIVEANDRESPONSIVEDESIGN291
8.1 Introduction,291
8.2 Theory,294
8.2.1 Terminology,294
8.2.2 QualityandCostFunctions,298
8.2.3 StrategyAllocationandOptimization,301
8.3 Application,309
8.4 Summary,316 KeyTerms,316 Exercises,317 References,318
9MIXED-MODESURVEYS321
9.1 Introduction,321
9.2 TheTheory,326
9.2.1 WhatisMixed-Mode?,326
9.2.2 WhyMixed-Mode?,334
9.3 MethodologicalIssues,343
9.3.1 PreventingModeEffectsThroughQuestionnaire Design,346
9.3.2 HowtoMixModes?,350
9.3.3 HowtoComputeResponseRates?,354
9.3.4 AvoidingandAdjustingModeEffectsforInference,359
9.3.5 Mixed-ModebyBusinessesandHouseholds,370
9.4 Application,384
9.5 Summary,386 KeyTerms,388 Exercises,388 References,390
10THEPROBLEMOFUNDER-COVERAGE399
10.1 Introduction,399
10.2 Theory,405
10.2.1 TheInternetPopulation,405
10.2.2 ARandomSamplefromtheInternetPopulation,406
10.2.3 ReducingtheNon-CoverageBias,410
10.2.4 Mixed-ModeDataCollection,413
10.3 Application,414
10.4 Summary,417 KeyTerms,418 Exercises,419 References,421
11THEPROBLEMOFSELF-SELECTION423
11.1 Introduction,423
11.2 Theory,431
11.2.1 BasicSamplingTheory,431
11.2.2 ASelf-SelectionSamplefromtheInternetPopulation,434
11.2.3 ReducingtheSelf-SelectionBias,439
11.3 Applications,444
11.3.1 Application1:SimulatingSelf-SelectionPolls,444
11.3.2 Application2:SundayShoppinginAlphena/dRijn,448
11.4 Summary,451
KeyTerms,452 Exercises,453 References,455
12WEIGHTINGADJUSTMENTTECHNIQUES457
12.1 Introduction,457
12.2 Theory,463
12.2.1 TheConceptofRepresentativity,463
12.2.2 Post-Stratification,465
12.2.3 GeneralizedRegressionEstimation,477
12.2.4 RakingRatioEstimation,486
12.2.5 CalibrationEstimation,490
12.2.6 ConstrainingtheValuesofWeights,491
12.2.7 CorrectionUsingaReferenceSurvey,492
12.3 Application,500
12.4 Summary,506
KeyTerms,508
Exercises,509 References,512
13USEOFRESPONSEPROPENSITIES513
13.1 Introduction,513
13.2 Theory,517
13.2.1 ASimpleRandomSampleWithNonresponse,517
13.2.2 ASelf-SelectionSample,520
13.2.3 TheResponsePropensityDefinition,521
13.2.4 ModelsforResponsePropensities,522
13.2.5 CorrectionMethodsBasedonResponsePropensities,529
13.3 Application,535
13.3.1 GenerationofthePopulation,536
13.3.2 GenerationofResponseProbabilities,537
13.3.3 GenerationoftheSample,537
13.3.4 ComputationofResponsePropensities,537
13.3.5 MatchingResponsePropensities,537
13.3.6 EstimationofPopulationCharacteristics,540
13.3.7 EvaluatingtheResults,541
13.3.8 ModelSensitivity,542
13.4 Summary,542 KeyTerms,543 Exercises,544 References,546
14WEBPANELS549
14.1 Introduction,549
14.2 Theory,555
14.2.1 Under-Coverage,555
14.2.2 Recruitment,557
14.2.3 Nonresponse,563
14.2.4 Representativity,577
14.2.5 WeightingAdjustmentforPanels,580
14.2.6 PanelMaintenance,582
14.3 Applications,585
14.3.1 Application1:TheWebPanelPilotofStatistics Netherlands,585
14.3.2 Application2:TheU.K.PollingDisaster,589
14.4 Summary,592 KeyTerms,593 Exercises,593 References,595
Preface
Thelast10yearshavewitnessedasignificantincreaseinInternetpenetration. Whatisparticularaboutthisgrowthisthatanumberofgenerationsarecurrently experiencingthecontemporaryandhighlytechnologicalenvironment.Social media,constantconnectivity,andon-demandentertainmentsareinnovationsthat Millennials(agedbetween23and38in2019)adaptedtoastheygrewup.For thosebornafter1996,theso-calledGenerationZ(agedbetween7and22in 2019),theseinnovationsaremostlytakenforgranted,havingbeenpartoftheir livesfromthebeginning.TheiPhonewaslaunchedin2007,whentheoldest membersofGenerationZwere10.Bythetimetheyareintheirteens,young AmericansaccesstheInternetmainlyviamobiledevices,Wi-Fi,andhighbandwidthcellularservices.Pre-Millennialgenerationsplayanimportantrole inthegeneralpopulation,butforthem,thisenvironmentbasedontechnological communicationisanewexperience.
Theimplicationsofsomepopulationsubgroupshavingadaptedtothetechnologicalenvironment(Millennialsandpre-Millennials)whileothershavelivedin this “alwayson” technologicalenvironmentalltheirlivesareofrelevanceforsurvey-basedresearch,particularlyinthecaseofwebsurveys.Thewaythatquestionnairesareadministeredundoubtedlyhasanimpactwhichdiffersaccordingto populationgroup.Furthermore,thebehavioroftherespondentswhileparticipatingdependsontheirdigitalexperience,theirgenerationalcharacteristics,andtheir attitudetowardtechnologyintheirlives.Therefore,surveys andinparticular webandmobilewebsurveys havetoadoptanumberofchangesintheirmethodologytotakeintoaccountanydifferencesintheculturalbackgroundsofpotentialsurveyparticipantsandthecharacteristicsoftheeventualdevicesused.
DuetohighInternetpenetrationandtherelativelylowcostofconducting websurveyscomparedwithothermethods,thenumberofsurveysbeingconductedviatheInternethasincreaseddramaticallyoverrecentyears.Thepanorama ofsurvey-basedresearchhaschangeddrasticallyoverthelastfewdecades.
Firsttherewasachangefromtraditionalpaper-and-pencilinterviewing(PAPI) tocomputer-assistedinterviewing(CAI).Sincethe1990s,therehasbeenagradual replacingofface-to-facesurveys(CAPI),telephonesurveys(CATI),andmailsurveys(CASI,CSAQ)withweb-basedsurveys.Withtherelativelyrecentdiffusionof smartphonesandothermobiledevices,ithasbecomepossibletorunmobileweb
surveys,i.e.,questionnairessenttointervieweesmaybesubmittedandalsocompletedviamobiledevices.Awebsurveyisasimplewaytoaccessalargegroupof potentialrespondents.Questionnairescanbedistributedatverylowcost.They requirenointerviewers,andtherearenomailingorprintingcostsinvolved.Surveys canbelaunchedrapidly,andlittletimeislostbetweenthemomentthequestionnaireisreadyandthemomentthatfieldworkbegins.Websurveysalsoofferinterestingnewpossibilities,suchastheuseofmultimedia(images,sound,animation, andvideo).Panelsurveysarealsomovingtowarddatacollectionviatheweb.
Therecenttrendtowardtheuseofbigdataandtheintegrationofdatasources willnotrendertheroleofwebsurveysobsolete,althoughtheymayinthefuture haveadifferentrole.
Atfirstsight,websurveysappeartohavemuchincommonwithothertypesof survey,seemingtobejustanotherwaytocollectdata,withquestionsaskedover theInternetinsteadofface-to-face,bytelephoneorviae-mail.Therearehowevera numberoffactorsthatmayrendertheresultsofwebsurveysunreliable.Some examplesareunder-coverage,self-selection,andmeasurementerrors.Thesecan causeestimatesofpopulationcharacteristicstobebiased,thusleadingtoincorrect conclusionsbeingdrawnfromthedatacollected.
Under-coverageoccurswhenthetargetpopulationiswiderthanthenumber ofpeoplewithInternetaccess.Thisleadstobiasinestimatesinthecaseofrelevant differencesbetweenthosewithInternetaccessandthosewithout.
Self-selectioniswhenaquestionnaireissimplymadeavailableviaInternetto all,withindividualsnominatingthemselves.
ArespondentisthereforeanyonewhohappenstohaveInternetaccess,visits thewebsite,anddecidestotakepartinthesurvey.Theseparticipantsgenerally differsignificantlyfromnonparticipants.
General-populationsurveysthataimtoprovidereliableandaccuratestatistics aretraditionallycarriedoutface-to-faceorbytelephone.Interviewersareusedto persuadepeopletotakepartandtohelprespondentstoprovidetherightanswers. Inwebsurveys,thereisnointerviewerassistance,afactthatcanhaveserious impactonthequalityofthedatacollected.
Thediffusionofsmartphonesincreasesthepossibilityforintervieweestobe reachedviatheirmobiledeviceandtohavethequestionnairecompletedviathe samedevice,resultinginthecurrenttrendinrunningmobilewebsurveys.Consequently,therearenewrisksforerrorinthesurveyduetodevicecharacteristics andthebehavioroftheuser.
Theresearchershouldhaveinmindthatwhenawebsurveyisrunamobile websurveytakesplace,ifquestionnaireisnotblockedagainstmobiledevices. Here,forsimplicitythetermwebsurveysisused,meaninigthemobilewebsurvey isincluded.Summingup,websurveysaffordseveralchallangesandneedreseacher beconsciousofthemethodologicalissuesforagoodsurvey.Atthetimebeeing, collectingdatathroughwebsurveysisgoingtobecomeacommonpracticebothin marketresearch,academicresearchandofficialstatistics.Knowledgeabouthowto manageawebsurvey,risks,errorsandadvantagesisimportant.
Thisbookprovidesaninsightintothepossibleuseofwebsurveysandmobile websurveysfordatacollection.Websurveysallowforlowerdatacollectioncosts.
Itisalsoexpectedthatwebsurveysleadtoincreasedresponserates.Isthisthecase? Whataboutthequalityofthedatacollected?Thisbookexaminesmanytheoretical andpracticalaspectsofmobilewebsurveysandcanthereforebeconsideredasa handbookforthoseinvolvedinpracticalsurveyresearch,includingsurvey researchersworkingwithofficialstatistics(e.g.,innationalstatisticalinstitutes), academics,andcommercialmarketresearch.
Thebook’stwoauthorshavewidespreadexpertiseinsurveymethodology. Theycomefromtwodifferentcountries(theNetherlandsandItaly)anddifferent researchorganizations(anationalstatisticalinstituteandauniversity).Theythereforeprovideabroadviewonthevarioustheoreticalandpracticalaspectsofmobile websurveying.
Thesecondeditionofthebookinvolvesarevisionofeachchapterwiththe followingcriteria:
(1)tomaintaintheexistingtextandcontentasmuchaspossible,(2)toupdate theexistingtextandcontentwithcommentsbasedonnewliteratureandresults,and (3)toaddnewparagraphs(ifnecessary)tocovernewrelevanttopics(seethecontents andchapterdescriptionbelow).Anumberofnewexampleshavebeenprovided, someoftheexistingexampleshavebeenupdatedorsubstituted,andsomeapplicationshavebeenreplaced.Updateshavealsobeenincludedtohighlightnewtrendsin websurveysandemergingsolutions.Therearetwonewchaptersontopicsconcerningmobilewebsurveys:onepresentingaflowcharttoillustratethestepsinvolvedin runningasurveyviawebandtheotherexaminingadaptivedesign.Itwastherefore necessarytorenumberthechaptersinrespecttothefirstedition.
Thefirsttwochaptersofthebookprovideanintroductionintowebsurveys. Chapter1providesahistoricaccountofdevelopmentsinsurveyresearchand showshowwebsurveyshavebecomeatoolfordatacollection.Section1.3.1examinestheBlaisesystem,whichhasbeenaroundformorethan30years.Newdevelopmentshavetakenplaceoverthelast10years,butnopapershavebeenwritten onthissubject.Thesectionlooksatthehistoryandrecentdevelopmentsregarding Blaise;itwaswrittenbyLonHofmanandMarkPierzchalaandispublishedforthe firsttimehere.
Chapter2isanoverviewofbasicaspectsofwebsurveys.Itdescribeshowand wheretheycanbeused.Officialstatisticsdepartments,researchinstitutions,marketresearchcompanies,andprivateforumsareallinterestedinwebsurveysstudyingbothhouseholds/individualsandbusinesses.
Chapter3presentsaflowchartillustratingthesteps(andsub-steps)ofweb surveys,eachaccompaniedbyashortdescription.Theflowchartisofpotential usetobothpractitionersasaguidelineforhowthesurveyprocessshouldbecarried outandtoresearchersinhighlightingandexplainingthepositioningoftheirstudiesatthedifferentstepsofthesurveyprocess.Itisalsousefulindiscussingthe errorsthatcanoccurindifferentsteps.Thechapterprovidesanintroduction totheframeworkanditsstructureanddiscussestherelevanceofbearinginmind theframeworkandthesurveystepswhenconsideringwebsurveyerrors.Itthen goesontodescribetheconceptofthestepandthestructureoftheflowchart, breakingdownthewebsurveyprocessintosixmainsteps.Theseareanalyzed indetail,andanoverviewofsurveyerrorsisprovided.
Chapter4examinestheaspectsofsampling.Itisstressedthatvalidpopulation inferenceispossibleonlyifsomeformofprobabilitysamplingisusedandthata propersamplingframeisrequiredforthis.Anumberofsamplingdesignsandestimationproceduresusefulforwebsurveysarediscussed.
Aresearcherconductingasurveymayencounteranumberofpracticalproblems,andChapter5providesanoverviewofpossibleerrors,withtwotypesof errorexaminedinfurtherdetail.Thefirstconcernserrorsinmeasurement.These canbecausedbyspecificissuesinquestionnairedesign,aswellasanumberof otheraspectssuchastechnology,incorrectunitdefinition,andsoon.Thesecond typeoferrorregardsnonresponse.Thisisaphenomenonthatcanaffectallsurveys, butthespecificaspectsofnonresponseinwebsurveysrequireparticularattention. Thechapterprovidesadviceonrelationshipsbetweenerrorsandinformationon thevarioustypes.
Awebsurveyisjustoneformofdatacollection.Thereareothers,suchasfaceto-face,telephone,mail,andmobilesurveys.Chapter6comparesthesevarious methodswithonlinedatacollection,discussingtheadvantagesanddisadvantages ofeachone.
Aswebsurveysdonotinvolveinterviewers,therespondentscompletethe questionnaireontheirown.Furthermore,whenquestionnairesaresentout,they mayverywellbereceivedandevencompletedonamobiledevice(suchassmartphones,whichareverywidespread).Thismeansthatquestionnairedesignisof crucialimportance.Questionnairesmustbeadaptedinordertobesuitablefor mobiledevices;otherwisetheycannotbeusedforthispurpose.Smallimperfectionsinthequestionnairemayhaveseriousconsequencesintermsofdataquality. QuestionnairedesignissuesareaddressedinChapter7.
Chapter8examinesstrategiesfordatacollectionwithadaptive/responsivesurveydesign.Inthiscase,strategiesarenotdefinedinadvance,butinsteadare adapted,ifnecessary,duringfieldwork.Thesedesignsmaycontributetocounteringgrowingproblemsofnonresponse.ThischapterwaswrittenbyAnnamaria BianchiandBarrySchouten,whoappliedtheirparticularexpertiseinthisfield tothesubjectexamined.
Awebsurveymaynotalwaysbethebestsolutionforprovidingreliableand accuratestatistics,withqualitybeingaffectedbyproblemsofunder-coverageand lowresponserates.Aninterestingalternativeistosetupamixed-modesurvey,in whichseveraldatacollectionmethodsarecombinedeithersequentiallyorconcurrently.Thisapproachislessexpensivethanasingle-modeinterviewer-assisted survey(conductedeitherfacetofaceorbytelephone)andsolvesunder-coverage problems,butatthesametimeitposesotherdifficulties,knownasmodeeffects, withoneofthemostsignificantofthesebeingmeasurementerror.Mixing modesisalsoofcriticalimportance,asisthefactthatinpractice,awebsurvey isalwaysmobile,unlessquestionnaireaccessviamobiledeviceisrestricted.All theseaspects,aswellasothersconcerningmixed-modesurveys,arediscussedin Chapter9.
Chapter10isdevotedtotheproblemofunder-coverage.Thisremainsan importantprobleminmanycountriesduetopoorInternetcoverageandthefact thatInternetaccessisoftenunevenlydistributedthroughoutthepopulation.The
chapterdemonstrateshowthiscanleadtosurveyestimatesbeingbiased. Anumberoftechniquesthatmayreduceunder-coveragebiasarediscussed.
Chapter11examinesself-selection.Thecorrectandscientificallywellfoundedprincipleistouseprobabilitysamplinginordertoselectsurveysubjects andthereforeallowreliableestimatesregardingpopulationcharacteristicstobe calculated.Nowadays,itiseasytosetupawebsurvey.Eventhosewithoutany surveyknowledgeorexperiencecancreateonethroughdedicatedwebsites.Many oftheresultingwebsurveysdonotapplyprobabilitysampling,butinsteadrelyon self-selectionofrespondents.Thiscausesseriousproblemswithestimation.Selfselectionanditsconsequencesintermsofsurveyresultsarediscussedinthischapter,demonstratingthatcorrectiontechniquesarenotalwayseffective,andthere aremanyreasonswhyweb-survey-basedestimatesarebiased.
Nonresponse,under-coverage,andself-selectionaretypicalexamples,and adjustmentweightingisoftenappliedinsurveysinordertoreducesuchbiases. Chapter12describesvariousweightingtechniques,suchaspost-stratification,generalizedregressionestimationandrakingratioestimation.Theeffectivenessofthese techniquesinreducingbiascausedbyunder-coverageorself-selectionisexamined.
Chapter13introducestheconceptofresponseprobabilities,describinghow theycanbeestimatedthroughresponsepropensities.Ifestimatedaccurately, responseprobabilitiescanbeusedtocorrectbiasedestimates.Here,twogeneral approachesaredescribed:responsepropensityweightingandresponsepropensity stratification.Thefirstattemptstoadjusttheoriginalselectionprobabilities,while thesecondisaformofpost-stratification.
Chapter14isdevotedtowebpanels.Therearemanysuchpanels,particularly inthefieldofcommercialmarketresearch.Onecrucialaspectishowthepanel members(households,individuals,companies,andshops)arerecruited.This canbecarriedoutviaaproperprobabilitysample,orthroughself-selection.There areconsequencesforthevalidityoftheresultsofthespecificsurveysconducted withthepanelmembers.Thechapterdiscussesseveralqualityindicators.
Theaccompanyingwebsite,www.web-survey-handbook.com,providesthe surveydatasetforthegeneralpopulationsurvey(GPS),whichhasbeenused formanyexamplesandapplicationsinthebook.ThedatasetisavailableinSPSS (SPSSCorporation,Chicago,IL)format.
SILVIA BIFFIGNANDI JELKE BETHLEHEM
Theeditorsacknowledgethecontributionsof:
LonHofman(ManagerBlaise,StatisticsNetherlands)andMarkPierzchala (ownerofMMPSurveyServices,Rockville,USA)whowroteSection1.3.1.
AnnamariaBianchi(UniversityofBergamo)andBarrySchouten(Statistics Netherlands)whowroteChapter8.
TheRoadtoWebSurveys
1.1 Introduction
Websurveysareanextstepintheevolutionprocessofsurveydatacollection. Collectingdataforcompilingstatisticaloverviewsisalreadyveryold,almostas oldasmankind.Allthroughhistory,rulersofcountriesusedstatisticstotake informeddecisions.However,newdevelopmentsinsocietyalwayshavehadtheir impactonthewaythedatawerecollectedforthesestatistics.
Foralongperiod,untiltheyear1895,statisticaldatacollectionwasbasedon completeenumerationofpopulations.Thecensusesweremostlyconductedto establishthesizeofthepopulation,todeterminetaxobligationsofthepeople, andtomeasurethemilitarystrengthofthecountry.Theideaofsamplinghad notemergedyet.
Theyear1895marksafundamentalchange.Populationshadgrownbigger andbigger.Itwastheperiodofindustrialization.Centralizedgovernments requiredmoreandmoreinformation.Thetimewasripeforsamplesurveys. Thefirstideasemergedaround1895.Therewasalotofdiscussionbetween 1895and1934abouthowtoselectsamples:bymeansofprobabilitysampling orsomeothersampleselectiontechnique.
By1934,itwasclearthatonlysurveysbasedonprobabilitysamplingcould providereliableandaccurateestimates.Suchmethodsofdatacollectionwere acceptedasascientific.Intheperiodfrom1940stothe1970s,mostsamplesurveyswereprobabilitybased.Questionnaireswereonpaperforms.Theywerecompletedinface-to-face,telephone,ormail.
HandbookofWebSurveys,SecondEdition.SilviaBiffignandiandJelkeBethlehem. ©2021JohnWiley&Sons,Inc.Published2021byJohnWiley&Sons,Inc.
Somewhereinthe1970sanothersignificantdevelopmentstarted.Thefast developmentofmicrocomputersmadeitpossibletointroducecomputer-assisted interviewing(CAI).Thismadesurveydatacollectionfaster,cheaper,andeasier andincreaseddataquality.ItwastimeinwhichacronymslikeCATI(computer-assistedtelephoneinterviewing)andCAPI(computer-assistedpersonal interviewing)emerged.
ThenextmajordevelopmentwasthecreationoftheInternetaround1982. WhenmoreandmorepersonsandcompaniesgotaccesstotheInternet,itbecame possibletousethisnetworkforsurveydatacollection.ThefirstInternetsurveys weree-mailsurveys.In1989theWorldWideWebwasdeveloped.Thissoftware allowedforfriendlygraphicaluserinterfacesforInternetusers.Thefirstbrowsers emergedandtheuseofInternetexploded.Inthemiddleof1990s,theWorld WideWebbecamewidelyavailable,ande-mailsurveyswereincreasinglyreplaced bywebsurveys.
Websurveysareattractivebecausetheyhaveanumberofadvantages.They allowforsimple,fast,andcheapaccesstolargegroupsofpotentialrespondents. Notsurprisingly,thenumberofconductedwebsurveyshasincreasedrapidlyover time.Thereare,however,alsopotentialmethodologicalproblems.Thereare ampleexamplesofwebsurveysnotbasedonprobabilitysampling.Therefore,generalizationofsurveyresultstothepopulationisquestionable.Theinterviewedmay accesstheInternetusingvarioustypesofdevices.Thus,websurveyscanbecompletedandreceivednotonlyonpersonalcomputer(PC)orlaptop;itishighly probablethesurveytobereceivedinthemobilephone.Theso-calledmobile websurveysarefullypartofwebsurveys.Thisimpliessomemethodologicalproblemstobeconsidered,andfurtherresearchontheimpactofmobileiscalledfor.
Thischapterdescribesthehistoricaldevelopmentsthathaveledtotheemergenceofwebsurveys.Asanillustration,Section1.3 showshowthesedevelopmentswereimplementedatStatisticsNetherlandsandledtonewsoftwarefor surveydatacollection.
1.2 Theory
1.2.1THEEVERLASTINGDEMANDFOR STATISTICALINFORMATION
Thehistoryofdatacollectionforstatisticsgoesbackintimeforthousandsofyears. AsfarbackasBabylonianera,acensusofagriculturewascarriedout.Thisalready tookplaceshortlyaftertheinventionoftheartofwriting.ThesamethinghappenedinChina.Thisempirecounteditspeopletodeterminetherevenuesandthe militarystrengthofitsprovinces.Therearealsoaccountsofstatisticaloverviews compiledbyEgyptianrulerslongbeforeChrist.Romeregularlytookcensusesof peopleandofproperty.Thecollecteddatawereusedtoestablishthepoliticalstatusofcitizensandtoassesstheirmilitaryandtaxobligationstothestate.
CensuseswererareintheMiddleAges.Themostfamousonewasthecensus ofEnglandtakenbytheorderofWilliamtheConqueror,KingofEngland.The compilationofhis DomesdayBook startedintheyear1086AD.Thebookrecordsa wealthofinformationabouteachmanorandeachvillageinthecountry.Collected informationwasaboutmorethan13,000places.Morethan10,000factswere recordedforeachcountry.
Tocollectallthisdata,thecountrywasdividedintoanumberofregions.In eachregion,agroupofcommissionerswasappointedfromamongthegreater lords.Eachcountywithinaregionwasdealtwithseparately.Sessionswereorganizedineachcountytown.Thecommissionerssummonedallthoserequiredto appearbeforethem.Theyhadpreparedastandardlistofquestions.Forexample, therewerequestionsabouttheownerofthemanor;thenumberoffreemanand slaves;theareaofwoodland,pasture,andmeadow;thenumberofmillsandfishponds,tothetotalvalue;andtheprospectsofgettingmoreprofit.The Domesday Book stillexists,andmanycountydatafilesareavailableonCD-ROMandthe Internet.
AnotherinterestingexampleofthehistoryofofficialstatisticsisintheInca Empirethatexistedbetween1000and1500AD.EachIncatribehaditsownstatistician,calledthe quipucamayoc.Thismankeptrecordsofthenumberofpeople, thenumberofhouses,thenumberofllamas,thenumberofmarriages,andthe numberofyoungmenthatcouldberecruitedforthearmy.Allthesefactsrecorded on quipus,asystemofknotsincoloredropes.Adecimalsystemwasusedforthis. Atregularintervals,couriersbroughtthequipustoCusco,thecapitalofthekingdom,whereallregionalstatisticswerecompiledintonationalstatistics.Thesystem ofquipucamayocsandquipusworkedremarkablywell.Thesystemvanishedwith thefalloftheempire.
AnearlycensusalsotookplaceinCanadain1666.JeanTalon,theintendant ofNewFrance,orderedanofficialcensusofthecolonytomeasuretheincreasein populationsincethefoundingofQuebecin1608.Name,age,sex,maritalstatus, andoccupationwererecordedforeveryperson.Itturnedouttherelived3,215 peopleinNewFrance.
ThefirstcensusesinEuropetookplaceintheNordiccountries.ThefirstcensusinSweden–Finlandtookplacein1749.Noteveryonewelcomedtheideaofa census.Particularlyreligiouspeoplebelievedthatpeopleshouldnotbecounted. TheyreferredtothecensusorderedbyKingDavidinbiblicaltimes,whichwas interruptedbyaterribleplagueandnevercompleted.Otherssaidthatapopulationcountwouldrevealthestrengthsandweaknessesofacountrytoforeignenemies.Nevertheless,censusestookplaceinmoreandmorecountries.Thefirst censusinDenmark–Norwayhasbeenin1769.In1795,atthetimeoftheBatavianRepublicunderNapoleon’sinfluence,thefirstcountofthepopulationofthe Netherlandstookplace.Thenewcentralizedadministrationwantedtogather quantitativeinformationtodeviseanewsystemofelectoralconstituencies(see DenDulkandVanMaarseveen,1990).
Intheperioduntilthelate1880s,thereweresomeapplicationsof partial investigations.Theywerestatisticalinquiriesinwhichonlypartofacomplete
humanpopulationhasbeeninterviewed.Thewaythepersonswereselectedfrom thepopulationwasgenerallyunclearandundocumented.
Inthesecondhalfofthe19thcentury,so-calledmonographstudiesbecame popular.TheywerebasedonQuetelet’sideaoftheaverageman.Accordingto Quetelet,manyphysicalandmoraldatahaveanaturalvariability.Thisvariability canbedescribedbyanormaldistributionaroundafixed,truevalue.Heassumed theexistenceofsomethingcalledthe truevalue.Queteletintroducedtheconcept of averageman (“l’hommemoyenne”)asapersonofwhichallcharacteristicswere equaltothetruevalue(seeQuetelet,2010,2012).
Theperiodofthe18thand19thcenturiesiscalledtheeraoftheIndustrial Revolution,too.Itledtoimportantchangesinsociety,science,andtechnology. Amongmanyotherthings,urbanizationstartedfromindustrializationanddemocratization.Allthesedevelopmentscreatednewstatisticaldemands.Thefoundationsformanyprinciplesofmodernstatisticswerelaid.Severalcentral statisticalbureaus,statisticalsocieties,conferences,andjournals,wereestablished soonafterthisperiod.Firstideasaboutsurveysamplingemergedintheworldof officialstatistics.Ifastartingyearmustbechosen,1895wouldbeagoodcandidate.AndersKiaer,thefounderandfirstdirectorofStatisticsNorway,startedin thisyearafundamentaldiscussionabouttheuseofsamplingmethods.Thisdiscussionledtothedevelopment,acceptance,andapplicationofsamplingasascientificmethod.
AndersKiaer(1838–1919)wasthefounderandadvocateofthesurvey methodthatisnowwidelyappliedinofficialstatisticsandsocialresearch.With thefirstpublicationofhisideasin1895,hestartedtheprocessthatendedin thedevelopmentofmodernsurveysamplingtheoryandmethods.Thisprocess isdescribedinmoredetailinBethlehem(2009).
Therehavebeenearlierexamplesofscientificinvestigationsbasedonsamples, buttheywerelackingproperscientificfoundations.Thefirstknownattemptof drawingconclusionsaboutapopulationusingonlyinformationaboutpartofit wasmadebytheEnglishmerchantJohnGraunt(1662).Heestimatedthesize ofthepopulationofLondon.Grauntsurveyedfamiliesinasampleofparishes wheretheregisterswerewellkept.Hefoundthatonaveragetherewerethreeburialsperyearin11families.Assumingthisratiotobemoreorlessconstantforall parishesandknowingthetotalnumberofburialsperyearinLondontobeabout 13,000,heconcludedthatthetotalnumberoffamilieswasapproximately48,000. Puttingtheaveragefamilysizeat8,heestimatedthepopulationofLondontobe 384,000.Sincethisapproachlackedaproperscientificfoundation,JohnGraunt couldnotsayhowaccuratehisestimateswere.
Morethanacenturylater,theFrenchmathematicianPierre-SimonLaplace realizedthatitwasimportanttohavesomeindicationoftheaccuracyofhisestimateoftheFrenchpopulation.Laplace(1812)implementedanapproachthatwas moreorlesssimilartothatofJohnGraunt.Heselected30departmentsdistributedovertheareaofFranceinsuchawaythatalltypesofclimatewererepresented. Moreover,heselecteddepartmentsinwhichaccuratepopulationrecordswere kept.Usingthecentrallimittheorem,Laplaceprovedthathisestimatorhada
normaldistribution.Unfortunately,hedisregardedthefactthatsamplingwaspurposively,andnotatrandom.Theseproblemsmadeapplicationofthecentrallimit theorematleastdoubtful.
In1895AndersKiaer(1895,1997),thefounderandfirstdirectorofStatistics Norway,proposedhis representativemethod.Itwasapartialinquiryinwhicha largenumberofpersonswerequestioned.Selectionofpersonswassuchthata “miniature” ofthepopulationwasobtained.AndersKiaerstressedtheimportance of representativity.Hearguedthatifasamplewasrepresentativewithrespectto variablesforwhichthepopulationdistributionwasknown,itwouldalsoberepresentativewithrespecttotheothersurveyvariables.Example1.1describesthe Kiaer’sexperimentabouttherepresentativemethod. □■
EXAMPLE1.1 TherepresentativemethodofAndersKiaer
AndersKiaerappliedhisrepresentativemethodinNorway.Hisideawasto surveythepopulationofNorwaybyselectingasampleof120,000people. Enumerators(hiredonlyforthispurpose)visitedthesepeopleandfilledin 120,000forms.About80,000oftheformswerecollectedbytherepresentativemethodand40,000formsbyaspecial(butanalogue)methodinareas wheretheworking-classpeoplelived.
Forthefirstsampleof80,000respondents,datafromthe1891census wereusedtodividethehouseholdsinNorwayintotwostrata.Approximately20,000peoplewereselectedfromurbanareasandtherestfrom ruralareas.
Therewasaselectionof13representativecitiesfromthe61cities inNorway.Allfivecitieshavingmorethan20,000inhabitantswere included,andeightcitiesrepresentingthemediumsizedandsmall towns,too.Theproportionofselect edpeopleincitiesvaried:inthe middle-sizedandsmallcities,theproportionwasgreaterthaninthe bigcities.Kiaermotivatedthischoicebythefactthatthemiddle-sized andsmallcitiesdidnotrepresentonlythemselvesbutalargernumber ofsimilarcities.
InKristiania(nowadaysOslo)theproportionwas1/16,inthe medium-sizedtownstheproportionvariedbetween1/12and1/9,and inthesmalltownsitwas1/4or1/3ofthepopulation.
Basedonthecensus,itwasknownhowmanypeoplelivedineachof the400streetsofKristiania,thecapitalofNorway.Thesortingofthe streetswasinfourcategoriesaccordingtothenumberofinhabitants.Then, therewasthespecificationofaselectionschemeforeachcategory:theadult populationenumerationwasin1outof20forthesmalleststreets.Inthe secondcategory,theadultpopulationenumerationwasinhalfofthehouses
in1outof10ofstreets.Inthethirdcategory,theenumerationconcerned one-fourthofthestreets,andtheenumerationwaseveryfifthhouse;andin thelastcategoryofthebiggeststreets,theadultpopulationenumeration wasonhalfofthestreetsandin1outof10housesinthem.
Inselectingthestreetstheirdistributionoverthecitywasconsideredto ensurethelargestpossibledispersionandthe “representativecharacter” of theenumeratedareas.
Inthemedium-sizedtowns,thesamplewasselectedusingthesame principles,thoughinaslightlysimplifiedmanner.Inthesmallesttowns, thetotaladultpopulationinthreeorfourhouseswasenumerated.
Thenumberofinformantsineachofthe18countiesintheruralpartof Norwaywasdecidedconsideringcensusdata.Toobtainrepresentativeness, municipalitiesineachcountry,itwasusedaclassificationaccordingtotheir mainindustry,eitherasagricultural,forestry,industrial,seafaring,orfishing municipalities.Inaddition,thegeographicaldistributionwasconsidered.
Thetotalnumberoftherepresentativemunicipalitiesamountedto 109,whichissixineachcountyonaverage.Thetotalnumberofmunicipalitieswas498.
Theselectionofpeopleinamunicipalitywasdoneinrelationtothe populationindifferentparishes,andsoalldifferentmunicipalitieswere covered.Thefinalstepwastoinstructenumeratorstofollowaspecificpath. Inaddition,instructiontotheenumeratorswastovisitdifferenthouses situatedclosetoeachother.Thatis,theyweresupposedtovisitnotonly middle-classhousesbutalsowell-to-dohouses,poor-lookinghouses,and one-personhouses.
Kiaerdidnotexplaininhispapershowhecalculatedestimates.Themain reasonprobablywasthattherepresentativesampleconstructionwasasaminiature ofthepopulation.Thismadecomputationsofestimatestrivial:thesamplemeanis theestimateofthepopulationmean,andtheestimateofthepopulationtotal couldbeattainedsimplybymultiplyingthesampletotalbytheinverseofsampling fraction.
Abasicproblemoftherepresentativemethodwasthattherewasnowayof establishingtheprecisionofpopulationestimates.Themethodlackedaformal theoryofinference.ItwasBowley(1906,1926)whomadethefirststepsinthis direction.Heshowedthatforlargesamples,selectedatrandomfromthepopulation,estimateshadanapproximatelynormaldistribution.Fromthismoment on,thereweretwomethodsofsampleselection:
• Kiaer’srepresentativemethod,basedonpurposiveselection,inwhichrepresentativityplayedanessentialroleandforwhichnomeasureoftheaccuracyof theestimatescouldbeobtained;
• Bowley’sapproach,basedonsimplerandomsampling,forwhichanindicationoftheaccuracyofestimatescouldbecomputed.
Bothmethodsexistedsidebysideuntil1934.InthatyearthePolishscientistJerzy Neymanpublishedhisfamouspaper(seeNeyman,1934).Neymandevelopeda newtheorybasedontheconceptoftheconfidenceinterval.Byusingrandom selectioninsteadofpurposiveselection,therewasnoneedanymoretomakeprior assumptionsaboutthepopulation.ThecontributionofNeymanwasnotonlythat heproposedtheconfidenceintervalasanindicatorfortheprecisionofestimates. HealsoconductedanempiricalevaluationofItaliancensusdataandprovedthat therepresentativemethodbasedonpurposivesamplingwasnotabletoprovide satisfactoryestimatesofpopulationcharacteristics.Heestablishedthesuperiority ofrandomsampling(alsoreferredtoas probabilitysampling)overpurposivesampling.Consequently,useofpurposivesamplingwasrejectedasascientificsamplingmethod.
Graduallyprobabilitysamplingfounditswayintoofficialstatistics.Moreand morenationalstatisticalinstitutesintroducedprobabilitysamplingforofficialstatistics.However,theprocesswasslow.Forexample,afirsttestofarealsample surveyusingrandomselectionwascarriedoutbyStatisticsNetherlandsonlyin 1941(seeCBS,1948).Usingasimplerandomsampleofsize30,000fromthe populationof1.75milliontaxpayers,itwasshownthatestimateswereaccurate.
Thehistoryofopinionpollsgoesbacktothe1820s,inwhichperiodAmericannewspapersattemptedtodeterminepoliticalpreferenceofvotersjustbefore thepresidentialelection.Theseearlypollsdidnotpaymuchattentiontosampling. Therefore,itwasdifficulttoestablishaccuracyofresults.Suchopinionpollswere oftencalled strawpolls.ThisexpressiongoesbacktoruralAmerica.Farmerswould throwahandfulofstrawsintotheairtoseewhichwaythewindwasblowing. Ittookuntilthe1920sbeforemoreattentionwaspaidtosamplingaspects. Lienhard(2003)describeshowGeorgeGallupworkedoutnewwaystomeasure interestinnewspaperarticles.Gallupused quotasampling.Theideawastoinvestigateagroupofpeoplethatcouldbeconsideredrepresentativeforthepopulation. Hundredsofinterviewersacrossthecountryvisitedpeople.Interviewerswere givenquotafordifferentgroupsofrespondents.Theyhadtointerviewsomany middle-classurbanwomen,somanylower-classruralmen,etc.Intotal,approximately3,000interviewswereconductedoutforasurvey.
Gallup’sapproachwasingreatcontrastwiththatofthe LiteraryDigest magazine,whichwasatthattimetheleadingpollingorganization.Thismagazineconductedregular “AmericaSpeaks” polls.Itbaseditspredictionsonreturned questionnaireformsthatweresenttoaddressestakenfromtelephonedirectories booksandautomobileregistrationlists.Thesamplesizeforthesepollswasonthe orderoftwomillionpeople.SothesamplesizewasmuchlargerthanthatofGallup ’spolls.
Thepresidentialelectionof1936turnedouttobedecisiveforbothmethods. ThisisdescribedbyUtts(1999).GallupcorrectlypredictedFranklinRooseveltto bethenewpresident,whereas LiteraryDigest predictedthatAlfLandonwould
beatFranklinRoosevelt.Thepredictionbasedontheverylargesamplesizeturned outtobewrong.Theexplanationwasthatthesamplingtechniqueof Literary Digest didnotproducerepresentativesamples.Inthe1930s,carsandtelephones weretypicallyownedbymiddle-andupper-classpeople.Thesepeopletendedto voteRepublican,whereaslower-classpeopleweremoreinclinedtovoteDemocrat. Consequently,Republicanswereoverrepresentedinthe LiteraryDigest sample.
Asaresultofthishistoricmistake,opinionresearcherslearnedthatthey shouldrelyonmorescientificwaysofsampleselection.Theyalsolearnedthat thewayasampleisselectedismoreimportantthanthesizeofthesample.
Theclassicaltheoryofsurveysamplingwasmoreorlesscompletedin1952. HorvitzandThompson(1952)developedageneraltheoryforconstructingunbiasedestimates.Whatevertheselectionprobabilitiesare,aslongastheyareknown andpositive,itisalwayspossibletoconstructausefulestimate.Horvitzand Thompsoncompletedtheclassicaltheory,andtherandomsamplingapproach wasalmostunanimouslyaccepted.Mostoftheclassicalbooksaboutsampling werealsopublishedbythen(Cochran,1953;Deming,1950;Hansen,Hurvitz, andMadow,1953;Yates,1949).
1.2.2TRADITIONALDATACOLLECTION
Therewerethreemodesofdatacollectionintheearlydaysofsurveyresearch:faceto-faceinterviewing,mailinterviewing,andtelephoneinterviewing.Eachmode haditsadvantagesanddisadvantages.
Face-to-faceinterviewing wasalreadyusedforthefirstcensuses.Thus,itisnot asurpriseitwasalsousedforsurveys.Face-to-faceinterviewingmeansthatinterviewersvisitthepersonsselectedinthesample.Well-trainedinterviewerswillbe successfulinpersuadingreluctantpersonstoparticipateinthesurvey.Therefore, responseratesofface-to-facesurveysareusuallyhigherthansurveysnotinvolving interviewers(forexample,mailsurveys).Interviewerscanalsoassistrespondentsin givingtherightanswerstothequestions.Thisoftenresultsinbetterdata.However,thepresenceofinterviewerscanalsobeadrawback.Researchsuggeststhat respondentsaremoreinclinedtoanswersensitivequestionsproperlyifthereareno interviewerspresent.
Surveyagenciesoftensendaletterannouncingthevisitoftheinterviewer. Suchalettercanalsogiveadditionalinformationaboutthesurvey,explain whyitisimportanttoparticipate,andassurethatthecollectedinformationistreatedconfidentially.Asaresult,therespondentsarenottakenbysurprisebythe interviewers.
Theresponserateofaface-to-facesurveyisusuallyhighandsoisqualityofthe collecteddata.Butapricehastobepaidliterally:face-to-faceinterviewingismuch moreexpensive.Ateamofinterviewershastobetrainedandpaid.Theyalsohave totravel,whichcoststimeandmoney.
Mailinterviewing ismuchlessexpensivethanface-to-faceinterviewing.Paper questionnairesaresentbymailtopersonsselectedinthesample.Theyareinvited toanswerthequestionsandtoreturnthecompletedquestionnairetothesurvey
