https://ebookmass.com/product/advances-in-questionnairedesign-development-evaluation-and-testing-paul-c-beatty/
Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...
Non-destructive Testing and Evaluation of Civil Engineering Structures Jean-Paul Balayssac
https://ebookmass.com/product/non-destructive-testing-and-evaluationof-civil-engineering-structures-jean-paul-balayssac/
ebookmass.com
Flight Simulation Software: Design, Development and Testing David Allerton
https://ebookmass.com/product/flight-simulation-software-designdevelopment-and-testing-david-allerton/
ebookmass.com
Reference Frame Theory : Development and Applications 1st Edition Paul C. Krause
https://ebookmass.com/product/reference-frame-theory-development-andapplications-1st-edition-paul-c-krause/
ebookmass.com
Phytoremediation Potential of Perennial Grasses 1st Edition Vimal Chandra Pandey
https://ebookmass.com/product/phytoremediation-potential-of-perennialgrasses-1st-edition-vimal-chandra-pandey/
ebookmass.com
Women and
Sustainable
Human Development: Empowering Women in Africa 1st ed. Edition Maty Konte
https://ebookmass.com/product/women-and-sustainable-human-developmentempowering-women-in-africa-1st-ed-edition-maty-konte/
ebookmass.com
Spring Rains (Whisper Ridge, Wyoming Book 3) Rj Scott
https://ebookmass.com/product/spring-rains-whisper-ridge-wyomingbook-3-rj-scott/
ebookmass.com
Sound and Reason: Synesthesia as Metacognition Sven Hroar Klempe
https://ebookmass.com/product/sound-and-reason-synesthesia-asmetacognition-sven-hroar-klempe/
ebookmass.com
Achilles in love: intertextual studies Fantuzzi
https://ebookmass.com/product/achilles-in-love-intertextual-studiesfantuzzi/
ebookmass.com
Burning Ivy: A Dark Mafia Romance (The KORT Series Book 1)
Brandy Hynes
https://ebookmass.com/product/burning-ivy-a-dark-mafia-romance-thekort-series-book-1-brandy-hynes/
ebookmass.com
https://ebookmass.com/product/design-and-manufacturing-of-structuralcomposites-lee-harper/
ebookmass.com
AdvancesinQuestionnaireDesign, Development,Evaluation,andTesting
AdvancesinQuestionnaireDesign, Development,Evaluation,andTesting
Editedby
PaulC.Beatty
DebbieCollins
LynKaye
Jose-LuisPadilla
GordonB.Willis
AmandaWilmot
Thiseditionfirstpublished2020 ©2020JohnWiley&SonsInc.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingor otherwise,exceptaspermittedbylaw.Adviceonhowtoobtainpermissiontoreusematerialfrom thistitleisavailableathttp://www.wiley.com/go/permissions.
TherightofPaulC.Beatty,DebbieCollins,LynKaye,Jose-LuisPadilla,GordonB.Willis,and AmandaWilmottobeidentifiedastheeditorialmaterialinthisworkhasbeenassertedin accordancewithlaw.
RegisteredOffice
JohnWiley&Sons,Inc.,111RiverStreet,Hoboken,NJ07030,USA
EditorialOffice 111RiverStreet,Hoboken,NJ07030,USA
Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWiley productsvisitusatwww.wiley.com.
Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Some contentthatappearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats.
LimitofLiability/DisclaimerofWarranty
Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakeno representationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthis workandspecificallydisclaimallwarranties,includingwithoutlimitationanyimpliedwarranties ofmerchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedby salesrepresentatives,writtensalesmaterialsorpromotionalstatementsforthiswork.Thefact thatanorganization,website,orproductisreferredtointhisworkasacitationand/orpotential sourceoffurtherinformationdoesnotmeanthatthepublisherandauthorsendorsethe informationorservicestheorganization,website,orproductmayprovideorrecommendationsit maymake.Thisworkissoldwiththeunderstandingthatthepublisherisnotengagedin renderingprofessionalservices.Theadviceandstrategiescontainedhereinmaynotbesuitable foryoursituation.Youshouldconsultwithaspecialistwhereappropriate.Further,readersshould beawarethatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthis workwaswrittenandwhenitisread.Neitherthepublishernorauthorsshallbeliableforanyloss ofprofitoranyothercommercialdamages,includingbutnotlimitedtospecial,incidental, consequential,orotherdamages.
LibraryofCongressCataloging-in-PublicationDataisappliedfor Paperback:9781119263623
CoverDesign:Wiley
CoverImage:©TuomasLehtinen/GettyImages
Setin10/12ptWarnockProbySPiGlobal,Chennai,India
PrintedintheUnitedStatesofAmerica 10987654321
Wededicatethisbooktosomeofourmostardentsupporters:
Paul:tomymother,MaryBeatty
Debbie:toMarc,Lilly,andGeorge
Lyn:toDebPotter,mymentorandmygoodfriend,whofirststartedmeon thispath
Jose-Luis:tomywife,Carmen;myson,Andres;andmymother,Josefina
Gordon:tothememoryofmymentor,MonroeSirken
Amanda:tomyhusbandLawrenceLeVasseurforhisloveandsupport,andin memoryofmymotherChristineWilmot
AndalsotothememoryofourcolleagueScottFricker,avaluedcontributorto thisbookandtoourfield,whowastakenfromusfartoosoon.
Contents
ListofContributors xvii
Preface xxiii
PartIAssessingtheCurrentMethodologyfor QuestionnaireDesign,Development,Testing,and Evaluation 1
1QuestionnaireDesign,Development,Evaluation,and Testing:WhereAreWe,andWhereAreWeHeaded? 3 GordonB.Willis
1.1CurrentStateoftheArtandScienceofQDET 3
1.2RelevanceofQDETintheEvolvingWorldofSurveys 11
1.3LookingAhead:FurtherDevelopmentsinQDET 16
1.4Conclusion 19 References 20
2AskingtheRightQuestionsintheRightWay:SixNeeded ChangesinQuestionnaireEvaluationandTesting Methods 25 DonA.Dillman
2.1PersonalExperienceswithCognitiveInterviewsandFocus Groups 25
2.2My2002ExperienceatQDET 29
2.3SixChangesinSurveyResearchthatRequireNewPerspectiveson QuestionnaireEvaluationandTesting 33
2.4Conclusion 42 References 43
3AFrameworkforMakingDecisionsAboutQuestion EvaluationMethods 47
RogerTourangeau,AaronMaitland,DarbySteiger,andTingYan
3.1Introduction 47
3.2ExpertReviews 48
3.3LaboratoryMethods 51
3.4FieldMethods 55
3.5StatisticalModelingforDataQuality 59
3.6ComparingDifferentMethods 63
3.7Recommendations 67 References 69
4AComparisonofFiveQuestionEvaluationMethodsin PredictingtheValidityofRespondentAnswerstoFactual Items 75
AaronMaitlandandStanleyPresser
4.1Introduction 75
4.2Methods 76
4.3Results 79
4.4Discussion 84 References 85
5CombiningMultipleQuestionEvaluationMethods:What DoesItMeanWhentheDataAppeartoConflict? 91 Jod’ArdenneandDebbieCollins
5.1Introduction 91
5.2QuestionnaireDevelopmentStages 92
5.3SelectionofCaseStudies 93
5.4CaseStudy1:ConflictingFindingsBetweenFocusGroupsand CognitiveInterviews 95
5.5CaseStudy2:ConflictingFindingsBetweenEye-Tracking, RespondentDebriefingQuestions,andInterviewerFeedback 97
5.6CaseStudy3:ComplementaryFindingsBetweenCognitive InterviewsandInterviewerFeedback 100
5.7CaseStudy4:CombiningQualitativeandQuantitativeDatato AssessChangestoaTravelDiary 104
5.8FrameworkofQTMethods 110
5.9SummaryandDiscussion 110 References 114
PartIIQuestionCharacteristics,ResponseBurden,and DataQuality 117
6TheRoleofQuestionCharacteristicsinDesigningand EvaluatingSurveyQuestions 119
JenniferDykema,NoraCateSchaeffer,DanaGarbarski,andMichaelHout
6.1Introduction 119
6.2OverviewofSomeoftheApproachesUsedtoConceptualize, Measure,andCodeQuestionCharacteristics 120
6.3TaxonomyofQuestionCharacteristics 127
6.4CaseStudies 132
6.5Discussion 141 Acknowledgments 147 References 148
7ExploringtheAssociationsBetweenQuestion Characteristics,RespondentCharacteristics,Interviewer PerformanceMeasures,andSurveyDataQuality 153
JamesM.Dahlhamer,AaronMaitland,HeatherRidolfo,AntuaneAllen,and DyneshaBrooks
7.1Introduction 153
7.2Methods 157
7.3Results 174
7.4Discussion 182 Disclaimer 191 References 191
8ResponseBurden:WhatIsItandWhatPredictsIt? 193
TingYan,ScottFricker,andShirleyTsai
8.1Introduction 193
8.2Methods 197
8.3Results 202
8.4ConclusionsandDiscussion 206 Acknowledgments 210 References 210
9TheSalienceofSurveyBurdenandItsEffectonResponse BehaviortoSkipQuestions:ExperimentalResultsfrom TelephoneandWebSurveys 213 FraukeKreuter,StephanieEckman,andRogerTourangeau
9.1Introduction 213
9.2StudyDesignsandMethods 216
x Contents
9.3ManipulatingtheInterleafedFormat 219
9.4DiscussionandConclusion 224 Acknowledgments 226 References 227
10AComparisonofFullyLabeledandTop-LabeledGrid QuestionFormats 229
JoleneD.SmythandKristenOlson
10.1Introduction 229
10.2DataandMethods 236
10.3Findings 243
10.4DiscussionandConclusions 253 Acknowledgments 254 References 255
11TheEffectsofTaskDifficultyandConversationalCueingon AnswerFormattingProblemsinSurveys 259 YfkeOngenaandSanneUnger
11.1Introduction 259
11.2FactorsContributingtoRespondents’FormattingProblems 262
11.3Hypotheses 267
11.4MethodandData 268
11.5Results 275
11.6DiscussionandConclusion 278
11.7FurtherExpansionoftheCurrentStudy 281
11.8Conclusions 282 References 283
PartIIIImprovingQuestionnairesontheWebandMobile Devices 287
12ACompendiumofWebandMobileSurveyPretesting Methods 289
EmilyGeisenandJoeMurphy
12.1Introduction 289
12.2ReviewofTraditionalPretestingMethods 290
12.3EmergingPretestingMethods 294 References 308
13UsabilityTestingOnlineQuestionnaires:Experiencesatthe U.S.CensusBureau 315 ElizabethNichols,EricaOlmsted-Hawala,TemikaHolland,andAmy AndersonRiemer
13.1Introduction 315
13.2HistoryofUsabilityTestingSelf-AdministeredSurveysattheUS CensusBureau 316
13.3CurrentUsabilityPracticesattheCensusBureau 317
13.4Participants:“RealUsers,NotUserStories” 320
13.5BuildingUsabilityTestingintotheDevelopmentLifeCycle 323
13.6MeasuringAccuracy 327
13.7MeasuringEfficiency 331
13.8MeasuringSatisfaction 335
13.9RetrospectiveProbingandDebriefing 337
13.10CommunicatingFindingswiththeDevelopmentTeam 339
13.11AssessingWhetherUsabilityTestRecommendationsWorked 340
13.12Conclusions 341 References 341
14HowMobileDeviceScreenSizeAffectsDataCollectedinWeb Surveys 349
DanieleToninelliandMelanieRevilla
14.1Introduction 349
14.2LiteratureReview 350
14.3OurContributionandHypotheses 352
14.4DataCollectionandMethod 355
14.5MainResults 361
14.6Discussion 368 Acknowledgments 369 References 370
15OptimizingGridQuestionsforSmartphones: AComparisonofOptimizedandNon-OptimizedDesignsand EffectsonDataQualityonDifferentDevices 375
TrineDaleandHeidiWalsoe
15.1Introduction 375
15.2TheNeedforChangeinQuestionnaireDesignPractices 376
15.3ContributionandResearchQuestions 378
15.4DataCollectionandMethodology 380
15.5MainResults 386
15.6Discussion 392 Acknowledgments 397 References 397
16LearningfromMouseMovements:ImprovingQuestionnaires andRespondents’UserExperienceThroughPassiveData Collection 403 RachelHorwitz,SarahBrockhaus,FelixHenninger,PascalJ.Kieslich,Malte Schierholz,FlorianKeusch,andFraukeKreuter
16.1Introduction 403
16.2Background 404
16.3Data 409
16.4Methodology 410
16.5Results 415
16.6Discussion 420 References 423
17UsingTargetedEmbeddedProbestoQuantifyCognitive InterviewingFindings 427 PaulScanlon
17.1Introduction 427
17.2TheNCHSResearchandDevelopmentSurvey 431 17.3Findings 433
17.4Discussion 445 References 448
18ThePracticeofCognitiveInterviewingThroughWeb Probing 451 StephanieFowlerandGordonB.Willis
18.1Introduction 451
18.2MethodologicalIssuesintheUseofWebProbingfor Pretesting 452
18.3TestingtheEffectofProbePlacement 453
18.4AnalysesofResponsestoWebProbes 455
18.5QualitativeAnalysisofResponsestoProbes 459
18.6QualitativeCodingofResponses 459
18.7CurrentStateoftheUseofWebProbes 462
18.8Limitations 465
18.9RecommendationsfortheApplicationandFurtherEvaluationof WebProbes 466
18.10Conclusion 468 Acknowledgments 468 References 468
PartIVCross-CulturalandCross-NationalQuestionnaire DesignandEvaluation 471
19OptimizingQuestionnaireDesigninCross-Nationaland Cross-CulturalSurveys 473
TomW.Smith
19.1Introduction 473
19.2TheTotalSurveyErrorParadigmandComparisonError 474
19.3Cross-CulturalSurveyGuidelinesandResources 477
19.4Translation 478
19.5DevelopingComparativeScales 480
19.6FocusGroupsandPretestinginCross-National/Cultural Surveys 483
19.7ToolsforDevelopingandManagingCross-NationalSurveys 484
19.8ResourcesforDevelopingandTestingCross-National Measures 485
19.9Pre-andPost-Harmonization 486
19.10Conclusion 488 References 488
20AModelforCross-NationalQuestionnaireDesignand Pretesting 493
RoryFitzgeraldandDianaZavala-Rojas
20.1Introduction 493
20.2Background 493
20.3TheEuropeanSocialSurvey 495
20.4ESSQuestionnaireDesignApproach 496
20.5CritiqueoftheSeven-StageApproach 497
20.6AModelforCross-NationalQuestionnaireDesignand Pretesting 497
20.7EvaluationoftheModelforCross-NationalQuestionnaireDesign andPretestingUsingtheLogicalFrameworkMatrix(LFM) 501
20.8Conclusions 512 References 514
21Cross-NationalWebProbing:AnOverviewofIts MethodologyandItsUseinCross-NationalStudies 521
DorothéeBehr,KatharinaMeitinger,MichaelBraun,andLarsKaczmirek
21.1Introduction 521
21.2Cross-NationalWebProbing–ItsGoal,Strengths,and Weaknesses 523
21.3AccesstoRespondentsAcrossCountries:TheExampleofOnline AccessPanelsandProbability-BasedPanels 526
21.4ImplementationofStandardizedProbes 527
21.5TranslationandCodingAnswerstoCross-CulturalProbes 532
21.6SubstantiveResults 533
21.7Cross-NationalWebProbingandItsApplicationThroughoutthe SurveyLifeCycle 536
21.8ConclusionsandOutlook 538 Acknowledgments 539 References 539
22MeasuringDisabilityEqualityinEurope:Designand DevelopmentoftheEuropeanHealthandSocialIntegration SurveyQuestionnaire 545 AmandaWilmot
22.1Introduction 545
22.2Background 546
22.3QuestionnaireDesign 548
22.4QuestionnaireDevelopmentandTesting 553
22.5SurveyImplementation 560
22.6LessonsLearned 563
22.7FinalReflections 566 Acknowledgments 567 References 567
PartVExtensionsandApplications 571
23Regression-BasedResponseProbingforAssessingthe ValidityofSurveyQuestions 573 PatrickSturgis,IanBrunton-Smith,andJonathanJackson
23.1Introduction 573
23.2CognitiveMethodsforAssessingQuestionValidity 574
23.3Regression-BasedResponseProbing 577
23.4Example1:GeneralizedTrust 579
23.5Example2:FearofCrime 580
23.6Data 581
23.7Discussion 586 References 588
24TheInterplayBetweenSurveyResearchandPsychometrics, withaFocusonValidityTheory 593 BrunoD.ZumboandJosé-LuisPadilla
24.1Introduction 593
24.2AnOver-the-ShoulderLookBackatValidityTheoryandValidation PracticeswithanEyetowardDescribingContemporaryValidity Theories 595
24.3AnApproachtoValiditythatBridgesPsychometricsandSurvey Design 602
24.4ClosingRemarks 606 References 608
25Quality-DrivenApproachesforManagingComplexCognitive TestingProjects 613 MarthaStapleton,DarbySteiger,andMaryC.Davis
25.1Introduction 613
25.2CharacteristicsoftheFourCognitiveTestingProjects 614
25.3IdentifyingDetailed,Quality-DrivenManagementApproachesfor QualitativeResearch 615
25.4IdentifyingPrinciplesforDevelopingQuality-DrivenManagement Approaches 616
25.5ApplyingtheConceptsofTransparencyandConsistency 617
25.6The13Quality-DrivenManagementApproaches 618
25.7DiscussionandConclusion 632 References 634
26UsingIterative,Small-ScaleQuantitativeandQualitative Studies:AReviewof15YearsofResearchtoRedesigna MajorUSFederalGovernmentSurvey 639 JoannePascale
26.1Introduction 639
26.2MeasurementIssuesinHealthInsurance 641
26.3MethodsandResults 645
26.4Discussion 660
26.5FinalReflections 663 References 664
27ContrastingStylizedQuestionsofSleepwithDiaryMeasures fromtheAmericanTimeUseSurvey 671 RobinL.Kaplan,BrandonKopp,andPollyPhipps
27.1Introduction 671
27.2TheSleepGap 672
27.3ThePresentResearch 674
27.4Study1:BehaviorCoding 675
27.5Study2:CognitiveInterviews 678
27.6Study3:QuantitativeStudy 682
27.7Study4:ValidationStudy 686
27.8GeneralDiscussion 689
27.9ImplicationsandFutureDirections 692 References 692
28QuestionnaireDesignIssuesinMailSurveysofAllAdultsina Household 697
DouglasWilliams,J.MichaelBrick,W.ShermanEdwards,andPamela Giambo
28.1Introduction 697
28.2Background 698
28.3TheNCVSandMailSurveyDesignChallenges 699
28.4FieldTestMethodsandDesign 704
28.5OutcomeMeasures 706
28.6Findings 708
28.7Summary 716
28.8Discussion 716
28.9Conclusion 719
References 720
29PlanningYourMultimethodQuestionnaireTestingBento Box:ComplementaryMethodsforaWell-BalancedTest 723 JakiS.McCarthy
29.1Introduction 723
29.2AQuestionnaireTestingBentoBox 725
29.3ExamplesfromtheCensusofAgricultureQuestionnaireTesting BentoBox 733
29.4Conclusion 743
References 744
30FlexiblePretestingonaTightBudget:UsingMultiple DependentMethodstoMaximizeEffort-Return Trade-Offs 749
MattJans,JodyL.Herman,JosephViana,DavidGrant,RoycePark,Bianca D.M.Wilson,JaneTom,NicoleLordi,andSueHoltby
30.1Introduction 749
30.2EvolutionofaDependentPretestingApproachforGenderIdentity Measurement 752
30.3AnalyzingandSynthesizingResults 759
30.4Discussion 764 Acknowledgments 766
References 766
Index 769
ListofContributors
AntuaneAllen HealthAnalytics,LLC SilverSpring,MD,USA
Jod’Ardenne
TheNationalCentreforSocial Research London,UK
DorothéeBehr SurveyDesignandMethodology, GESIS
LeibnizInstitutefortheSocial Sciences Mannheim,Germany
MichaelBraun SurveyDesignandMethodology, GESIS
LeibnizInstitutefortheSocial Sciences Mannheim,Germany
J.MichaelBrick Westat Rockville,MD,USA
SarahBrockhaus InstituteforStatistics,Ludwig MaximilianUniversityofMunich Munich,Germany and MannheimCentreforEuropean SocialResearch(MZES) UniversityofMannheim Mannheim,Germany
DyneshaBrooks MontgomeryCountyPublicSchools Rockville,MD,USA
IanBrunton-Smith DepartmentofSociology UniversityofSurrey Guildford,UK
DebbieCollins TheNationalCentreforSocial Research London,UK
JamesM.Dahlhamer DivisionofHealthInterview
Statistics NationalCenterforHealthStatistics Hyattsville,MD,USA
TrineDale DepartmentofResearchand Development KantarTNS Oslo,Norway
MaryC.Davis
OfficeofSurveyandCensus Analytics
USCensusBureau Washington,DC,USA
DonA.Dillman DepartmentofSociology WashingtonStateUniversity Pullman,WA,USA
JenniferDykema UniversityofWisconsinSurvey Center UniversityofWisconsin-Madison Madison,WI,USA
StephanieEckman
RTIInternational Washington,DC,USA
W.ShermanEdwards Westat Rockville,MD,USA
RoryFitzgerald ESSERICHQ.City UniversityofLondon London,UK
StephanieFowler
OfficeoftheDirector,AllofUs ResearchProgram NationalInstitutesofHealth Bethesda,MD,USA
ScottFricker
OfficeofSurveyMethodsResearch BureauofLaborStatistics Washington,DC,USA
DanaGarbarski DepartmentofSociology LoyolaUniversity Chicago,IL,USA
EmilyGeisen RTIInternational ResearchTrianglePark,NC,USA
PamelaGiambo Westat Rockville,MD,USA
DavidGrant RANDCorporation SantaMonica,CA,USA
FelixHenninger MannheimCentreforEuropean SocialResearch(MZES) UniversityofMannheim Mannheim,Germany and DepartmentofPsychology UniversityofKoblenz-Landau Landau,Germany
JodyL.Herman TheWilliamsInstitute UCLASchoolofLaw LosAngeles,CA,USA
TemikaHolland DataCollectionMethodology& ResearchBranch,Economic StatisticalMethodsDivision USCensusBureau Washington,DC,USA
SueHoltby PublicHealthInstitute Oakland,CA
RachelHorwitz USCensusBureau Washington,DC,USA
MichaelHout DepartmentofSociology NewYorkUniversity NewYork,NY,USA
JonathanJackson DepartmentofMethodology LondonSchoolofEconomicsand PoliticalScience London,UK
MattJans ICF Rockville,MD,USA
LarsKaczmirek LibraryandArchive Services/AUSSDA UniversityofVienna Vienna,Austria and
ANUCentreforSocialResearchand Methods CollegeofArtsandSocialSciences AustralianNationalUniversity Australia and GESIS–LeibnizInstituteforthe SocialSciences,MonitoringSociety andSocialChange Mannheim,Germany
RobinL.Kaplan OfficeofSurveyMethodsResearch BureauofLaborStatistics Washington,DC,USA
FlorianKeusch DepartmentofSociology,Schoolof SocialSciences UniversityofMannheim Mannheim,Germany
PascalJ.Kieslich MannheimCentreforEuropean SocialResearch(MZES) UniversityofMannheim Mannheim,Germany and DepartmentofPsychology,Schoolof SocialSciences UniversityofMannheim Mannheim,Germany
BrandonKopp OfficeofSurveyMethodsResearch BureauofLaborStatistics Washington,DC,USA
xx ListofContributors
FraukeKreuter JointPrograminSurvey Methodology UniversityofMaryland CollegePark,MD,US; SchoolofSocialSciences UniversityofMannheim Mannheim,Germany; InstituteforEmploymentResearch Nuremberg,Germany
NicoleLordi PublicHealthInstitute Oakland,CA
AaronMaitland Westat Rockville,MD,USA
JakiS.McCarthy NationalAgriculturalStatistics Service
USDepartmentofAgriculture Washington,DC,USA
KatharinaMeitinger DepartmentofMethodologyand Statistics
UtrechtUniversity Utrecht,TheNetherlands
JoeMurphy RTIInternational Chicago,IL,USA
ElizabethNichols CenterforBehavioralScience Methods
USCensusBureau Washington,DC,USA
EricaOlmsted-Hawala CenterforBehavioralScience Methods
USCensusBureau Washington,DC,USA
KristenOlson DepartmentofSociology UniversityofNebraska-Lincoln Lincoln,NE,USA
YfkeOngena DepartmentofCommunication studies,CentreforLanguageand Cognition UniversityofGroningen,Groningen TheNetherlands
José-LuisPadilla FacultyofPsychology UniversityofGranada Granada,Spain RoycePark UCLACenterforHealthPolicy Research LosAngeles,CAUSA
JoannePascale CenterforBehavioralScience Methods
USCensusBureau Suitland,MD,USA
PollyPhipps OfficeofSurveyMethodsResearch BureauofLaborStatistics Washington,DC,USA
StanleyPresser SociologyDepartmentandJoint PrograminSurveyMethodology UniversityofMaryland CollegePark,MD,USA
MelanieRevilla RECSMUniversityPompeuFabra Barcelona,Spain
AmyAndersonRiemer DataCollectionMethodology& ResearchBranch,Economic StatisticalMethodsDivision USCensusBureau Washington,DC,USA
HeatherRidolfo ResearchandDevelopmentDivision NationalAgriculturalStatistics Service Washington,DC,USA
PaulScanlon CollaboratingCenterfor QuestionnaireDesignandEvaluation Research NationalCenterforHealthStatistics CentersforDiseaseControland Prevention UnitedStatesDepartmentofHealth andHumanServices Hyattsville,MD,USA
NoraCateSchaeffer DepartmentofSociology UniversityofWisconsin-Madison Madison,WI,USA
and UniversityofWisconsinSurvey Center UniversityofWisconsin-Madison Madison,WI,USA
MalteSchierholz InstituteforEmploymentResearch Nuremberg,Germany
TomW.Smith NORC,UniversityofChicago Chicago,IL,USA
JoleneD.Smyth DepartmentofSociology UniversityofNebraska-Lincoln Lincoln,NE,USA
MarthaStapleton InstrumentDesign,Evaluation,and Analysis(IDEA)Services Westat,Rockville,MD,USA
DarbySteiger InstrumentDesign,Evaluation,and Analysis(IDEA)Services Westat,Rockville,MD,USA
PatrickSturgis DepartmentofMethodology LondonSchoolofEconomicsand PoliticalScience London,UK
JaneTom Independentscholar USA
xxii ListofContributors
DanieleToninelli DepartmentofManagement EconomicsandQuantitative Methods
UniversityofBergamo Bergamo,Italy
RogerTourangeau Westat Rockville,MD,USA
ShirleyTsai OfficeofTechnologySurvey Processing BureauofLaborStatistics Washington,DC,USA
SanneUnger CollegeofArtsandSciences LynnUniversity BocaRaton,FL,USA
JosephViana LosAngelesCountyDepartmentof PublicHealth LosAngeles,CA,USA
HeidiWalsoe DepartmentofResearchand Development KantarTNS Oslo,Norway
DouglasWilliams Westat Rockville,MD,USA
GordonB.Willis BehavioralResearchProgram DivisionofCancerControland PopulationSciences,CancerInstitute National NationalInstitutesofHealth Rockville,MD,USA
AmandaWilmot Westat Rockville,MD,USA
BiancaD.M.Wilson TheWilliamsInstitute,UCLASchool ofLaw LosAngeles,CA,USA
TingYan StatisticsandEvaluation SciencesUnit Westat Rockville,MD,USA
DianaZavala-Rojas DepartmentofPoliticaland SocialSciences UniversitatPompeuFabra RECSM,Barcelona,Spain
BrunoD.Zumbo Measurement,Evaluation,and ResearchMethodologyProgram,and TheInstituteofAppliedMathematics UniversityofBritishColumbia Vancouver,Canada
Preface
ThechaptersinthisvolumewereallinvitedpapersattheSecondInternational ConferenceonQuestionnaireDesign,Development,Evaluation,andTesting (QDET2),whichwasheldinMiami,Florida,November9–13,2016.Overthe courseoffourdays,QDET2broughttogether345attendeesfrom29countries,unitedbyaninterestinimprovingtheartandscienceofproducingsurvey instrumentsthatcollecthigh-qualitydata.
PlanningforQDET2beganin2013,whenparticipantsofaninternational workinggroupknownasQUEST(QuestionEvaluationStandards)discussed thepossibilityofsuchaconferenceandinvitedAmandaWilmottoserve asChair.Sincethelate1990s,QUESThasconsistedofasmallgroupof researchers,mainlyfromstatisticalagenciesandacademicinstitutions,who meetabouteverytwoyearstodiscussguidelinesandpotentialadvancesto questionevaluationpractices.ManyoftheregularQUESTparticipantshad alsoparticipatedinthefirstQDETconference,heldin2002.QDET,asthefirst majorconferenceonitstopic,playedanimportantroleincreatingabroad understandingofthetoolsavailableforquestionnairedevelopment,evaluation, andtesting.Manyofthecommonmethodshadevolvedsomewhatdifferently invariousinstitutions,notnecessarilybasedonthesameassumptionsorusing thesameterminology,andwithoutasharedunderstandingofthestrengths andweaknessesofalternativeapproaches.TheQDETconferencehelped tobuildthoseunderstandings,andthebookresultingfromtheconference (Presseretal.2004)wasthefirsttocompilethecollectivemethodological guidanceintoonevolume.
Morethanadecadehadpassedsincethispivotalconference,anditseemed likelythatthefieldhadsufficientlymaturedsincethentojustifyasecondone. WhereasthefirstQDEThadhelpedtoestablishthekeyelementsofthetoolkit, thesecondwoulddocumentthecontinuingevolutionofthesemethods,aswell asnewadditions.Itwouldalsofostermorecriticalassessmentsoftherelative contributionsofthevariousmethods.Also,thescopeoftheconferencewas expandedtoincludequestionnairedesign;afterall,questionnairesneedtobe createdbeforetheycanbeevaluatedandimproved.
Atthesametime,someofthemostimportantaspectsofthefirstQDET wereretainedorevenenhanced.Mostcritically,thisincludedacallforinternationalparticipation,togeneratethewidestpossibleincorporationanddisseminationofnewideas.Theconferencewasdesignedtoshareknowledge throughacombinationofpresentations–keynoteaddresses,formalpapers, electronicposters–aswellasworkshops,demonstrations,exhibits,courses, andless-formalnetworkingopportunities.Callsforparticipationbeganin2015 andwereencouragedacrosstheacademic,government,andprivatesectors.
Atfirstglance,paperproposalsseemedtospanmuchofthesamemethodologicalterritoryastheoriginalQDET–forexample,includingnumerous papersabouttheuseofprobestounderstandinterpretations,andexperiments toidentifytheeffectofparticulardesignvariations.Closerinspectionrevealed thatagreatdealhadchanged,notnecessarilyintermsoftheintroductionof completelynovelmethods,butthroughnewapplicationsoftheminresponse tomajorchangesinthesurveylandscape.
Oneofthemostsignificantofthesechanges,describedbyDillmanin Chapter2ofthisvolume,wasasubstantialmoveawayfromtraditional interviewer-basedsurveystowardself-administeredmodes.Suchmovement increasestheburdenonquestions,andthechallengeforquestionnaire designers,asquestionsmustbeunderstandablewithoutthepresenceofan interviewtohelpnavigateuncertaintiesofmeaning.Thesechallengesare compoundedbytheproliferationofmobiledevices,withsmallscreensizes thatsignificantlyconstrainquestionlengthandcomplexity.Mostresearchers wouldnotchoosesuchdevicesastheprimaryresponsemodeiftheycould avoidit–butincreasingly,obtainingrespondentcooperationrequiresacceptingtheirpreferredmodeforinteractingwiththelargerworld.Theincreasein self-administrationingeneral,andofsmallscreensinparticular,hasrequired methodologiststopaygreaterattentiontovisualdesignprincipleswhen developingquestionnaires.
Thesechangeshavealsorequiredmethodologiststoadjusttheirapproaches forquestionnaireevaluation.Oneexampleisanincreasedprominenceof directobservationofself-response,andtheincorporationofmorestandardizedprobesintoself-administeredinstruments,relativetotraditional interviewer-administeredtestingprotocols.Anotherisanincreaseinthe prominenceofusabilitytesting,whichfocusesonrespondentinteraction withthesurveyinstrument,andisanimportantcomplementtothecognitive testingthatfocusesmoreonmattersofquestionwording.Electronicdevices alsocreateparadata–informationabouttheresponseprocessitself,suchas howrespondentsnavigatethroughtheinstrument–whichmayhaveuseful implicationsforevaluatingquestionnaires.
AnothersignificantchangesincethefirstQDETconferenceisthatratesof surveyparticipationhavecontinuedtodeclineprecipitously.Infact,inthe earlystagesofQDET2planning,weconsideredthepossibilitythattraditional
questionnairesmayplayasignificantlyreducedroleinafuturedominated byBigDataandotheralternativestosurveys.Althoughweconcludedthat questionnairesareunlikelytobecompletelyreplacedanytimesoon,evenas otherdatasourcesriseinprominence,thechallengesofmaximizingrespondentcooperationareimpossibletoignore.Consequently,surveyburdenwasa prominenttopicatQDET2.Severalchaptersinthebookdealwiththistopic explicitly–definingitandunderstandingitsroleinsurveyparticipation. However,thetopicisalsointhebackgroundofmanyotherchapters,as contributorsconsidertherelationshipbetweendesignfeaturesandvarious formsofsatisficing,aswellasquestionnaireevaluationstrategiesthatrequire minimaladditionaleffortfromrespondents.
Surveyquestionnairescontinuetoplayauniqueroleinproducingdataon attitudesandbehaviorsforwhichthereisnoviablealternative.Ourneedfor suchdatahasneverbeengreater–fordistinguishingactualpublicopinionfrom speculativecommentary,forproducingdemographicandbehavioraldatato guidepolicyplanningandresourceallocation,andfordevelopingcomparative measuresofwell-beingonaglobalscale–amongothers.Itisourhopethat thechapterspresentedherewillhelppractitionersaddressrecentchangesand challenges,andenhancetheviabilityofsurveyquestionnairesforyearstocome. Withthosegoalsinmind,wehaveorganizedthisbookintofivemajorsections.
Thefirstsection,“AssessingtheCurrentMethodologyforQuestionnaire Design,Development,Testing,andEvaluation,”beginswithtwobroad overviews.Willis’schapter,basedonhiskeynoteaddressatQDET2,offers analysisofthe“trends,development,andchallenges”facingthefieldfrom theperspectiveofaresearcher-practitionerimmersedinquestionnaireevaluationforseveraldecades.Dillman’schapteroffersthesomewhatdifferent perspectiveofamethodologistwhoseworkcentersmoreondesignand surveyparticipationthantestingmethodsperse.Bothpointtoimportant futuredevelopments.Forexample,Willisconsiderstheapplicationofcurrent methodstoemergingdatasources,whileDillmancallsforthefieldtoexpand itsfocustorespondentmotivation,visualaspectsofquestionnaires,and multimodeevaluation.Theremainingchaptersinthissectionofferbroad assessmentsofthevariousmethodscurrentlyavailablewhileaddressing severalpracticalquestions:howdowedecideamongcurrentmethods,what candifferentmethodstellusaboutresponsevalidity,andhowdowemake senseofapparentlycontradictoryfindings?
Chaptersinthesecondsection,“QuestionCharacteristics,ResponseBurden,andDataQuality,”centermoreonmattersofdesignthanevaluation methodology.BoththeDykema,Schaeffer,etal.andDahlhamer,Maitland, etal.chaptersbroadlyconsidertherelationshipbetweenquestioncharacteristicsanddataquality–collectivelyprovidinginsightintohowvariousdesign decisionsaffectdifferentindicators,andprovidinganextensiveframeworkto guidefutureinvestigations.Thenexttwochaptersconsidertherelationship
betweenquestionnaireattributesandperceptionsofburden(Yan,Fricker, andTsai)orrespondentbehaviorsthatseemtoindicateburden(Kreuter, Eckman,andTourangeau),whilealsoconsideringtheoverarchingissueof whatburdenactuallyis.Thefinaltwochaptersinthissectionfocusonhow particularquestionnairecharacteristicsaffectparticularresponsebehaviors. Findingsinthesechaptersmaybeinterestingnotonlyfortheirspecificdesign implications,butalsoasmodelsforadditionalresearchonrelatedtopics.
Chaptersinthethirdsection,“ImprovingQuestionnairesontheWeband MobileDevices,”addressmethodologicalissuesarisingfromincreaseduse ofself-responsethroughelectronicmodesofdatacollection.Aswithother sectionsofthebook,thisbeginswithrelativelybroadoverviews,including GeisenandMurphy’sreviewoftheoveralltoolkitforwebandmobilesurvey pretesting,andNicholsetal.’soverviewofusabilitytestingmethodsforonline instruments.Empiricalstudiesofmorespecifictopicsfollow,respectively addressingthechallengesofdesigninginstrumentswithvariousscreensizes; anapplicationofparadatafrommousemovementstoidentifymeasurement issues;andtheuseofwebprobing,aresponsetobothproliferationofweb surveysandtheneedforlargerpretestingsamples.
Asitstitlesuggests,thefourthsection,“Cross-CulturalandCross-National QuestionnaireDesignandEvaluation,”takesontheparticularchallengesof measurementacrossnationsandcultures.Thechapterswithinaddressconceptualconsiderationsofdesign(Smith),pretestingmethodology(Fitzgerald andZavala-Rojas;Behr,Meitingeretal.),andboth,inpracticalapplication (Wilmot).Developingmeasuresthatfunctionsobroadlyissurelyoneofthe biggestmeasurementchallengesmethodologistscurrentlyface.
Finally,thefifthsection,“ExtensionsandApplications,”bringstogether chaptersofferingmethodologicalinnovation,andexamplesofnoveldesign andevaluationmethodologyinpractice.Offeringsinthissectionarequite wide-ranging,includingnewanalyticapplicationstofamiliarverbalprotocols (Sturgis,Brunton-Smith,andJackson),anefforttobridgeconceptswiththe sisterdisciplineofpsychometrics(ZumboandPadilla),developingmanagementstandardsforquestionevaluationprojects(Stapleton,Steiger,andDavis), inadditiontoaseriesofstudiesthatcombine,enhance,andapplymethods indifferentwaystoaddressthecontinuallyevolvingchallengesofcollecting high-qualitydatathroughsurveyquestions.
Inassemblingthisbook,ourobjectivewastocreatenotsimplyaproceedingsvolumeoftheQDET2conference,butratheranintegratedvolumereflectingthecurrentstateofthefieldthatpointstopromisingnewdirections.We selectedchaptersforinclusionanddesignedoureditorialprocesswiththatend inmind.Eachchapterwasreviewedbyatleasttwoeditors,oneprimaryand onesecondary.Editorswerechosenpurposivelybasedonkeyinterests,butalso mixedasmuchaspossiblegiventhoseinterests,suchthatveryfewchapters werereviewedbythesamepair.Webelievethatthisresultedinahighdegreeof
Preface xxvii
editorialcollaborationandinterminglingofideas,suchthatthebookasawhole reflectsourcollectiveguidance.Wearehopefulthattheresultsareusefuland interesting,butofcoursedeferthefinalevaluationtothereader.
Forus,thecompletionofthisbookmarkstheendoffiveyearsofcontinuousinvolvementwiththeQDET2conference.Wearegratefultomanyfor supportingusthroughoutthiseffortthatconsumedmuchtimeandenergy, includingouremployinginstitutions,colleagues,andfamilies.Wealsogratefullyacknowledgethepatienceofthecontributingauthorsthroughtheextensiveeditorialprocess.QDET2mightnothavehappenedatallwithoutthefinancialandlogisticalsupportoftheAmericanStatisticalAssociation;weareall gratefulfortheircommitmentandconfidenceintheeffort.Finally,wethankall whocontributedtotheQDETconference–toomanytonameindividually,but certainlyincludingallwhoservedontheorganizingandprogramcommittees, inadditiontothesponsors,volunteers,presenters,andofcoursetheattendees.
PaulC.Beatty DebbieCollins LynKaye
Jose-LuisPadilla GordonB.Willis AmandaWilmot
Disclaimer
Anyviewsexpressedherearetheauthors’anddonotnecessarilyreflectthose ofourinstitutions,inparticulartheUSCensusBureau.
Reference
Presser,S.,Rothgeb,J.M.,Couper,M.P.etal.(eds.)(2004). MethodsforTestingand EvaluatingSurveyQuestionnaires.Hoboken,NJ:Wiley.