theoxfordhandbookof EXPERIMENTAL SYNTAX
Editedby JONSPROUSE
GreatClarendonStreet,OxfordOX26DP, UnitedKingdom
OxfordUniversityPressisadepartmentoftheUniversityofOxford. ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,scholarship, andeducationbypublishingworldwide.Oxfordisaregisteredtrademarkof OxfordUniversityPressintheUKandincertainothercountries ©editorialmatterandorganizationJonSprouse2023 ©thechapterstheirseveralauthors2023
Themoralrightsoftheauthorshavebeenasserted FirstEditionpublishedin2023
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedin aretrievalsystem,ortransmitted,inanyformorbyanymeans,withoutthe priorpermissioninwritingofOxfordUniversityPress,orasexpresslypermitted bylaw,bylicenceorundertermsagreedwiththeappropriatereprographics rightsorganization.Enquiriesconcerningreproductionoutsidethescopeofthe aboveshouldbesenttotheRightsDepartment,OxfordUniversityPress,atthe addressabove
Youmustnotcirculatethisworkinanyotherform andyoumustimposethissameconditiononanyacquirer PublishedintheUnitedStatesofAmericabyOxfordUniversityPress 198MadisonAvenue,NewYork,NY10016,UnitedStatesofAmerica
BritishLibraryCataloguinginPublicationData Dataavailable
LibraryofCongressControlNumber:2022935323
ISBN978–0–19–879772–2
DOI:10.1093/oxfordhb/9780198797722.001.0001
Printedandboundby CPIGroup(UK)Ltd,Croydon,CR04YY LinkstothirdpartywebsitesareprovidedbyOxfordingoodfaithand forinformationonly.Oxforddisclaimsanyresponsibilityforthematerials containedinanythirdpartywebsitereferencedinthiswork.
Preface
Listoffiguresandtables
PARTIJUDGMENTMETHODSIN
1.Acceptabilityjudgments 3 JonSprouse
2.Acceptabilityjudgmentsofbindingandcoreference: Methodologicalconsiderations 29 ElsiKaiserandJeffreyRunner
3.(Quantifier)scopejudgments
KrisztaEszterSzendrői
4.Experimentalsyntaxandlinguisticfieldwork
PARTIIACQUISITIONMETHODSIN
5.Behavioralacquisitionmethodswithinfants 137 LaurelPerkinsandJeffreyLidz
6.Behavioralacquisitionmethodswithpreschool-agechildren 171 KristenSyrett
7.Modelingsyntacticacquisition 209
LisaS.Pearl
8.Artificiallanguagelearning 271
JenniferCulbertson AnnotatedbibliographyforPartII 301
PARTIIIPSYCHOLINGUISTICMETHODS INSYNTACTICTHEORY
9.Self-pacedreading 313
MasayaYoshida
10.Eye-trackingandexperimentalsyntax 333
DaveKushandBrianDillon
11.Speed–accuracytrade-offmodelinganditsinterfacewith experimentalsyntax 363
StephaniForaker,IanCunnings,andAndreaE.Martin
12.Formalmethodsinexperimentalsyntax 393
TimHunter
13.Investigatingsyntacticstructureandprocessinginthe auditorymodality 453
MaraBreenandKatyCarlson
14.Language-processingexperimentsinthefield 491
MatthewWagersandSandraChung AnnotatedbibliographyforPartIII 513
PARTIVNEUROLINGUISTICMETHODSIN
15.Electrophysiologicalmethods 533
JonSprouseandDiogoAlmeida
16.Hemodynamicmethods 559
JonathanR.Brennan
Preface
Thefieldofsyntaxhasalwaysbeeninterdisciplinary.Partofthisissimplythenatureof cognitivescience—theimmensityoftheproblemposedbyhumancognitionrequires aconcertedeffortfrommultipledisciplines.Andpartofthisisthenatureofsyntactic theory:Itmediatesbetweensoundandmeaning,itisatheoryoftherepresentations constructedduringsentence-processing,anditisatheoryoftheendstateforlanguage acquisition.Astechnologyhasadvanced,sotoohavethemethodsthatsyntacticians havebroughttobearonthecentralquestionsofthefield.Thepasttwodecadesinparticularhaveseenanexplosionintheuseofvariousexperimentalmethodsforprobing thesyntaxofhumanlanguages.ThisHandbookisanattempttobringthesestrands ofresearchtogetherintoasinglevolume.IhavethreegoalsforthisHandbook:(i)to providehigh-levelreviewsoftheexperimentalworkthathasbeendrivingthefieldof experimentalsyntax,(ii)toinspirenewresearchthatwillpushtheboundariesofthe theoryofsyntax,and(iii)toprovidehigh-levelmethodologicalguidanceforresearchers whowishtoincorporateexperimentalmethodsintotheirownresearch.Ihopereaders willagreethatthecontributorstothisvolumehavecreatedchaptersthatsucceedinall threegoals.
Forthishandbook,Ihaveintentionallydefined experimentalsyntax inthebroadest possibleterms—astheuseofany(andall)experimentalmethodsinserviceofsyntactic theory.Iamawarethattheterm experimentalsyntax issometimesusedinanarrower sensethatismoreorlesssynonymouswith formalacceptabilityjudgmentexperiments (Ihaveuseditthatwaymyselfinmyownwork),butIbelievethissynonymyismerely asymptomoftheimportantrolethatacceptabilityjudgmentsplayinsyntactictheory, andnotameaningfuldelimiterofthetypesofmethodsthatsyntacticianscanprofitably employintheirresearch.Thespaceofpossiblemethodsislarge—toolargeforanysingle volume.InassemblingthisHandbook,Ihavechosentofocusonmethodsthatare(i) relativelywell-understood,(ii)relativelypracticalintermsoftheequipmentrequired, and(iii)relativelylikelytoyieldinformationthatisrelevanttosyntactictheory.Allof thesechoicesaresubjective.Idonotintendtheexclusionofanygivenmethodtomean thatitdoesnot,orcouldnot,fallunderthebroaddefinitionofexperimentalsyntax. Infact,whatIhopethisHandbookshowsisthatthisbroaderdefinitionofexperimentalsyntaxisstillinitsinfancy.Wehavenotyetexploredallofthemethodsthatcould potentiallycontributetotheoriesofsyntax,norhaveweseenthefullpotentialofthe methodsthatwehaveexplored.Assuch,thishandbookisaspirational—itissimultaneouslyasnapshotoftheknowledgewehavecollectedtodateandapointertothekind ofworkthatwillbepossibleinthefuture.
xpreface
Onecriticalcomponentofallexperimentalworkisalinkinghypothesis—ahypothesisthatlinkstheobserveddatatotheunobservedtheoreticalconstructsthatareunder investigation.Inexperimentalsyntax,weneedhypothesesthatlinkeachofthemethods discussedinthishandbookbacktosyntactictheory.Becausethereislikelynomethod thatprovidesadirectlinktosyntactictheory(atleastatourcurrentleveloftechnology), foreachandeverymethodinthishandbook,creatingalinkinghypothesisbetween thedataandsyntactictheoryentailscreating(orinvestigating)alinkinghypothesis betweensyntactictheoryandanothercomponentofthetheoryoflanguage,suchas thetheoryofsentence-processingorthetheoryoflanguageacquisition.Manyofthe chaptersinthisHandbookdiscussthisissueindetail,soIwillnotbelaborthepoint here.Thepracticalconsequenceofthisisthatthereisatheoreticalthemethroughoutthishandbook—thelinkingofsyntactictheorytoothercomponentsofacomplete theoryoflanguage.IhaveorganizedtheHandbookaroundthistheme.Therearefour sections,eachcorrespondingtothelinkinghypothesesnecessarytoleveragethemethodsineachsectioninserviceofsyntactictheory:(i)judgmentmethods,whichrequire alinkbetweenthetheoryofofflinejudgmentsandsyntax,(ii)acquisitionmethods, whichrequirealinkbetweenthetheoryofthelanguage-acquisitionprocessandsyntax, (iii)psycholinguisticmethods,whichrequirealinkbetweenthetheoryofsentenceprocessingandsyntax,and(iv)neurolinguisticsmethods,whichrequirealinkbetween neurobiology,sentence-processing,andsyntax.
AfewnotesontheorganizationoftheHandbookareinorder.First,IhaveaskedOxfordUniversityPresstokeepthereferencesforeachchapterwiththatchapter(andnot inaglobalreferencelistattheendoftheHandbook).Myhopeisthatthiswillallowthe chaptersinthisHandbooktotrulyserveasaguideforexploringthepotentialtheoreticalcontributionsthateachmethodcanmaketosyntax,withthereferencelistsserving asafirstreadinglist.Second,becausethesemethodsarenotusefuliftheycannotbe learnedbynewresearchers,Ihaveaskedthecontributorsofeachchaptertocreatean annotatedlistofresourcesforlearningthemethodintheirchapter.Ihavecollatedthe listsbysection,compilingthemintofourstand-alonechaptersthatoccurattheendof eachsection.Finally,Ihaveaskedeachcontributortowriteamini-essayaboutwhat theyseeasthefutureofexperimentalsyntax.Myhopeisthatthesemini-essayswill provideinspirationtoreaderswhoareconsideringadoptingexperimentalsyntaxinto theirownresearchprograms,andalsoserveasasortoftimecapsulebywhichwecan measuretheprogressofthefieldinfutureyears.Ihavecollectedthesemini-essaysinto asinglechapterattheendofthehandbook.
Thisvolumecouldnotexistwithoutthetalent,energy,andeffortofinnumerable colleagues.First,IwouldliketothankthecommissioningeditoratOxfordUniversity Press,JuliaSteer,forlayingthefoundationofthisvolumebyencouragingmetoexplore abroaddefinitionofexperimentalsyntaxratherthananarrowreviewofworktodate. Second,Iwouldliketothankeachofthecontributorsforsharingboththeirvisions andtheirexpertise.Ifthisvolumesucceedsinanyofitsthreegoals,itwillbebecause
prefacexi
oftheirhardworkanddedication,bothinwritingtheirchaptersandindoingthekind ofresearchthatpushestheboundariesofthefield.Finally,Iwouldliketothankeveryonewhohassupportedmethroughoutmycareer—advisors,collaborators,colleagues, students,family,andfriends.Scienceisacommunityeffort.AndIamgratefulbeyond wordsforthecommunitythatIhavesomehowbeengiveninthislife.
Listoffiguresandtables
Figures
1.1Thetwopredictionsofthe2×2designforwhether-islands(leftpanel andcenterpanel),andtheobservedresultsofanactualexperiment (rightpanel) 17
1.2Threedemonstrationsofthecontinuousnatureofacceptability judgments 21
2.1SceneverificationdisplayfromtheexperimentbyKaiseretal.(2009) 43
2.2PictureselectiondisplayfromtheexperimentbyKaiseretal.(2009) 44
3.1OutcomeofexampleteststoryfromConroyetal.’s(2009)TVJTtask 87
7.1ModeloftheacquisitionprocessadaptedfromLidzandGagliardi(2015) 213
11.1SATfunctionforonecondition,illustratingthethreephasesof processing 369
11.2IdealizeddifferencesinthethreephasesoftheSATfunctionsfortwo conditions 370
11.3Idealizeddifferencesinthefinishingtimedistributionscorrespondingto theSATdifferencesshowninFig.11.2 370
12.1Wecanusesurprisaltoformulatealinkinghypothesiswhich,taken togetherwithaprobabilitydistributionoversentences,produces empiricalpredictionsaboutsentencecomprehensiondifficulty 401
12.2Sincesurprisalcanactasatestofprobabilitydistributionsand probabilitydistributionscanbeseenasconsequencesofhypothesized grammars,surprisalcanactasatestofhypothesizedgrammars 402
12.3Graphicalillustrationoflc-predictandlc-connect 431
13.1ExampleitemfromBreenetal.(2010)designedtoelicitnaturalistic productions 459
14.1AsampleitemsetfromSussmanandSedivy(2003) 494
14.2SomeculturallyspecificillustrationscreatedfortheChamorro PsycholinguisticsnaProject 497
16.1TheBOLDsignal 562
xivlistoffiguresandtables
16.2Schematicrepresentationofsyntax-relatedbrainregionsoftheleft hemisphere 566
16.3Linkinghypothesesconnectpropertiesofthegrammarwithneural signals 582
17.1AcceptabilityjudgmentdatareproducedfromLinebargeretal.(1983) 611
17.2Functionalneuroanatomyoflanguageandworkingmemory(WM)as relevanttoourproposal 618
17.3Aschematicofhealthyandagrammaticsentence/phraseproduction withrespecttothedorsalandventralpathwaystoarticulation 622
Tables
3.1Percentageofsurfacescoperesponseforcomprehensionquestion 64
3.2Summaryofexperimentalfindingsofthelanguageacquisitionstudies reviewedinthispaper 74
4.1Experimentalparadigmforstudyingsubjectpreference, morphologicallyergativelanguages 108
7.1ThequalitativefitYangdiscoveredbetweentheunambiguousdata advantage(Adv)perceivedbyaVarLearnerinitsacquisitionalintake andtheobservedageofacquisition(AoA)inchildrenforsixparameter valuesacrossdifferentlanguages 236
7.2Optionalinfinitiveexamplesinchild-producedspeechindifferent languages,andtheirintendedmeaning 236
8.1Summaryofkeyartificiallanguagelearningmethods 276
12.1Afirstillustrationofbottom-upparsing 417
12.2Theeffectofcenter-embeddingonbottom-upparsing 420
12.3Theeffectofleft-embeddingonbottom-upparsing 421
12.4Theeffectofright-embeddingonbottom-upparsing 423
12.5Afirstillustrationoftop-downparsing 424
12.6Theeffectofcenter-embeddingontop-downparsing 426
12.7Theeffectofleft-embeddingontop-downparsing 427
12.8Theeffectofright-embeddingontop-downparsing 428
12.9Afirstillustrationofleft-cornerparsing 430
12.10Theeffectofcenter-embeddingonleft-cornerparsing 433
12.11Theeffectofleft-embeddingonleft-cornerparsing 434
12.12Theeffectofright-embeddingonleft-cornerparsing 435
16.1Syntacticrepresentationsstandinamany-to-manyrelationshipwith sentence-processingoperations 564
16.2Summaryofbrainregionsrelatedtosyntax 565
16.3Examplesofphrasesandsentencesmadeofofrealwordsornonsense pseudo-words 572
16.4Hypotheticalcountsforutteranceswithintransitiveverbs 580
17.1Examplesofstimulifromeachcondition 612
Thecontributors
DiogoAlmeida isanAssociateProfessorofPsychologyatNewYorkUniversityAbu Dhabi.Hisresearchcapitalizesonbehavioralandelectrophysiologicaldata(EEGand MEG)toinvestigatequestionsaboutlinguisticrepresentationsandprocessesatmultiplelevels(phonology,morphology,andsyntax).HeholdsanMAincognitivescience fromtheÉcoledesHautesÉtudesenSciencesSociales(2003)andaPhDinlinguistics fromtheUniversityofMaryland(2009),andcompletedhispost-doctoraltrainingat theUniversityofCalifornia,Irvine.
MaraBreen isanAssociateProfessorintheDepartmentofPsychologyandEducation atMountHolyokeCollege.Herresearchexplorestheroleofprosodyinspeechperceptionandproduction.Usingbehavioraltechniques,eye-tracking,andevent-related potentials,sheinvestigateshowspeakersuseprosodiccuestoprovidemeaning,how listenersuseprosodytocomprehend,andhowimaginedprosodyduringreadingcan affectunderstanding.Inaddition,sheexploreshowsoundcuesareprocessedsimilarly acrossmusicandlanguage.Herworkappearsinjournalssuchas Cognition,Journal ofMemoryandLanguage,JournalofExperimentalPsychology:General,and Language, Cognition,andNeuroscience.
JonathanR.Brennan isanAssociateProfessorofLinguisticsandPsychologyatthe UniversityofMichigan,wherehedirectstheComputationalNeurolinguisticsLaboratory.HereceivedaPhDinlinguisticsfromNewYorkUniversityin2010andcompleted post-doctoraltrainingattheChildren’sHospitalofPhiladelphia.
KatyCarlson isaProfessorofEnglishintheDepartmentofEnglishatMoreheadState University.Herresearchconcentratesprimarilyonhowprosodycanaffectsentenceprocessing,withspecialinterestsinfocuseffectsinellipsissentencesandprosodic influencesonattachment.Shestudiesbothpitchaccentsandprosodicboundaries,and haspublishedinjournalssuchasLanguageandSpeech,Glossa,andLanguage,Cognition andNeuroscience.
SandraChung isDistinguishedProfessor(emerita)ofLinguisticsattheUniversity ofCalifornia,SantaCruz.Herresearchinvestigatestheoreticalissuesinsyntaxand otherareasthroughfieldworkonChamorroandotherAustronesianlanguages.Shehas collaboratedonresearchinsemanticswithWilliamA.Ladusaw,andonresearchinpsycholinguisticswithMatthewWagers.Since2009shehasbeeninvolvedinacommunitybasedeffortintheNorthernMarianaIslandstoupgradethedocumentationofthe Chamorrolanguage.
xviiithecontributors
JenniferCulbertson isaProfessorintheDepartmentofLinguisticsandEnglishLanguageattheUniversityofEdinburgh,andafoundingmemberoftheCentrefor LanguageEvolution.Sheusesexperimentalandcomputationaltoolstoinvestigate howthehumancognitivesystemshapeslinguistictypology.ShereceivedtheRobert J.GushkoPrizeforOutstandingDoctoralDissertationsinCognitiveSciencein2012, waselectedtotheYoungAcademyofEuropein2019,andcurrentlyholdsaEuropean ResearchCouncilStartingGrant.
IanCunnings isanAssociateProfessorofPsycholinguisticsintheSchoolofPsychologyandClinicalLanguageSciencesattheUniversityofReading,UK.Hismainresearch interestsareinsentenceanddiscourse-processingindifferentpopulationsofspeakers. Hisworkhasexaminedthememory-encoding,storage,andretrievalmechanismsthat subservetheresolutionofdifferenttypesoflinguisticdependenciesduringlanguage comprehension.Hismostrecentresearchexamineshowthesedifferentmemoryoperationscaninformourunderstandingofthefactorsthatinfluencesuccessfulsentence comprehensioninnativeandnon-nativespeakerpopulations.
BrianDillon isanAssociateProfessorintheDepartmentofLinguisticsattheUniversityofMassachusetts,Amherst.Hisresearchfocusesonadultsentencecomprehension, aimingtounderstandhowlinguisticconstraintsaredeployedinreal-timetoconstrain sentenceprocessing.Inhiswork,heintegratesinsightsfromlinguistictheorywith process-levelcognitivemodels,withaparticularinterestintheprocessingofagreement andpronominalreference.
StephaniForaker isanAssociateProfessorintheDepartmentofPsychologyatState UniversityofNewYorkCollegeatBuffalo,specializingincognitionandpsycholinguistics.Hermainresearchinterestsareinsentenceanddiscourseprocessing,focusingon theroleofmemoryandfocalattention.Shehasusedthespeed–accuracytradeoff(SAT) proceduretoinvestigatelong-distancedependenciesandpronounresolution,training underBrianMcElree,whopioneeredtheapplicationofSATtopsycholinguisticissues. Hercurrentresearchexaminesthecontributionofhandgesturesaspartofencoding, storage,andretrievaloperations,particularlyinanaphoraresolution.
TimHunter isanAssociateProfessorintheDepartmentofLinguisticsattheUniversityofCalifornia,LosAngeles.Thebulkofhisresearchusescomputationalperspectives toinvestigatetheformalpropertiesofnaturallanguagegrammar,withonemaingoal beingtoclarifytheconsequencesoftakinglinguistictheoriestobetestablecognitivehypothesis.Thislineofworkincludesstudiesconnectingminimalistsyntaxto experimentalworkinlanguage-processing,andstudiesoftherelationshipbetweendeterminers’truth-conditionsandverificationprocedures.Hehasalsoworkedonthe argument/adjunctdistinctionandthesyntaxofellipsis.
ElsiKaiser isaProfessorintheDepartmentofLinguisticsattheUniversityofSouthern California.ShereceivedherPhDinlinguisticsfromtheUniversityofPennsylvania,afteraBAinGermaniclanguagesandliteraturesfromPrincetonUniversityandanMAin
thecontributorsxix
psychologyfromtheUniversityofPennsylvania.Herresearchfocusesontheprocesses andrepresentationsinvolvedincomprehensionandproduction,especiallyindomains involvingmultipleaspectsoflinguisticrepresentation(syntax,semantics,pragmatics), suchasreferenceresolution.Shehasinvestigatedmultiplelanguages(e.g.Finnish,Estonian,French,German,andDutch,includingcollaborativeworkonBangla/Bengali, Hindi,Italian,Korean,Chinese,andVietnamese).
DaveKush isanAssistantProfessorofLinguisticsatUniversityofToronto.Heis interestedinsentence-processing,syntactictheory,andcross-linguisticvariation.
JeffreyLidz isDistinguishedScholar-TeacherandProfessorofLinguisticsattheUniversityofMaryland.Hisresearchexploreslanguageacquisitionfromtheperspective ofcomparativesyntaxandsemantics,focusingontherelativecontributionsofexperience,extralinguisticcognition,anddomain-specificknowledgeinlearners’discovery oflinguisticstructureandlinguisticmeaning.
AndreaE.Martin isaLiseMeitnerResearchGroupLeaderattheMaxPlanckInstitute forPsycholinguistics,andaPrincipalInvestigatorattheDondersCentreforCognitiveNeuroimagingatRadboudUniversityinNijmegen,theNetherlands.Herwork hasspannedstructuralandsemanticaspectsofsentenceprocessing.Shehasusedthe speed–accuracytrade-offprocedureandcognitiveneuroimagingtostudytheroleof memoryinsentenceprocessingviaellipsis,alineofresearchbegunwithBrianMcElree, whopioneeredapplicationofSATtopsycholinguisticissues.Thecurrentfocusofher lab,LanguageandComputationinNeuralSystems,isondevelopingtheoriesandmodelsoflanguagerepresentationandprocessingwhichharnessthecomputationalpower ofneuraloscillations,suchthatformalproperties(viz.,constituency,compositionality) canberealizedinbiologicalandartificialneuralnetworks.
WilliamMatchin isanAssistantProfessorofCommunicationSciencesandDisorders intheArnoldSchoolofPublicHealthattheUniversityofSouthCarolina.Aspartof theCenterfortheStudyofAphasiaRecovery,hedirectstheNeuroSyntaxlab,using functionalneuroimagingandlesion–symptommappingandincorporatinginsightsof linguistictheorytounderstandthearchitectureoflanguageinthebrain.Heiscurrently investigatingthenatureofgrammaticaldeficitsinaphasia,includingparagrammatism andagrammatism.
LisaS.Pearl isaProfessorintheDepartmentofLanguageScienceattheUniversity ofCalifornia,Irvine.Herresearchliesattheinterfaceoflanguagedevelopment,computation,andinformationextraction,includingbothcognitivelyorientedresearchand appliedlinguisticresearchthatcombinestheoreticalandcomputationalmethods.Her cognitivelyorientedresearchfocusesonchildlanguageacquisition,withaparticular focusontheoryevaluationviaacquisition-modeling,andhowchildren’sinputaffects theirlinguisticdevelopment.
LaurelPerkins isanAssistantProfessorintheDepartmentofLinguisticsattheUniversityofCalifornia,LosAngeles.SheearnedherPhDinlinguisticsfromtheUniversity
xxthecontributors
ofMarylandandheldapostdoctoralfellowshipintheLaboratoiredeSciencesCognitivesetPsycholinguistiqueattheÉcoleNormaleSupérieure.Herresearchstudiesthe earlieststagesofsyntaxacquisitionininfancy,drawingfromformallinguistics,developmentalpsychology,andcomputationalcognitivemodelling.Sheisarecipientofa Post-DoctoralStudyGrantfromtheFyssenFoundation,aDoctoralDissertationImprovementGrantfromtheNationalScienceFoundation,andaGlushkoDissertation PrizefromtheCognitiveScienceSociety.
MariaPolinsky isProfessorofLinguistics,AssociateDirectoroftheLanguageScience Center,andDirectorofResearchFieldStationsattheUniversityofMaryland.Shehas conductedextensiveprimaryworkonseverallanguagesoftheCaucasus,Austronesian languages,andChukchi.Sheisalsoengagedinacomprehensiveresearchprogramon heritagelanguages.Herworkemphasizestheimportanceoflesser-studiedlanguages fortheoreticallinguistics.RecentpublicationsincludeDeconstructingErgativity(2016), HeritageLanguagesandTheirSpeakers (2018),and TheOxfordHandbookofLanguages oftheCaucasus (2021).
CorianneRogalsky isanAssociateProfessorofSpeechandHearingScienceinthe CollegeofHealthSolutionsatArizonaStateUniversity(ASU).AsDirectorofASU’s CommunicationNeuroscienceandNeuroimagingLab,Rogalskyusesbehavioraland neuroimagingtechniquestobetterunderstandtheneuralandcognitiveresourcesthat supporteffectivecommunicationineverydaylifeforindividualswhohaveexperiencedabraininjurysuchasastroke.Rogalsky’scurrentfocusisinvestigatinghow executivefunctionssuchasselectiveattentionandworkingmemorysupportspeech comprehensioninneurotypicaladults,andhowthatsupportmaychangeafterastroke.
JeffreyRunner isaProfessorofLinguisticsandBrain&CognitiveSciences,Deanof theCollege,andViceProvostandUniversityDeanforUndergraduateEducationat theUniversityofRochester.HeearnedaBAinlinguisticsattheUniversityofCalifornia,SantaCruz,in1989andaPhDinlinguisticsattheUniversityofMassachusettsat Amherstin1995.HejoinedthedepartmentofLinguisticsattheUniversityofRochester in1994.Hisresearchusesexperimentalmethodologiestoinvestigatenaturallanguage syntax.In2017,hebecamedeanoftheCollegeinArts,SciencesandEngineering, andisresponsibleforthecurricular,co-curricular,andextra-curricularundergraduate experience.
JonSprouse isaProfessorofPsychologyatNewYorkUniversityAbuDhabi.HereceivedanABinlinguisticsfromPrincetonUniversity(2003)andaPhDinlinguistics fromtheUniversityofMaryland(2007).Hisresearchfocusesontheuseofexperimentalsyntaxtechniques,includingacceptabilityjudgments,EEG,andcomputational modeling,toexplorefundamentalquestionsinsyntax.Hehasauthoredoverfortyjournalarticlesandbookchaptersonexperimentalsyntax.Hisworkhasbeenrecognized bytheBestPaperin Language award,theEarlyCareeraward,andtheC.L.Baker mid-careerawardfromtheLinguisticSocietyofAmerica.
thecontributorsxxi
KristenSyrett isanAssociateProfessorintheDepartmentofLinguisticsatRutgers,the StateUniversityofNewJersey–NewBrunswick,withaco-appointmentattheCenter forCognitiveScience(RuCCS).SheistheDirectoroftheLaboratoryforDevelopmental LanguageStudies.Herresearchfocusesonsemanticsanditsinterfacewithpragmatics andsyntaxinlanguageacquisitionanddevelopment,andonexperimentalsemantics andpragmaticsinadultpsycholinguistics.
KrisztaEszterSzendrői isaProfessorofTheoreticalandExperimentalLinguisticsat theUniversityofVienna.Sheworksoninformationstructure,includingitssyntaxand prosody,usingboththeoreticalandexperimentalmeans,workingwithbothadults andchildren.Shehasalsoworkedonthesyntaxofscope,especiallyontheacquisitionofscope.Sheisalsointerestedintheinteractionsbetweeninformationstructure andscope.Onadifferentnote,forthepastfewyearsshehasbeenleadingaresearch projectstudyingthegrammarofContemporaryHasidicYiddish.
MatthewWagers isProfessorofLinguisticsattheUniversityofCalifornia,Santa Cruz,wherehehastaughtsince2009.Thefocusofhisresearchishowsyntacticinformationisrepresentedinmemoryandhowmorphologicalcuesguideincremental interpretation.Thesearecross-cutbyaninterestinbroadeningthecontributionofpsycholinguisticallyunder-investigatedlanguagestotheorydevelopment.HeholdsaPhD inlinguisticsfromtheUniversityofMaryland(2008)andanABinMolecularBiology fromPrincetonUniversity(2003).
MasayaYoshida isanAssociateProfessorintheDepartmentofLinguisticsatNorthwesternUniversity.Researchinterestsincludeonlinesentenceprocessingandsyntax. Hehasworkedonthesyntaxandprocessingofellipsisconstructions,long-distance dependencies,andislands.Someofhisrecentstudieshaveexploredthestructureassociatedwiththeellipsissiteinclausalellipsisconstructions,andhowstructureinthe ellipsissiteisbuiltduringonlinesentence-processing.
acceptability judgments jonsprouse
1.1Introduction
Thegoalofexperimentalsyntax,atleasttomymind,isstraightforward:touseexperimentalmethodstocollectdatathatisrelevantfortheconstructionandevaluationof syntactictheories.Fordatatypesthatcanonlybecollectedusingaformalexperiment, suchasreactiontimesorEEG,theworkofexperimentalsyntaxissimplytheworkof leveragingthesemethodsforquestionsintheoreticalsyntax.However,thingsappearto beabitmorecomplicatedwhenthedatatypeinquestionisacceptabilityjudgments,as acceptabilityjudgmentscanbecollectedbothrelativelyinformally,asistypicalinmuch ofthesyntaxliterature,orrelativelyformally,asistypicalintheexperimentalsyntax literature.Itakethecoexistenceofthesetwomethodsofjudgmentcollectiontoimply thatthegoalofexperimentalsyntaxwithrespecttoacceptabilityjudgmentsisnotsimplytocollectacceptabilityjudgments,becausethatiswhatisdoneinallsyntacticwork, butrathertoexplorewaysinwhichtheformalcollectionofjudgmentscanaddnewinsightsoverandabovethosethatderivefrominformalmethods.Therefore,mygoalin thischapteristoidentifyfourareasinwhichformaljudgmentexperimentshavemade substantialcontributions—twothatleantowardmethodologicalissues,andtwothat leantowardtheoreticalissues—andtoreviewthecurrentstateoftheevidencethatwe haveforeachofthoseareas.Tobeclear,thischapterisnotintendedasanexhaustivereviewofallpossibleareasinwhichformaljudgmentexperimentscouldpotentiallymake acontribution;rather,itisintendedasastartingpointforthinkingaboutthekindsof questionsintheoreticalsyntaxthatmightbenefitfromformalacceptabilityjudgment experiments.Myhopeisthatthesequestionswillhelptoinspirenewquestions,and newwork,inthegrowingfieldofexperimentalsyntax.
4jonsprouse
Beforedelvingintotheprimarycontentofthischapter,Iwouldliketobrieflymentionafewassumptions(and/ordecisions)thatIammaking.ThefirstisthatIassume, followingmanyworkingsyntacticians,thatacceptabilityjudgmentsareinprinciple valuablefortheconstructionandevaluationofsyntactictheories.Iwill,therefore,not attempttomotivatetheuseofacceptabilityjudgmentsingeneral(seeSchütze1996 foracomprehensivediscussionofthis).ThesecondisthatIwillassumearelatively minimallinkinghypothesisbetweenacceptabilityjudgmentsandthecognitivepropertiesofsentenceprocessing.Underthislinkinghypothesis,anacceptabilityjudgment isarelativelyautomaticbehavioralresponsethatariseswhenaspeakercomprehends asentence,andthatthisbehavioralresponseisimpactedbyalargenumberofcognitivefactors,suchasthegrammaticalityofthesentence,theprocessingdynamicsof thesentence,thesentence-processingresourcesrequiredbythesentence,themeaning ofthesentence,theplausibilityofthesentencerelativetotherealworld,andeventhe propertiesofthespecifictaskthatisgiventothespeaker.Ibelievewholeheartedlythat amorepreciselinkinghypothesiswouldbehelpfulforusingjudgmentsasevidencein syntax;however,Ialsobelievethattheminimallinkinghypothesisaboveismorethan sufficienttobegintoexplorethevalueofformalacceptabilityjudgmentexperiments insyntax.Mythirdassumptionisthatthereisnosubstantivequalitativedifference between“informal”and“formal”judgmentexperiments.Bothareexperimentsinthe sensethattheyinvolvethemanipulationofonevariable(syntacticstructure)torevealacausalrelationshipwithanothervariable(acceptability).Thereforebothinvolve allofthecomponentsthattypifypsychologyexperiments:asetofconditions,asetof itemsineachcondition,asetofparticipants,ataskfortheparticipantstocomplete usingtheitems,andaprocessforanalyzingtheresultsofthetask.Thedifferenceappearstometobeprimarilyquantitative,inthat“formal”experimentstendtoinvolve moreconditions,moreitemspercondition,moreparticipants,andmorecomplexanalysisprocesses.Tomymind,thelabels“informal”and“formal”simplypointtoward differentendsofthisquantitativespectrum.Inpractice,whenIsaythatformalexperimentsarevaluableinsomeway,whatImeanisthatincreasingthenumberof conditions,items,orparticipants,and/orincreasingthecomplexityoftheanalysis, canyieldinsightsthatfewerconditions,items,participants,and/orlesscomplexanalysescannot.Thelabels“informal”and“formal”areamoreconcisewaytoexpress thisidea.MyfourthassumptionisthatSchütze1996alreadyprovidesacomprehensivereviewofexperimentalsyntaxworkthatwaspublishedbefore1996.Therefore, inordertoprovidesomethingnewforthefield,Iwillfocushereonworkpublished after1996.
Finally,thischapterisnotahow-toforconstructingformaljudgmentexperiments. Thegoalisforthistobethechapteronereads,eitherbeforeorafterreadingahowto,forinspirationaboutthetypesofquestionsonecanaskwiththemethod.Iwill providesomereferencesforlearningacceptabilityjudgmentmethodsintheannotated bibliographyforPartIofthisHandbook.
1.2Thevalidityandreliabilityof acceptabilityjudgments
Perhapsthemostfrequentlyaskedquestionintheexperimentalsyntaxliteratureisto whatextenttheinformallycollectedjudgmentsthathavebeenpublishedintheliteraturecanbetrustedtoformtheempiricalbasisofsyntactictheory.Thisquestionhas arisensincetheearliestdaysofgenerativegrammar(Hill1961;Spencer1973);itplayed acentralroleinthetwobooksthatusheredinthemostrecentwaveofinterestinexperimentalsyntax(Schütze1996;Cowart1997);andithasgivenrisetoanumberof high-leveldebatesintheexperimentalsyntaxliteratureoverthepast15yearsorso (seeEdelmanandChristiansen2003;Ferreira2005;WasowandArnold2005;Featherston2007;GibsonandFedorenko2013forsomeconcernsaboutinformalmethods;see Marantz2005andPhillips2009forsomerebuttals,andMyers2009foraproposalthat attemptstosplitthedifferencebetweeninformallycollectedjudgmentsandfull-scale formalexperiments).Theexistenceofthisquestionisunderstandable.First,informally collectedjudgmentsformthevastmajorityofthedatapointspublishedinthe(generative)syntaxliterature.Second,thepropertiesofinformalcollectionmethodsarenot identicaltothepropertiesoftheformalexperimentalmethodsthatareoftenusedin otherdomainsofcognitivescience:Informalmethodsofteninvolveasmallernumberofparticipants,thoseparticipantsareoftenprofessionallinguistsinsteadofnaïve participants,theparticipantsareoftenpresentedasmallernumberofitems,andtheresultsareoftenonlyanalyzeddescriptively(withoutinferentialstatistics).Ifonebelieves thatthepropertiesofformalexperimentsarewhattheyaretoensurethequalityofthe data,thenitislogicallypossiblethatthedifferencesbetweeninformalmethodsandformalexperimentscouldleadtolower-qualitydata.Theconsequencesofthiscannotbe understated.Iftherearesystemicproblemswithinformallycollectedjudgments,then therearelikelytobesystemicproblemswith(generative)syntactictheories.
Thisquestiontouchesuponanumberofissuesinpsychometricsandthebroader philosophyofmeasurement.Thefirstquestionis:Whatdowemeanwhenwesaythat datacanbe“trusted”toformthebasisforatheory?Psychometrictheorieshaveidentifiedanumberofpropertiesthatgoodmeasurementmethodsshouldhave.HereIwill mentiontwo(andonlyinacoarse-grainedway,settingasidesubtypesoftheseproperties):validityandreliability.Ameasurementmethodis valid ifitmeasurestheproperty itispurportedtomeasure.Ameasurementmethodis reliable ifityieldsconsistentresultsunderrepeatedmeasurements(withunchangedconditions).Theconcernsabout informalmethodsthathavefiguredmostprominentlyintheliteratureappeartobea combinationofconcernsaboutvalidityandreliability,suchastheconcernthatsmall samplesizeswillleadtoanundueinfluenceofrandomvariation,theconcernthata smallnumberofexperimentalitemswillleadtoanundueinfluenceoflexicalproperties,andtheconcernthattheparticipationofprofessionallinguistswillleadtotheoreticalbias.Ineachcase,theconcernseemstobethatinformallycollectedjudgments
6jonsprouse
willnotreflectthetrueacceptabilityofthesentence(validity),andfurthermorethatthe judgmentsthemselveswillbeinconsistentoverrepeatedmeasurements(reliability).
Thisleadstoasecondquestion:Howdoesoneestablishvalidityforthemeasurement ofacognitivepropertylikeacceptability?Thedirectmethodforestablishingvalidityis tocomparetheresultsofthemeasurementmethodwithasecond,previouslyvalidated, measurementmethod.Thisisobviouslyunavailableformostcognitiveproperties—if cognitivescientistshadamethodtodirectlymeasurethecognitivepropertyofinterest, wewouldnotbotherwiththeunvalidatedmeasurementmethod.Thatleavesonlyindirectmethodsofvalidation.Oneindirectmethodistoaskwhetherthetheorythatresults fromthedatahasthepropertiesofagoodscientifictheory.This,ofcourse,interacts withbroaderissuesinthephilosophyofscienceaboutwhatpropertiesagoodtheory wouldhave,soIwillnotattempttoprovideanexhaustivelist.Buttwopossiblecriteria are:(i)makingpotentiallyfalsifiablepredictions,and(ii)explainingmultiplephenomenawitharelativelysmallnumberoftheoreticalconstructs.Inthecaseofacceptability judgments,Iwouldarguethattheresultingtheoryofsyntaxdoes,indeed,havethese properties.Anotherindirectmethodistoaskwhetherotherdatatypesprovidecorroboratingevidence,modulothelinkingtheoriesbetweenthedatatypesandtheunderlying theory.Inthecaseofacceptabilityjudgments,wecanaskwhethertheresultingsyntactictheorycanbelinkedtoasentence-processingtheoryinawaythatmakespotentially falsifiablepredictionsaboutotherpsycholinguisticmeasures,suchasreadingtimes,eye movements,orEEG,andultimatelywhetherthesemeasurescorroboratethesyntactic theory.Iwouldarguethatthecurrentresultsintheliteratureconnectingsyntactictheoriesandsentence-processingtheoriesarepromising.Thatsaid,indirectmethodscannot guaranteevalidity.Itislogicallypossiblethatacceptabilityjudgmentscouldgiveriseto atheorythathasallofthehallmarksofagoodtheory,butthatdoesnotultimately explainhumansyntax(perhapstheresultingtheoryisactuallyaboutprobability,or plausibility,orevenprescriptivegrammaticalrules).
Thisleadstothefinalquestion:Howdoesoneestablishreliability?Inprinciple,establishingreliabilityisrelativelystraightforward,asitsimplyentailsreplicatingthemeasurement.Theexactreplicationcanvarybasedonthetypeofreliabilityoneisinterested in:Between-participant(orinter-rater)reliabilityaskswhetherthesamejudgmentsare obtainedwithdifferentsetsofparticipants;within-participant(ortest–retest)reliability askswhetheronesetofparticipantswillgivethesamejudgmentsattwodifferenttimes; between-taskreliabilityaskswhetherdifferentjudgmenttaskswillyieldthesamejudgments(eitherbetween-participantorwithin-participant).Inpractice,establishingthe reliabilityofinformalmethodsiscomplicatedbytheirinformality.Bydefinition,informalmethodscontrolthevariouspropertiesofthejudgmentcollectionprocessless strictlythanformalmethods,makingastrictreplicationdifficultifnotimpossible.One waytocircumventthisproblemistocomparetheresultsofinformalmethods,perhaps asreportedinthesyntacticliterature,withtheresultsofformalexperiments.Thiswould beatypeofbetween-taskreliabilityforinformalandformalmethods,andtotheextent thatthetwosetsofresultsconverge,itwouldestablishakindofreliabilityforinformal methods.Manyoftheresultsreportedbelowtestpreciselythiskindofreliability.But
acceptabilityjudgments7
itisimportanttonotethatwhileconvergencebetweenthetwomethodscanbeinterpretedasestablishingatypeofreliabilityforboth,divergencebetweenthetwomethods canbeinterpretedinthreeways:Itcouldbethecasethatinformalmethodsareunreliable,oritcouldbethecasethatformalmethodsareunreliable,orboth.Itistempting toassumethatformalmethodsenjoysomesortofepistemologicalpriorityinadivergence(i.e.thatformalmethodsrevealthegroundtruth),butasmanylinguistshave pointedout,itiseasytoimagineexperimentalmaterialsthatleadtounreliablejudgmentsfromnon-linguistparticipants,butnotfromlinguistparticipants(suchasgarden pathsentenceslike Thehorseracedpastthebarnfell).Resolvingthesourceofthedivergencebetweentwomethodsrequiresfollow-upexperimentsthatmanipulatespecific hypothesesforthedivergence.Tomyknowledge,thoughtherehavebeenmanystudies oftheconvergence/divergencebetweeninformalandformalmethods,therehavebeen nosystematicstudiesofthesourceofthedivergencesthatdoarise(presumablybecause, aswewillseepresently,therearerelativelyfewdivergencesbetweenthemethods).
Inreviewingtheevidencecollectedsofarontheconvergencebetweeninformaland formalmethodsforjudgmentcollection,itisusefultomakeadistinctionbetweenstudiesthatsampledthedatapointstoretestwithbias,andstudiesthatsampledthedata pointstoretestrandomly.Biasedsamplingmeansthatthedatapointswerechosenbecauseofsomepropertythattheyhave;inthesestudies,thisistypicallythebeliefthatthe specificdatapointsareinvalidorunreliable.Typicallythisbeliefcomesfromdebates intheliteratureaboutthestatusofthedatapoint,ortheresearchers’own(informally collected)judgments.Biasedsamplingstudiescanbeusedtoestablishthattheconvergencebetweeninformalmethodsandformalmethodsisnotperfectbyshowingthat thedatapointsinquestiondonotreplicateusingformalmethods.Butbiasedsampling cannotbeusedtoestimateaspecificconvergencerate.Abiasedsamplecouldeither overestimateorunderestimatetheactualconvergenceratebyvirtueofthebiasedselectionprocedure:Aprocedurethatfocusesonselectingknowninvalidorunreliabledata pointswillalmostcertainlyunderestimatethetrueconvergencerate;similarly,aprocedurethatfocusesonselectinglikelyuncontroversialdatapoints(e.g.ajudgmentfor Thisisapen)islikelytooverestimatetheconvergencerate.Thereareonlytwooptions fordeterminingthetrueconvergencerate:Anexhaustivecomparisonofalldatapoints, whichwouldestablishtheconvergenceratewithcertainty,orarandomsamplingprocedure,whichwouldestimatetheconvergenceratewithinamarginoferrordetermined bythesizeoftherandomsamplerelativetothesizeofthepopulationinquestion.
Biasedsamplingstudiesdominatedmuchofthedebateaboutthevalidityandreliabilityofinformallycollectedjudgmentsuntilrelativelyrecently,presumablybecause ofthetimeandfinancialcostassociatedwithtestinglargenumbersofdatapointsprior tothecreationofcrowdsourcingplatformslikeAmazonMechanicalTurk.HereIwill brieflyreviewsomeofthemoreprominentbiasedsamplingstudies.WasowandArnold (2005)testedaclaimfromChomsky(1957)thattheorderingpreferencebetweenNPs andparticlesinverb–particleconstructionsisbasedonthecomplexityoftheNP,not thelength.TheyfoundthatthejudgmentsfollowChomsky’sreportedjudgmentswhen averagedovertheentiresampleofparticipants,butthatsomeindividualparticipants
