Practical Assessment, Research, and Evaluation Practical Assessment, and Evaluation
Volume 9 Volume9,2004
2004
Article 11
Sample size and subject to item ratio in principal components
Sample size and to item ratio principal components analysis analysis
Jason W. Osborne
Anna B. Costello
Follow this and additional works at: https://scholarworks.umass.edu/pare
Recommended Citation Recommended Citation
Osborne, Jason W. and Costello, Anna B. (2004) "Sample size and subject to item ratio in principal components analysis," PracticalAssessment,Research,andEvaluation: Vol. 9 , Article 11.
DOI: https://doi.org/10.7275/ktzq-jq66
Available at: https://scholarworks.umass.edu/pare/vol9/iss1/11
This Article is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Practical Assessment, Research, and Evaluation by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu
Apeer-reviewedelectronicjournal.
Copyrightisretainedbythefirstorsoleauthor,whograntsrightoffirstpublicationtoPracticalAssessment, Research&Evaluation.Permissionisgrantedtodistributethisarticlefor nonprofit,educationalpurposesifitiscopiedinitsentiretyandthejournaliscredited.PAREhastherighttoauthorizethirdpartyreproductionofthisarticleinprint,electronicand databaseforms.
Volume9,Number11,June,2004
Samplesizeandsubjecttoitemratioinprincipalcomponentsanalysis.
JasonW.OsborneandAnnaB.Costello NorthCarolinaStateUniversityISSN=1531-7714
Statisticianshavewrestledwiththequestionofsamplesizeinexploratoryfactoranalysisand principalcomponentanalysisfordecades,somelookingattotalN,someattheratioofsubjectsto items. Althoughmanyarticlesattempttoexaminethisissue,fewexaminebothpossibilities comprehensivelyenoughtobedefinitive. Thisstudyexaminesapreviouslypublisheddatasetto examinewhetherNorsubjecttoitemratioismoreimportantinpredictingimportantoutcomesin PCA. Theresultsindicateaninteractionbetweenthetwo,wherethebestoutcomesoccurinanalyses wherelargeNsandhighratiosarepresent.
Exploratoryfactoranalysis(EFA)andPrincipalComponentsanalysis(PCA)havebothbeenimportanttoolsfor researchersforthebetterpartofacenturynow,andhavebecomeincreasinglycommonwithubiquitousaccessto computing. YetEFAandPCAremainodditiesinquantitativeanalysis,astherearenoinferentialstatisticaltests,and nowaytocalculateorcontroltheprobabilityofmakinganerrorofinference. Whilethefieldasawholeaspiresto maintainerrorratesof5%orless,wehavenowayofknowingwhatproportionofEFAsorPCAsresultinerrorsof inference.
ItiscrucialthatstatisticiansusesoundmethodologywhenconductingstudiesinvolvingEFAorPCAtominimizeerror ratesandmaximizethegeneralizabilitytothepopulationofinterest. Thegoalofthispaperistoexaminehowand whethersamplesizeaffectsthegoodnessofseveralimportantoutcomesrelatingtoprincipalcomponentsanalysis. PCA isoneofthemostcommonlyusedexploratorydatareductionproceduresusedinthesocialsciences,andisconceptually andmathematicallydistinctfromexploratoryfactoranalysis,althoughtheconclusionsreachedinthispapershould generalizetoEFA.
Whysizematters
Largersamplesarebetterthansmallersamples(allotherthingsbeingequal)becauselargersamplestendtominimize theprobabilityoferrors,maximizetheaccuracyofpopulationestimates,andincreasethegeneralizabilityofthe results. Unfortunately,therearefewsamplesizeguidelinesforresearchersusingEFAorPCA,andmanyofthesehave minimalempiricalevidence(e.g.,Guadagnoli&Velicer,1988).
Thisisproblematicbecausestatisticalproceduresthatcreateoptimizedlinearcombinationsofvariables(suchas multipleregression,canonicalcorrelation,andEFA\PCA)tendto"overfit"thedata. Thismeansthattheseprocedures optimizethefitofthemodelthegivendata;yetnosampleisperfectlyreflectiveofthepopulation. Thus,thisoverfitting canresultinerroneousconclusionsifmodelsfittoonedatasetareappliedtoothers. Inmultipleregressionthis manifestsitselfasinflatedR2(shrinkage)andmis-estimatedvariableregressioncoefficients(Cohen&Cohen,1983,p. 106). InEFAorPCAthis“overfitting”canresultinerroneousconclusionsinseveralways,includingtheextractionof erroneousfactorsormis-assignmentofitemstofactors(e.g.,Tabachnick&Fidell,2001,p.588)
Theultimateconcerniserror. Attheendoftheanalysis,ifonehastoosmallasample,errorsofinferencecaneasily occur,particularlywithtechniquessuchasEFAorPCA.
Publishedsamplesizeguidelines
Inmultipleregressiontextssomeauthors(e.g.,Pedhazur,1997,p.207)suggestsubjecttovariableratiosof15:1or30:1 whengeneralizationiscritical. ButtherearefewexplicitguidelinessuchasthisforEFAorPCA(Baggaley,1983). Two differentapproacheshavebeentaken: suggestingaminimumtotalsamplesize,orexaminingtheratioofsubjectsto variables,asinmultipleregression.
ComfreyandLee(1992)suggestthat“theadequacyofsamplesizemightbeevaluatedveryroughlyonthefollowing scale:50–verypoor;100–poor;200–fair;300–good;500–verygood;1000ormore–excellent”(p.217). Guadagnoli andVelicer(1988)reviewseveralstudiesthatconcludethatabsoluteminimumsamplesizes,ratherthansubjecttoitem ratios,aremorerelevant. ThesestudiesrangeintheirrecommendationsfromanNof50(Barrett&Kline,1981)to400
(Aleamoni,1976).
Thecaseforratios. TherearefewinthemultipleregressioncampwhowouldarguethattotalNisasuperiorguideline thantheratioofsubjectstovariables,yetindividualsfocusingonthePCAand/orEFAmethodologiesoccasionally vehementlydefendthisposition. Itisinterestingpreciselybecausethegeneralgoalforbothanalysesarethesame: to takeindividualvariablesandcreateoptimallyweightedlinearcomposites. Whilethemathematicsandprocedures differinthedetails,theessenceandthepitfallsarethesame. BothEFA/PCAandmultipleregressionexperience shrinkage,theover-fittingoftheestimatestothedata(Bobko&Schemmer,1984),bothsufferfromlackof generalizabilityandinflatederrorrateswhensamplesizeistoosmall.
Wefindabsolutesamplesizessimplisticgiventhevarianceinthetypesofscalesresearchersexamine. Eachscale differsinthenumberoffactorsorcomponents,thenumberofitemsoneachfactor,themagnitudeoftheitem-factor correlations,andthecorrelationbetweenfactors,forexample. Thisdiscomforthasledsomeauthorstofocusontheratio ofsubjectstoitems,ormorerecently,theratioofsubjectstoparameters(aseachitemwillhavealoadingforeachfactor orcomponentextracted),asauthorsdowithregression,ratherthanabsolutesamplesizewhendiscussingguidelines concerningEFAandPCA.
Gorsuch(1983,p.332)andHatcher(1994,p.73)recommendaminimumsubjecttoitemratioofatleast5:1inEFA,but theyalsohavestringentguidelinesforwhenthisratioisacceptable,andtheybothnotethathigherratiosaregenerally better.Thereisawidely-citedruleofthumbfromNunnally(1978,p.421)thatthesubjecttoitemratioforexploratory factoranalysisshouldbeatleast10:1,butthatrecommendationwasnotsupportedbypublishedresearch. Thereisno oneratiothatwillworkinallcases;thenumberofitemsperfactorandcommunalitiesanditemloadingmagnitudescan makeanyparticularratiooverkillorhopelesslyinsufficient(MacCallum,Widaman,Preacher,&Hong,2001).
Previousresearchonratios. Unfortunately,muchoftheliteraturethathasattemptedtoaddressthisissue,particularly thestudiesattemptingtodismisssubject:parameterratios,useflaweddata. Wewillpurposelynotcitestudieshere,but consideritsufficienttosaythatmanyofthesestudieseithertendtousehighlyrestrictedrangesofsubject:itemor subject:parameterratiosorfailtoadequatelycontrolfororvaryotherconfoundingvariables(e.g.,factorloadings, numberofitemsperscaleorperfactor/component)orrestrictedrangeofN. Someofthesestudiespurportingtoaddress subjecttoitemratiofailtoactuallytestsubjecttoitemratiointheiranalyses.
ThusresearchersseekingguidanceconcerningsufficientsamplesizeinEFAorPCAareleftbetweentwoentrenched camps--thosearguingforlookingattotalsamplesizeandthoselookingatratios. Thisisunfortunate,becauseboth probablymatterinsomesense,andignoringeitheronecanhavethesameresult:errorsofinference. Failuretohavea representativesampleofsufficientsizeresultsinunstableloadings(Cliff,1970),random,non-replicablefactors (Aleamoni,1976;Humphreys,Ilgen,McGrath,&Montanelli,1969),andlackofgeneralizabilitytothepopulation (MacCallum,Widaman,Zhang,&Hong,1999).
EFAandPCAinpractice
Ifoneweretotakeeithersetofguidelines(e.g,10:1ratiooraminimumNof400-500)asreasonableacasualperusalof thepublishedliteratureshowsthatalargeportionofstudiescomeupshort. OnecaneasilyfindarticlesutilizingEFA or(morecommonly)PCAbasedonsampleswithfewersubjectsthanitemsorparametersestimatedthatnevertheless drawsubstantiveconclusionsbasedonthesequestionableanalyses.Manymorehavehopelesslyinsufficientsamplesby eitherguideline.
Forexample,Ford,MacCallum,andTait(1986)examinedcommonpracticeinfactoranalysisinindustrialand organizationalpsychologyduringthetenyearperiodof1974-1984.Theyfoundthatoutof152studiesutilizingEFAor PCA,27.3%hadasubjecttoitemratiooflessthan5:1;56%hadaratiooflessthan10:1. Asimilar,morerecentsurvey of1076journalarticlesutilizingPCAorEFAinpsychologyrevealedthat40.5%ofpeer-reviewed,publishedstudies utilizedlessthana5:1subjecttoitemratio,and63.2%utilized10:1orunder(Costello&Osborne,2003). Giventhe stakesandtheempiricalevidenceontheconsequencesofinsufficientsamplesize,thisisnotexactlyadesirablestateof affairs.
ThePresent Study
Thisstudyfocusesononeparticularlyinterestingandwell-executedstudyonthisissue—thatofGuadagnoliandVelicer (1988). Inthisstudy,theauthorsusedmontecarlomethodstoexaminetheeffectsofnumberofcomponents(3,6,9,18), thenumberofvariables(36,72,108,and144),averageitem-componentcorrelation(.40,.60,or.80),andnumberof subjects(Nsof50,100,150,200,300,500,and1000)onthestabilityofcomponentpatternsinprincipalcomponents analysis. Intheseanalyseseachitemloadedononlyonecomponent,allitemsloadedequallyoneverycomponent,and eachcomponentcontainedanequalnumberofvariables. Thisstudyrepresentsoneofthefewstudiestomanipulateall oftheseimportantaspectsacrosstherangeofvariationseenintheliterature(withthetwopossibleexceptions: first, peopleoftenhavelessthan36itemsinanEFAorPCAanalysis,andsecond,thefactorloadingpatternsarerarelyas clearandhomogenousasinthesedata).
GuadagnoliandVelicer’s(1988)studywasalsointerestinginthattheyusedseveraldifferenthigh-qualityfit/agreement
https://scholarworks.umass.edu/pare/vol9/iss1/11
DOI: https://doi.org/10.7275/ktzq-jq66
indices. Equallyinterestingistheauthors’strongassertionthattotalsamplesizeiscritical,althoughtheynever actuallyoperationalizesubjecttoitemratio,nortestwhethertotalNisabetterpredictorofimportantoutcomesthan subjecttoitemratio,althoughgiventheirdataitwaspossibletodoso. Finally,theauthors’completedatatableswere published,allowingforreanalysesofthedata.
ThegoalofthisstudyistodirectlyexaminecompetingclaimsregardingtheimportanceofsamplesizetoPCA--to determinewhethereitheroverallsamplesizeorsubjecttoitemratiouniquelycontributetothe“goodness”ofoutcomes inPCA,beyondthecontributionsofotherimportantvariables,suchasnumberofvariablesorcomponentsandaverage itemloadingthathavebeenidentifiedasimportantintheliterature.
METHODS
ThedataforthisstudyweretakendirectlyfrompublisheddatainGuadagnoliandVelicer(1988). Thesedatawere generatedviamontecarlomethodsoutlinedintheirarticleindetail. Ingeneral,theauthorsgeneratedmultipledata sets,eachsetrepresentingaspecificcombinationofconditions,discussedbelow. Thesedatasetswerethensubjectedto PCA,andtheoutcomeswererecordedforanalysis.
Variablesincludedbytheauthors
Numberoffactors(m). Thenumberofcomponentsexaminedincluded3,6,9,and18.
Loadings. Theauthorsusedloadingsof.40,.60,and.80. Itshouldbenotedthatinthesedatasets,itemsnot intendedtoloadonacomponentwereassignedaloadingof0.00,makingthesepatternmatricesartificially clear.
Numberofitems(p). Thenumberofitemsintheanalysesincluded36,72,108,and144.
Numberofsubjects(N). Thenumberofsubjectsintheanalysesincluded50,100,150,200,300,500,and 1000. Notehowever,thatcertaincaseswereomittedoralteredbytheauthors,suchaswhenNwaslessthan thenumberofitemsintheanalysis.
Patterncomparison(g2). Inordertocomparesamplecomponentpatternswithpopulationcomponent patterns,theaverageofthesquareddifferencesbetweenthetwomatriceswascomputed. Furthermore,the authorsidentifiedg2=.01asthemaximumvaluethatindicatesacceptablefit.
Patternagreement (kappa). Salientvariables(loadings>.40)andnon-salientvariables(loadings<.40)were identifiedandnotedindecisiontables. Thesedecisiontableswerethencomparedtothepopulationdecision tableviathekappastatistic. Askappaapproaches1.0thetwomatricesbecomemoreinagreementwitheach other.A0indicatesrandomchancelevelofagreement,andnegativekappasindicatepoorerthanchance agreement.
TypeI errors. Theauthorscalculatedthepercentofvariablesthatshouldnothavebeenconsideredsalient butwereinaparticulardataset,indicatingTypeIerrorclassifications.
TypeII errors. Theauthorsalsocalculatedthepercentofvariablesthatshouldhavebeenconsideredsalient butwerenotfoundtobeso,indicatingTypeIIerrorclassifications.
Variablescalculatedforthisanalysis
Forourpurposeswecalculatedthefollowingvariablesbasedontheinformationobtainedfromthedataset:
Subject-to-itemratio. Theratioofthenumberofsubjectsperiteminaparticularanalysiswascalculated fromtheinformationgiven.
Variable-to-component ratio. Assomeauthorshavearguedthatthenumberofvariablespercomponentor factorisimportant(seeGuadagnoli&Velicer,1988)weincludedthisvariableinanalyses.
Extramatrices. IndescribingtheirdatagenerationproceduresGuadagnoliandVelicer(1988)indicatedthat undercertainconditions“thecomponentpatternsdidnotpossessastructuredefinedwellenoughforaoneto-onecomponentmatchwiththepopulationcomponentstructuretobeattained.”(p.267). Inotherwords, certaindatasetsproducederrorsofinferenceregardingthenumberoffactorsextractedfromthedata. The authorsdiscardedthesedatamatricesandreplacedthemuntil5goodmatricesforaparticularsetofcriteria wereobtained. Theynotedcaseswhereupto10additionalmatriceswererequiredbefore5goodmatrices wereobtained,andcaseswhere10ormorematriceswererequired(aphenomenallyhigherrorrate). From anappliedresearchpointofview,thiscouldbeviewedasanimportantoutcome,wherearesearcherwould findresultsthatdifferradicallyfromthepopulation,andthusshouldbeexaminedasavariableofinterest. Thus,thisvariablewascodedintothedatasetforthecurrentanalysesas0(noextramatrices),1(upto10 extramatricesrequired),or2(morethan10extramatricesrequired)asthatishowthisinformationwas
reported.
Correct factorstructure. WhatmanypeoplearelookingforwhentheydoaEFAorPCAanalysisisthe pattern—whatvariables“load”onwhatcomponents. Researchersaregenerallylessinterestedinthe absolutemagnitudeoftheloading(aboveacertain“salient”level--thatisasourceofdebateinandofitself) thanwhichvariablegoeswithwhichfactor. Thus,weincludedinformationinourdatasetthatindicated whenthishadorhadnotoccurred,basedonthenumberofTypeIandTypeIIerrors. Matricesthathadno errorswereconsidered“correct,”whilematriceswitherrorswereconsiderednotcorrect. Somemightthink thisastrictcriterion,andtheyarecorrect.However,thepresenceoftheseerrorscansignificantlyalterthe interpretationofanexploratoryanalysis(eitherEFAorPCA). Inthisstudy,34.3%ofthecasesfailedto faithfullyreplicatethepatternfoundinthepopulation.
Data
Theauthorsgeneratedfivesamplesforeachofthe205validconditionsdescribed. Theaverageg2,kappa,typeIerror, typeIIerrorforthefivesamplesineachconditionwerereportedintables. Thus,theseresultsrepresentanalysesofthe dataaggregatedacrossfivesamplesineachcondition.
RESULTS
Maineffects
Withtheexceptionofthenewly-calculatedvariablesdescribedabove,weattemptedtofaithfullyreproducetheauthors’ analyses. Weperformedmultipleregressionanalysesonthedependentvariables(g2 ,kappa,TypeIerror,andTypeII error),abinomiallogisticmultipleregressionpredictingcorrectfactorstructure,andamultinomiallogisticregression predictingthepresenceofextramatrices. Asintheoriginalarticle,weexaminedallpossibletwo-wayinteractions. The differenceisthatwenowsimultaneouslycanexaminetotalNandsubjecttoitemratiofortheiruniqueandjoint contributionstothegoodnessofPCAoutcomes.
TheresultsoftheseanalysesarepresentedinTables1-3. AsTable1indicates,thenumberofcomponentswasnot significantpredictorofanydependentvariableonceothervariableswerecontrolledfor. Aspreviousresearchhas reported,itemloadingmagnitudeaccountedforsignificantuniquevarianceintheexpecteddirectioninallbutonecase, andinmostcaseswasthestrongestuniquepredictorofcongruencebetweensampleandpopulation. Specifically,as itemloadingsincreased,averagesquareddiscrepancybetweenpopulationandsampleresults(g2)decreased,agreement (kappa)increased,TypeIIerrorsdecreased,andtheoddsofgettingthecorrectcomponentpatternincreased dramatically.
Table1:Predictors ofcomponent pattern stability—main effects
Note: Statistics reportedrepresent betas (standardizedregression coefficients)when all predictors are in theequation. Betas in parentheses arefrom regression equations with tworatiovariables removed. * p< .05, ** p< .01, *** p< .001.
1. Odds ratioreported. For loadings, as therewas only .40, .60, and.80for values, this was considered acategorical variable. Thus, thefirst odds ratiorepresents therelativeodds ofgetting correct pattern structures with a.40vs. a.80averageloading, whilethesecondodds ratiorepresents therelativeodds ofgetting correct pattern structures with a.60vs. an .80averageloading.
Contrarytootherstudies,neitherthenumberofvariablesnorNhadasignificantuniqueeffectwhenallothervariables wereheldconstant(exceptfortherelationshipbetweenNandtheoddsofaTypeIIerror). Thelackoffindingsforthese
https://scholarworks.umass.edu/pare/vol9/iss1/11 DOI: https://doi.org/10.7275/ktzq-jq66
twovariablesmightbedirectlyattributabletothepresenceoftheratiosofsubjecttoitemandvariabletocomponent, whichwouldlikelysharevariance. Totestthishypothesis,ablockwisemultipleregressionwasperformedentering numberofcomponents,loadings,numberofvariables,andNinblock1,andsubjecttoitemratioandvariabletofactor ratioinblock2. Numberofvariableswasasignificantpredictorintwoofthefiveanalyseswherethetworatiovariables werenotintheequation. TotalNwassignificantinallfiveanalyseswherethetworatiovariableswerenotinthe equation.
Theratioofsubjectstoitemshadasignificantandsubstantialinfluenceonthreeoutcomesintheexpecteddirection. As subjecttoitemratioincreased,thesquareddiscrepancybetweenpopulationandsamplematricesdecreased,theoddsof aTypeIerrordecreased,andtheoddsofgettingacorrectcomponentpatternmatrixincreased. Finally,theratioof variablestocomponentshadnouniqueeffect.
Themultinomiallogisticregressionpredictingtheneedforextramatricesdidnotidentifyanysignificantpredictors,nor didabinomiallogisticregressionanalysispredictingtheneedforanyextramatricesornone. Whateverthereasonfor thislackofresults,itisclearthatthisoutcomeisrelatedtosubjecttoitemratio,numberofsubjects,andtheratioof variablestocomponents(allp<.001whenanalyzedindividually),asTable2shows. Notethatthiseventonlyoccurred whenloadingswererelativelyweak(.40),andthusloadingswasheldconstant.
Interactions
Totestforinteractioneffects,ablockwisemultipleregressionanalysiswasperformedenteringallmaineffectsinblock1 andallinteractionsinblock2. Inallcases,whenblock2wasenteredtherewasasignificantchangeinRandR2(all p< .0001).
AstheresultsinTable3show,therewereseveralinterestinginteractionspresentinthesedata. Therewereno significantinteractionsincludingtheratioofvariablestocomponents.
Table3:Predictors ofcomponent pattern stability—interactions
Numberofcomponentsandcomponent loadings. Therewasasignificantinteractionbetweenthenumberofcomponents extractedandthemagnitudeofcomponentloadings. Thenatureoftheinteractionindicatedthatmorecomponents tendedtoinflateg2whenloadingswererelativelyweak,buthadlessofaneffectwhentheloadingswereverystrong.
Component loadingsandthenumberofvariables. Thisinteractionindicatedthat,whilestrongercomponentloadings arerelatedtolowerg2andTypeIerrorrates,loadingshadlessofaneffectasthenumberofvariablesincreased.
Component loadingsandthenumberofsubjects. Thisinteractionindicatedthat,whilestrongercomponentloadingsare relatedtohigherkappasandlowerTypeIIerrorrates,loadingshadlessofaneffectasthenumberofsubjectsincreased.
Component loadingsandtheratioofsubjectstovariables. Thisinteractionindicatedthat,whilestrongercomponent loadingsweregenerallyrelatedtolowerg2andTypeIerrorrates,loadingshadlessofaneffectastheratioofsubjectsto variablesincreased.
https://scholarworks.umass.edu/pare/vol9/iss1/11
Numberofvariablesandthenumberofsubjects. Whilethenumberofvariableswasgenerallyrelatedtomorefavorable outcomes(lowerg2 ,higherkappa,andlowerTypeIerrorrates),asthenumberofsubjectsincreasedtheeffectofthe numberofvariablesdecreased.
Numberofsubjectsandtheratioofsubjectstovariables. Whileincreasingratiosofsubjectstovariableswasgenerally relatedtomorefavorableoutcomes(lowerg2 ,higherkappa,andlowerTypeIandTypeIIerrorrates),asNincreased, thiseffectbecamelessimportant.
DISCUSSION
Whiletheoriginalauthorsofthisstudyconcludedthattheonlytwoimportantfactorsindeterminingthe correspondencebetweenaPCAandthepopulationweretherawnumberofsubjectsandthemagnitudeofthe componentloadings,theexaminationoftheratioofsubjectstovariablesandvariablestocomponentsandtheirvarious interactionstellaslightlymoresubtlestory.
First,whilethemagnitudeofcomponentloadingshasalargeinfluenceongoodnessoftheanalyses,therawnumberof subjectshadasignificantinfluenceontheaveragepercentofTypeIIerrors. Theratioofsubjectstovariableshada significantuniqueeffectong2,TypeIerrorrates,andobtainingthecorrectloadingpattern. InlookingatTable1itis difficulttodismisscomponentloadingsandtheratioofsubjectstovariablesasthemostconsistentpredictorsofthese variables. EquallynotablewastherelativelackofuniqueimpactofNoncetheratioofsubjecttovariableswas accountedfor.
Thesemaineffectswereinsomewaysqualifiedbyseveralinteractions. Forexample,theratioofsubjectstovariables appearedtohavealargereffectwhentherawnumberofsubjectswaslower,thenumberofsubjectsappearedtohave lessofaneffectwhentherewerefewervariablesintheanalysis,andthenumberofvariables,thenumberofsubjects, andthesubjecttovariableratiohadalargereffectswhenthecomponentloadingsweresmaller.
TheinteractionofNandsubjecttovariableratiowasparticularlyinteresting. Althoughtheratioofsubjecttovariable isanimportantpredictorofthegoodnessofaPCAorEFA,itappearsthatastotalNincreases,thisratiobecomesless important(theconverseisalsotrue--asthesubjecttoitemratioincreases,totalNbecomeslessimportant). Insome sense,then,authorsfrombothsidesofthisdebatearecorrect--totalNmatters(butmoresowhensubject:itemratiois low),andtheratioofsubjectstoitemsmatters(butmoresowhenNisrelativelylow),andifyouhavealargeNorlarge ratio,yourresultswillbemorereliable. Itshouldbeclearfromboththemaineffectandinteractionanalysesthatitis difficulttodismissanyofthesefactorsindiscussingthereproducibilityofpopulationvaluesandpatternsinsample analyses.
Caveats
Itshouldalsobenotedthat,althoughthesedataweresuperiorthanthedatausedinmanyotherarticlesonthetopic, theyarenotnecessarilyideal. Forexample,theoriginalauthorschosetoanalyzelargenumberofitems(36minimum, 144maximum),whichmaynotberepresentativeofwhatresearchersgenerallyinvestigate. Thenumberoffactorswas generallylarge,rangingfrom3to18,ignoringsingle-ortwo-factoranalyses. Finally,themediansubjecttoitemratio was3.5(witharangeof1.04:1to27.78:1),whichisfarfromideal. Thisgeneralrestrictionofrangeandthehighly skewednatureoftheratiovariablemayleadtoanunderestimationoftheeffectofthisratio,asmostguidelinescallfor atleast10:1ormore.
Itisalsoimportanttonotethattheresultsreportedherereflectprincipalcomponentanalyses. Manystatisticiansand methodologistswillpointoutthatEFAandPCAaredistinctprocedures. However,fromapractitionerpointofview,the mathematicsandprocessesbehindeacharerelatedandsimilar,inpracticetheoutcomeofaPCAandEFAisoften identical,andtheseresultsrelatingtoPCAshouldgeneralizetoEFAhandily. However,thecaveatisthattheeffectof thesevariablesonEFAhasnotbeenrigorouslydemonstratedyet.
Finally,itshouldbenotedthateventheoriginalauthorsnotedtheartificially“clean”natureofthepatternsbeing replicated. Noresearcherworkingwithempiricaldatawillseepatternsof0sand.60s,forexample. Infact,many researcherswillnotseeaverageloadingsof.80atall;anythingover.50isgenerallyclassifiedasa“strong”itemloading. Moderateandweakloadings,whicharefrequentlytheruleinbehavioralresearch,rangefrom.32(whichequatesto approximately10%ofthevarianceaccountedfor)upto.50. GuadagnoliandVelicer(1988)alsocompletelyignore crossloaders,whichareitemsthatloadabovethe.32levelonmorethanonefactors.Theseareparticularlyproneto occurduringtheinitialresearchphaseofmeasureconstruction,whenitemsarebeingtestedforinclusion.Theincidence ofcrossloadingscanbedramaticallyeffectedbybothsamplesizeandsubject:itemratio,particularlywhenitemloadings areinthelowtomoderaterange(Costello&Osborne,2003).Frequentlycrossloadersdisappearwhenthesamplesizeis adequate,butwhenthesamplesizeorsubject:itemratioissmallthereisnowaytodeterminewhetheracrossloading itemistheresultofasamplingerrorortheindicationofapooritem.
Paststudieshavefoundthatvariablessuchasthenumberofitemspercomponent/factor,andthemagnitudeoftheitem loadingstendtoreducethesamplesizeneededforvalidinference. Wedonottakeissuewiththesepreviousfindings. Whileitispossibleforresearcherstocontrolthenumberofitemsiftheresearchersisalsoascaledesigner,itseemstobe
eveneasiertocontrolsamplesizeinmanycases.
CONCLUSIONS
Althoughsomeauthorseschewtheconceptofsubjecttovariableratioasanimportantinfluenceinthe“goodness”of exploratoryfactoranalysisorprincipalcomponentsanalysis,thatseemsshort-sightedandsimplistic. Researchersneed torememberthatEFAandPCA(andothertechniqueslikestructuralequationmodeling)arelarge-sampletechniques, notwell-suitedtothesmallsamplesizessomeresearchersusethemon. Theseanalysesdemonstratethat,atleastfor thisdataset(whichwebelieveisoneofthebetteronesoutthere),holdingallothervariablesconstant,subjectto variableratiomakesasignificantcontributionbeyondthatofmeresamplesize,particularlywhenoverallsamplesizeis notoverwhelminglylarge.
Buttheseeffectsdonotshowevidenceofa“criticalmass”or“criticalratio”--theydonotplateau,thelinesarenot asymptotic. Therearediminishingreturns,butevenatlargesubjecttoitemratiosandNs(suchas20:1ratioorN> 1000)andwithunrealisticallystrongfactorloadingsandclearfactorstructures,EFAandPCAcanproduceerrorrates upto30%(Costello&Osborne,2003),leavingroomforimprovementvialargersamples.
Thus,themostvalidconclusionregardingsamplesizeisthatmoreisalwaysbetter. Period. Ifsubjecttoitemratios appealintuitivelytosomeresearchers,andifitleadsresearcherstoutilizesamplesofamoreappropriatesize,itis useful. Whynotencouragethiswayofthinking?
References
Aleamoni,L.M.(1976).Therelationofsamplesizetothenumberofvariablesinusingfactoranalysistechniques. EducationalandPsychologicalMeasurement,36,879-883.
Baggaley,A.R.(1983).Decidingontheratioofnumberofsubjectstonumberofvariablesinfactoranalysis. MultivariateExperimentalClinicalResearch,6(2),81-85.
Barrett,P.T.,&Kline,P.(1981).Theobservationtovariableratioinfactoranalysis.Personalitystudyandgroup behavior,1,23-33.
Bobko,P.,&Schemmer,F.M.(1984).Eigenvalueshrinkageinprincipalcomponentbasedfactoranalysis.Applied PsychologicalMeasurement,8,439-451.
Cliff,N.(1970).Therelationbetweensampleandpopulationcharacteristicvectors.Psychometrika,35,163-178.
Cohen,J.,&Cohen,P.(1983).Appliedmultipleregression/correlationanalysisforthebehavioralsciences.Hillsdale,NJ: LawrenceErlbaumAssociates,Inc.
Comfrey,A.L.,&Lee,H.B.(1992).AFirst CourseinFactorAnalysis.Hillsdale,NJ:LawrenceErlbaumAssociates.
Costello,A.B.,&Osborne,J.W.(2003).Exploring best practicesinFactorAnalysis:Fourmistakesappliedresearchers make.PaperpresentedatthePaperpresentedattheannualmeetingoftheAmericanEducationalResearch Association,Chicago,Ill,April.
Ford,J.K.,MacCallum,R.C.,&Tait,M.(1986).Theapplicationofexploratoryfactoranalysisinappliedpsychology:A criticalreviewandanalysis.PersonnelPsychology,39,291-314.
Gorusch,R.L.(1983).FactorAnalysis(2nded.).Hillsdale,NJ:LawrenceErlbaumAssociates.
Guadagnoli,E.,&Velicer,W.F.(1988).relationofsamplesizetothestabilityofcomponentpatterns.Psychological Bulletin,103,265-275.
Hatcher,L.(1994).AStep-by-StepApproachtoUsing theSAS®SystemforFactorAnalysisandStructuralEquation Modeling.Cary,N.C.:SASInstitutte,Inc.
Humphreys,L.G.,Ilgen,D.,McGrath,D.,&Montanelli,R.(1969).Capitalizationonchanceinrotationoffactors. EducationalandPsychologicalMeasurement,29(2),259-271.
MacCallum,R.C.,Widaman,K.F.,Preacher,K.J.,&Hong,S.(2001).Samplesizeinfactoranalysis:Theroleofmodel error.MultivariateBehavioralResearch,36,611-637.
MacCallum,R.C.,Widaman,K.F.,Zhang,S.,&Hong,S.(1999).Samplesizeinfactoranalysis.PsychologicalMethods, 4,84-99.
Nunnally,J.C.(1978).PsychometricTheory(2nded.).NewYork:McGrawHill.
Pedhazur,E.J.(1997).MultipleRegressioninBehavioralResearch:ExplanationandPrediction.FortWorth,TX:
https://scholarworks.umass.edu/pare/vol9/iss1/11
DOI: https://doi.org/10.7275/ktzq-jq66
Osborne and Costello: Sample size and subject to item ratio in principal components ana
HarcourtBraceCollegePublishers.
Tabachnick,B.G.,&Fidell,L.S.(2001).Using MultivariateStatistics(4thed.).NewYork::HarperCollins.
AuthorContact information:
JasonW.Osborne, DeptofCurriculumandInstruction
NorthCarolinaStateUniversity
PoeHall602,CampusBox7801 RaleighNC27695-7801
919-515-1714
jason_osborne@ncsu.edu
Authornotes:
Asoftenhappensinscience,theimpetusforthispaperwasamethodologicaldebatearisingoutofthesecondauthor’s Master’sthesis. Wedecidedtotakeanempiricalandscholarlyapproachtoinformingthedebate. Communication regardingthispapershouldbedirectedviaemailtojason_osborne@ncsu.eduorblandy_costello@ncsu.edu
Descriptors:FactorAnalysis;PCA;PrincipalComponents;SampleSize
Citation:Osborne,JasonW.&AnnaB.Costello(2004).Samplesizeandsubjecttoitemratioinprincipalcomponentsanalysis.Practical Assessment,Research&Evaluation,9(11).Availableonline:http://PAREonline.net/getvn.asp?v=9&n=11