50 minute read
Chapter 11: Test Bias
from Psychometrics: An Introduction 4th Edition R. Michael Furr ISBN 9781071824085, 1071824082. All Chapt
by ACADEMIAMILL
Test Bank
Multiple Choice
1. A researcher wants to test whether there are sex differences in math ability. He gives his students a multiplication test and finds that females score lower, on average, than males. In terms of construct bias, why might he need to be very cautious before interpreting this finding as evidence that females truly have lower “multiplication ability” (on average) than males?
a. The males and females in his class might not accurately represent the “population” of males and females.
b. The test scores might not reflect multiplication ability equally well in the two groups (e.g., the test scores might not be a good measure of multiplication ability for most/many females) c. The males in his class might have had more lucky guesses than females, thus earning higher scores. d. Math ability is not an important construct, so bias is inherent in its examination.
Ans: B
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium a. There is no concern about construct bias, according to these results. b. Among Canadians, NE test scores generally reflect differences in Anger. c. Among Russians, NE test scores generally reflect differences in Sadness. d. If we compare the NE test scores across cultures, we’re mainly comparing Anger (for Canadians) to Sadness (for Russians)
2. Suppose we are cross-cultural psychologists, and we study cultural differences in negative emotionality (NE, the tendency to experience negative emotions) We administer a three-item test of NE to a group of Canadians and a group of Russians. Because we’re aware of the potential for construct bias, we conduct a factor analysis of the NE items separately for each group. Within each group, we seem to have a onefactor structure, with the factor loadings in the table below. Based on these results, which of the following can we not conclude about the NE test?
Ans: A
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Application
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Hard a. group differences in a test’s means b. group differences in a test’s factorial structure c. group differences in the correlation between test scores and a key criterion variable d. group differences in admission/hiring rates, where admission/hiring is at least partially based upon test scores
3. Which of the following would indicate the presence of construct bias?
Ans: B
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium a. item discrimination index b. rank order c. reliability d. factor analysis
4. Which approach to evaluating construct bias would suggest if a test is not quite "working" in the same way in two different groups?
Ans: C
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Reliability
Difficulty Level: Medium a. item discrimination index b. rank order c. reliability d. differential item functioning analysis
5. Which approach to evaluating construct bias might involve a 10-item math test that can be evaluated in order of difficulty?
Ans: B
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Rank Order
Difficulty Level: Medium a. The items are less biased among males than among females b. The test’s items do not differentiate well between males who have higher and lower aptitude, but they do differentiate well between females who have higher and lower aptitude. c. The test likely suffers from predictive bias d. The test’s items reflect the construct equally well for both genders.
6. A test developer creates a new test to assess academic aptitude. She examines item discrimination values and finds that the test items have low discrimination values among males and high discrimination values among females. What does this mean?
Ans: B
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Item Discrimination Index
Difficulty Level: Medium a. item discrimination index b. rank order c. reliability d. factor analysis
7. Which approach to evaluating construct bias reflects the degree to which an item is related to the total test score?
Ans: A
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Item Discrimination Index
Difficulty Level: Medium a. item discrimination index b. differential item functioning analysis c. reliability d. factor analysis
8. Which approach to evaluating construct bias was developed in association with classical test theory and is particularly useful for binary items?
Ans: A
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Item Discrimination Index
Difficulty Level: Medium a. item discrimination index b. rank order c. reliability d. factor analysis
9. Which approach to evaluating construct bias is an important tool for evaluating the internal structure or dimensionality of a test?
Ans: D
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Factor Analysis
Difficulty Level: Medium a. item discrimination index b. rank order c. differential item functioning analysis d. factor analysis
10. Which approach to evaluating construct bias is a statistical procedure for partitioning the variance or covariance among test items into clusters that in some sense "hang together"?
Ans: D
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Factor Analysis
Difficulty Level: Medium a. intercept b. mean c. factor loading d. fairness
11. If an item’s ____________________ is not invariant across group 1 and group 2, then the psychological difference between two people in group 1 who score 1 point different on the item is NOT the same as the psychological difference between two people in group 2 who score 1 point different on the item. For example, imagine that Allison and Barbara are members of group 1, with Allison scoring 1 point higher than Barbara on a particular test item. Now imagine that Carl and Dave are members of group 2, with Carl scoring 1 point higher than Dave on the same particular test item. If the item’s ____________________ is not invariant across the groups, then the psychological difference between Allison and Barbara is NOT the same as the psychological difference between Carl and Dave, even though the observed/test difference is identical. (Note: both blanks are the same answer.)
Ans: C
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Application
Answer Location: Factor Analysis
Difficulty Level: Hard
12. Say we are examining a new personality scale to assess “feelings of loneliness.” We conduct a factor analysis of the scale separately for males and females, and we find the same factor structure in the two groups. This would be evidence of: a. convergent validity b. discriminant validity c. a lack of construct bias d. the presence of construct bias
Ans: C
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Factor Analysis
Difficulty Level: Medium
13. A test developer develops a new ability test and estimates the reliability of the new test’s scores separately for males and females. She finds that the test is much more reliable among males than among females. This indicates that: a. females are more likely to get an artificially high score on the test than males b. the test is pretty good at telling us which males have a better or worse ability, but it is not very good at telling us which females have better or worse ability c. the test is pretty good at telling us which females have a better or worse ability, but it is not very good at telling us which males have a better or worse ability d. the test is not good at telling us if either females or males have a better or worse ability
Ans: B
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Factor Analysis
Difficulty Level: Medium a. item discrimination index b. rank order c. differential item functioning analysis d. factor analysis
14. Which approach to evaluating construct bias is a feature of a psychometric approach called item response theory?
Ans: C
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Knowledge
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Easy
15. One way that test developers try to detect and understand construct bias is through a procedure called Item Response Theory (IRT) In Item Response Theory, an “Item Characteristic Curve” reflects: a. The connection between a) trait levels, and b) the likelihood of making a correct response to a test item b. The connection between a) trait levels, and b) the likelihood of passing a test c. The connection between a) the likelihood of making a correct response to a test item, and b) the likelihood of passing the whole test d The connection between a) trait levels, b) the likelihood of making a correct response to a test item, and c) the likelihood of passing the whole test
Ans: A
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Medium a. This suggests the presence of bias in the item even if a female and male have *the same exact level* of mechanical aptitude (say 1 SD below the mean), the female has a lower chance of answering the item correctly. b. This suggests the presence of bias in the item males on average have a lower level of mechanical aptitude. c. This suggests the absence of bias in the item—the pattern of connection between item responses and trait levels is the same for males and females. d. This suggests the absence of bias in the item females on average have a higher level of mechanical aptitude.
16. One way that test developers try to detect and understand construct bias is through a procedure called Item Response Theory (IRT). Say that a test developer is evaluating a new measure of mechanical aptitude, gathers test data from a sample of males and a sample of females, conducts an IRT-based analysis of the data, and obtains the two Item Characteristics Curves below for one of the items on the test. Which of the following statements is a correct interpretation of this result?
Ans: A
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Application
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Hard a. criterion score bias b. predictive bias c. slope bias d. intercept bias
17. Detection of ___________ begins with the assumption that "one size fits all."
Ans: B
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Knowledge
Answer Location: Detecting Predictive Bias: External Evaluation of a Test
Difficulty Level: Easy
18. Suppose a teacher creates a multiplication test, and its items involve questions related to American football (e.g., “how many yards does a team need to gain in order to get three first-downs?”). So, to do well on the test, respondents must be familiar with football and have an understanding of multiplication. This could raise concerns about construct bias because: a. Among people who are familiar with football, a poor test score indicates a “lack of understanding of multiplication,” however, among people who are NOT familiar with football, the same poor test score does not necessarily indicate “lack of understanding of multiplication.” b. Among people who are familiar with football, those who do well on the test actually “lack an understanding of multiplication” whereas those who do poorly actually have a good “understanding of multiplication.” c. Among people who are not familiar with football, those who do well on the test truly do have a good “understanding of multiplication,” whereas those who do poorly truly have a poor “understanding of multiplication.” d. Among people who are not familiar with football, a poor test score indicates a good "understanding of multiplication," whereas those who do well on a test have a poor "understanding of multiplication."
Ans: A
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Analysis
Answer Location: Detecting Predictive Bias: External Evaluation of a Test Difficulty Level: Medium a. When creating and revising tests, test developers employ multiple strategies to predict which items will show group-based biases and which will not. If the test developers identify items that they believe will probably suffer from bias, then predictive bias occurs. b. Test scores are used to represent or predict actual levels of the characteristic being measured by the test (e.g., scores on the WAIS intelligence test are used to represent or predict actual levels of intelligence). If a test’s scores are not related to “true levels” of the underlying characteristic in the same way for different groups, then the test suffers from predictive bias. c. Decision-makers use test-based information to help choose which people to admit, select, hire, promote, etc. This practice is based on the idea that test scores can help the decision-makers predict which people will perform well. Predictive bias concerns the degree to which test scores are related similarly to performance for different groups of test-takers. d. Test developers avoid writing items with double negatives because such items would prevent us from detecting predictive bias and could prevent us from intervening to deal with any biases that may have occurred.
19 Which of the following is the best description of predictive bias?
Ans: C
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Comprehension
Answer Location: Detecting Predictive Bias: External Evaluation of a Test Difficulty Level: Medium
20. Imagine that we are hiring for a sales department of a large corporation. We believe that the personality trait “Extraversion” is relevant to performance as a salesperson, so we want to use applicants’ extraversion test scores to help us decide who to hire (preferring people who score higher on extraversion). Say we gather data in which we measure the extraversion levels of salespeople in the corporation and we examine extraversion’s link to “number of dollars in sales.” We run a regression analysis in which we use Extraversion to predict the “number of dollars in sales ” We get the following results:
Sales = 10,000 + 5,000 (Extraversion) a. A salesperson with an average level of extraversion is expected to have $10,000 in sales. b. A salesperson with a score of 0 on the extraversion test is expected to have $10,000 in sales. c. A salesperson with an average level of extraversion is expected to have $5,000 in sales d. A salesperson with a score of 0 on the extraversion test is expected to have $5,000 in sales
Based on these results (and assuming that the intercept and slope are statistically significant), which of the following statements is valid?
Ans: B
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Application
Answer Location: Intercept Bias
Difficulty Level: Hard
21 Let’s imagine now that we run the regression analysis separately for female and male salespeople in the sample. Say we get the following results:
Females: Sales = 15,000 + 5,000 (Extraversion)
Males: Sales = 5,000 + 5,000 (Extraversion) a. A female with an extraversion score of 20 is predicted to have $115,000 in sales, while a male with the same score is predicted to have only $105,000 in sales b. A one-unit increase in extraversion is associated with a larger increase in sales for females than for males. c. For every 5,000-point increase in extraversion test scores, sales increases by one dollar, and this is true for both females and males. d. “Slope bias” exists.
Which of the following is an implication of this result?
Ans: A
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Application
Answer Location: Intercept Bias
Difficulty Level: Hard
22. Say we ran the regression analysis for female and male salespeople using both extraversion and sex as predictors of sales in the entire sample. In the sample, we found the following results:
Females: Sales = 10,000 + 2,000 (Extraversion)
Males: Sales = 10,000 + 8,000 (Extraversion) a. There is no predictive bias, as the sales for the average male and the average female are identical at $10,000 b. The difference (in sales) between females and males is not the same at all levels of extraversion (e.g., the sex difference in sales is relatively small at low levels of extraversion, but relatively large and higher levels of extraversion) c. Extraversion is more strongly associated with sales among females than among males d. The average male has $6,000 more in sales than does the average female.
Which of the following is an implication of this result?
Ans: B
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Application
Answer Location: Intercept Bias
Difficulty Level: Hard
23 If we do a regression analysis and find ________________, then we would conclude that two people (from different groups) who have the same actual trait/ability obtain different scores on the measure of that trait/ability a. intercept bias b. construct bias c. regression bias d. criterion bias
Ans: A
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Comprehension
Answer Location: Intercept Bias
Difficulty Level: Medium
24 A researcher studies potential test bias in a new test designed to measure mechanical aptitude (in terms of gender bias). She administers the test to a large sample of males and females who were hired at an automobile assembly plant Later, she collects information about their performance in a variety of mechanical tasks to obtain a criterion variable of “mechanical performance” She conducts a regression analysis to evaluate the degree to which test scores are more predictive of performance for males than for females (or vice versa). Here are the results of her analysis:
Females: Performance = 43.56 + 4.56 (Test score)
Males: Performance = 39.32 + 4.56 (Test score) a. the degree of bias changes across scores on the test b. a male with a “0” on the test is predicted to achieve a score of 39.32 on the performance variable c. for a male and female who have the same test score, the male is predicted to have a higher score on the performance variable. d. among females, a one-unit difference in test scores is associated with a 4.56-point difference in the performance variable.
Which of the following is not a legitimate conclusion based on these results?
Ans: A
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Analysis
Answer Location: Intercept Bias
Difficulty Level: Medium a. criterion score bias b. predictive bias c. slope bias d. intercept bias
25. _____________ can occur when there is differential validity and is more accurate for one group than for another.
Ans: C
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Knowledge
Answer Location: Slope Bias
Difficulty Level: Easy
True/False
1. For composite tests, the overall degree of test score bias is determined by the bias associated with each of the items in the test.
Ans: T
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium
2. If two discrimination index values are approximately equal, this would indicate the item reflects the construct in a different way for each item, and we can conclude that it is not biased.
Ans: F
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Item Discrimination Index
Difficulty Level: Medium
3. Factor analysis can be used to evaluate the internal structure of a test separately for two groups of people.
Ans: T
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Factor Analysis
Difficulty Level: Medium
4. IRT analyses are simple and straightforward, therefore have little to no downside.
Ans: F
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Knowledge
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Easy
5. Predictive bias is of great concern in important areas of applied psychological and educational measurement.
Ans: T
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Comprehension
Answer Location: Detecting Predictive Bias: External Evaluation of a Test
Difficulty Level: Medium
6 Say we ran a multiple regression analyses, using both extraversion and sex as predictors of sales in the entire sample. Say the “sex” variable is coded as 0 = Male, 1 = Female: Sales = 5,000 + 5,000 (Extraversion) + 10,000 (Sex).
For a male and female who have the same extraversion test score, the female is likely to have $10,000 more in sales
Ans: T
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Analysis
Answer Location: Intercept Bias
Difficulty Level: Medium
7. Test fairness is the degree to which a test’s scores are equally valid (in representing a particular construct) for two groups of respondents
Ans: F
Learning Objective: 11-3: Distinguish test fairness from test score bias and why test fairness is an important issue in psychological testing.
Cognitive Domain: Knowledge
Answer Location: Test Fairness
Difficulty Level: Easy
8 Test bias is how a test is used or misused in terms of a broad set of social/economic/etc. considerations
Ans: F
Learning Objective: 11-3: Distinguish test fairness from test score bias and why test fairness is an important issue in psychological testing.
Cognitive Domain: Knowledge
Answer Location: Test Fairness
Difficulty Level: Easy
9. Test fairness refers to a psychometric property of a test.
Ans: F
Learning Objective: 11-3: Distinguish test fairness from test score bias and why test fairness is an important issue in psychological testing.
Cognitive Domain: Knowledge
Answer Location: Test Fairness
Difficulty Level: Easy
10. UC’s analysis of the SAT provided a real-life examination of test bias.
Ans: T
Learning Objective: 11-3: Distinguish test fairness from test score bias and why test fairness is an important issue in psychological testing.
Cognitive Domain: Comprehension
Answer Location: Example: Is the SAT Biased in Terms of Race or Socioeconomic Status?
Difficulty Level: Medium
Short Answer
1. If a test suffers from “construct bias,” then its (choose A or B below): a. test scores are not related to a criterion variable in the same way for people in two different groups b. test scores do not mean the same thing, psychologically speaking, for people in two different groups
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium a. This difference in group averages is evidence of test bias against males. b. This difference in group averages might (or might not) be due to test bias against males.
2. A test developer creates a new test to assess academic aptitude. Imagine that males tend to score lower on this test, on average, than do females. Which of the following statements is true, psychometrically speaking: A or B?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium
3. Construct bias is often evaluated by examining responses to individual items on a test. Therefore, when would a test item be considered biased? Name one of the two qualifications.
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium
4. What is one of the methods for detecting construct bias?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Detecting Construct Bias: Internal Evaluation of a Test
Difficulty Level: Medium
5. Which approach to evaluating construct bias is independent of the number of people in the groups that are being compared who answer an item correctly?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Item Discrimination Index
Difficulty Level: Medium
6. “Measurement invariance” is most associated with what approach to evaluating construct bias?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Factor Analysis
Difficulty Level: Medium
7 A procedure called Confirmatory Factor Analysis (CFA) is often used to evaluate construct bias. From the perspective of CFA, this issue is framed in terms of “measurement invariance.” Measurement invariance is (A or B): a. good, because it means that the factorial structure (dimensionality) of a test is the same across groups, suggesting no bias b. bad, because it means that the factorial structure differs across groups, suggesting bias
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Analysis
Answer Location: Factor Analysis
Difficulty Level: Medium
8. Which approach to evaluating construct bias is a feature of a psychometric approach called item response theory?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Medium
9. Is it uniform or non-uniform bias in a situation where ICCs differ in shape as well as location?
Ans: Answer may vary.
Learning Objective: 11-1: Describe the procedures used to estimate the existence and degree of construct bias.
Cognitive Domain: Comprehension
Answer Location: Differential Item Functioning Analyses
Difficulty Level: Medium
10 If a test suffers from “predictive bias,” then its (A or B): a. test scores are not related to a criterion variable in the same way for people in two different groups b. test scores do not mean the same thing, psychologically speaking, for people in two different group
Ans: Answer may vary.
Learning Objective: 11-2: Explain the concerns of predictive bias and how researchers detect prediction bias.
Cognitive Domain: Analysis
Answer Location: Detecting Predictive Bias: External Evaluation of a Test
Difficulty Level: Medium
Chapter 12: Confirmatory Factor Analysis
Test Bank
Multiple Choice
1. When is EFA most appropriate in test use?
a. early phases b. mid-phase c. later phases d. anytime; EFA is not more appropriate at one time or another
Ans: A
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium a. early phases b. mid-phase c. later phases d. anytime; EFA is not more appropriate at one time or another
2. When is CFA most appropriate in test use?
Ans: C
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium a. CFA b. EFA c. a hybrid of CFA and EFA d. either factor analysis would be equally appropriate
3. What kind of factor analysis is most appropriate for use when test developers have gone through initial evaluations of item properties and dimensionality and after significant revisions of test content?
Ans: A
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium a. CFA b. EFA c. a hybrid of CFA and EFA d. either factor analysis would be equally appropriate
4. What kind of factor analysis is most appropriate for use when test developers are clarifying their understanding of the constructs and of the test itself?
Ans: B
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium a. CFA b. EFA c. a hybrid of CFA and EFA d. either factor analysis would be equally appropriate
5. Which factor analysis allows test developers and test evaluators to understand the degree to which their hypothesized measurement models are consistent with actual data produced by respondents?
Ans: A
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: Using CFA to Evaluate Measurement Models
Difficulty Level: Medium a. self-alienation b. authentic living c. accepting external influence d. dynamic self-exploration
6. Which of the following is not part of the authenticity scale?
Ans: D
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Overview of CFA and an Example
Difficulty Level: Medium a. the degree to which a person really understands himself or herself b. the degree to which a person behaves and expresses emotion in a way that is an honest reflection of his or her self-perception c. the degree to which a person understands that other people can influence one’s life and conforms to these influences d. the degree to which a person continually seeks new experiences as a way toward inner exploration
7. What is the definition of self-alienation?
Ans: A
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Overview of CFA and an Example
Difficulty Level: Easy a. the degree to which a person really understands himself or herself b. the degree to which a person behaves and expresses emotion in a way that is an honest reflection of his or her self-perception c. the degree to which a person understands that other people can influence one’s life and conforms to these influences d. the degree to which a person continually seeks new experiences as a way toward inner exploration
8. What is the definition of authentic living?
Ans: B
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Overview of CFA and an Example
Difficulty Level: Easy a. the degree to which a person really understands himself or herself b. the degree to which a person behaves and expresses emotion in a way that is an honest reflection of his or her self-perception c. the degree to which a person understands that other people can influence one’s life and conforms to these influences d. the degree to which a person continually seeks new experiences as a way toward inner exploration
9. How is ‘accepting external influence’ best defined?
Ans: C
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Overview of CFA and an Example
Difficulty Level: Easy a. clarifying the psychological construct to be assessed and initial development of the test items b. collecting a large number of responses to his test c. reversing any negatively keyed items d. specifying the measurement model
10. Rahul is conducting a CFA and working through the preliminary steps. What is the first step Rahul is likely to take?
Ans: A
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Analysis
Answer Location: Preliminary Steps
Difficulty Level: Medium a. specification of the measurement model b. computations c. interpreting and reporting output d. model modification and reanalysis
11. Which step of conducting a CFA involves translating our hypothesized measurement model into a statistical software package designed to conduct CFA?
Ans: A
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Step 1: Specification of the Measurement Model
Difficulty Level: Medium a. specification of the measurement model b. computations c. interpreting and reporting output d. model modification and reanalysis
12. Which step of conducting a CFA involves asking our statistical software to conduct the CFA based on specifications and the data we have collected?
Ans: B
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Step 2: Computations
Difficulty Level: Medium a. specification of the measurement model b. computations c. interpreting and reporting output d. model modification and reanalysis
13. When Asa is in the _______________ step of conducting a CFA, he examines the fit indices that address the overall adequacy of his hypothesized measurement model.
Ans: C
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Analysis
Answer Location: Step 3: Interpreting and Reporting Output
Difficulty Level: Medium a. specification of the measurement model b. computations c. interpreting and reporting output d. model modification and reanalysis
14. What is the last step of conducting a CFA?
Ans: D
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Step 4: Model Modification and Reanalysis (If Necessary)
Difficulty Level: Medium a. it is not the same across groups b. it is not good c. it has evidence against construct bias d. its test scores likely have different interpretations
15. Which of the following is true about a test’s internal structure that is invariant?
Ans: C
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: The Meaning of Measurement Invariance
Difficulty Level: Medium a. it is not the same across groups b. it is good c. it has evidence against construct bias d. its test scores likely have similar interpretations
16. Which of the following is true about a test’s internal structure that lacks invariance?
Ans: A
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: The Meaning of Measurement Invariance
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
17. Which of the following has some invariance (the least amount of invariance)?
Ans: A
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Knowledge
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
18. Which of the following has invariance for all item parameters?
Ans: D
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Knowledge
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
19. Which of the following has much more invariance, but not all?
Ans: C
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Knowledge
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
20. Which of the following has somewhat more invariance (more than ‘some’ but less than ‘even more’)?
Ans: B
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates
Cognitive Domain: Knowledge
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
21. Which level of variance has residual variances (i.e., each item’s error variance is the same in each group)?
Ans: D
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
22. Which level of variance does not have exact factor loadings (i.e., each item’s factor loading is the same across groups)?
Ans: A
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
23. Which level of variance does not have associations between latent variables?
Ans: A
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. configural MI b. weak/metric MI c. strong/scalar MI d. strict MI
24. Which level of variance has group differences in items (i.e., item covariances, variances, and means) due to group differences on the latent variables?
Ans: D
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium a. Items are affected by the same LVs in each group (i.e., the items reflect the same LVs) b. The groups have the same unit of measurement c. No systematic response biases across groups d. Group differences in the items (i.e., item covariances, variances, and means) are due to group differences in the latent variables
25. Which of the following implications affects each level of measurement invariance?
Ans: A
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: Levels of Invariance: Meaning and Detection
Difficulty Level: Medium True/False
1. EFA is most appropriate during later phases of test use.
Ans: F
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium
2. CFA is most appropriate during later phases of test use.
Ans: T
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium
3. CFA can be an iterative, back-and-forth process.
Ans: T
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Overview of CFA and an Example
Difficulty Level: Medium
4. The last step of CFA involves specifying the links between items and factors.
Ans: F
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Step 1: Specifications of the Measurement Model
Difficulty Level: Easy
5. In the first phase of CFA computations, the collected data are used to compute the items’ variances and the covariances among the items.
Ans: T
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Step 2: Computations
Difficulty Level: Easy
6. When conducting a CFA of a test, test developers and evaluators often evaluate competing measurement models.
Ans: T
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Comparing Models
Difficulty Level: Medium
7. The alpha method allows us to estimate reliability in a way that is more widely valid and applicable than a CFA-based approach.
Ans: F
Learning Objective: 12-3: Summarize how confirmatory factor analysis can evaluate reliability.
Cognitive Domain: Comprehension
Answer Location: Estimating Reliability (Omega Index)
Difficulty Level: Medium
8. Omega may be valid when test items are essentially tau-equivalent, but not parallel or congeneric.
Ans: F
Learning Objective: 12-3: Summarize how confirmatory factor analysis can evaluate reliability.
Cognitive Domain: Analysis
Answer Location: Estimating Reliability (Omega Index)
Difficulty Level: Medium
9. Multitrait-multimethod can be used with CFA to evaluate convergent and discriminant validity.
Ans: T
Learning Objective: 12-4: Summarize how confirmatory factor analysis can evaluate validity.
Cognitive Domain: Comprehension
Answer Location: CFA and Validity
Difficulty Level: Medium
10. There has been a rapidly decreasing use of CFA to examine group differences in tests’ psychometric properties.
Ans: F
Learning Objective: 12-5: Discuss measurement invariance, its levels, and how it relates to confirmatory factor analysis.
Cognitive Domain: Comprehension
Answer Location: CFA and Measurement Invariance
Difficulty Level: Medium
Short Answer
1. Is EFA considered more hypothesis-driven or exploratory?
Ans: Answer may vary.
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium
2. Is CFA considered more hypothesis-driven or exploratory?
Ans: Answer may vary.
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Analysis
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Medium
3. What technique is a blend of EFA and CFA?
Ans: Answer may vary.
Learning Objective: 12-1: Compare and contrast exploratory factor analysis and confirmatory factor analysis.
Cognitive Domain: Knowledge
Answer Location: The Frequency and Roles of EFA and CFA
Difficulty Level: Easy
4. What is the name given to the scale which was intended to measure the degree to which a person “knows himself or herself” and “acts accordingly”?
Ans: Answer may vary.
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Overview of CFA and an Example
Difficulty Level: Medium
5. What is the first step needed before conducting a CFA? Ans: Answer may vary.
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Preliminary Steps
Difficulty Level: Easy
6. Which step of conducting a CFA involves linking items to only one latent variable? Ans: Answer may vary.
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Knowledge
Answer Location: Step 1: Specification of the Measurement Model
Difficulty Level: Easy
7. What is one of the phases of computation? Ans: Answer may vary.
Learning Objective: 12-2: Describe the process of conducting confirmatory factor analysis and how to interpret its results.
Cognitive Domain: Comprehension
Answer Location: Step 2: Computations
Difficulty Level: Medium
8. What is one of the (at least) two important uses of CFA when evaluating reliability? Ans: Answer may vary.
Learning Objective: 12-3: Summarize how confirmatory factor analysis can evaluate reliability.
Cognitive Domain: Comprehension
Answer Location: CFA and Reliability
Difficulty Level: Medium
9. What is one of the steps that can help test users understand which (if any) specific CTT-based measurement model fits a test’s responses?
Ans: Answer may vary.
Learning Objective: 12-3: Summarize how confirmatory factor analysis can evaluate reliability.
Cognitive Domain: Knowledge
Answer Location: Evaluating Types of CTT Measurement Models
Difficulty Level: Easy
10. What is one way CFA can be used to examine convergent validity? Ans: Answer may vary.
Learning Objective: 12-4: Summarize how confirmatory factor analysis can evaluate validity.
Cognitive Domain: Comprehension
Answer Location: CFA and Validity
Difficulty Level: Medium
Chapter 13: Generalizability Theory
Test Bank
Multiple Choice
1. Which of the following is true about CTT?
a. error can be differentiated b. can tease apart the effects that multiple facets of a measurement strategy might have on scores c. applicable to simple measurement designs d. has one or more facets
Ans: C
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Analysis
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium a. error can be differentiated b. cannot tease apart the effects that multiple facets of a measurement strategy might have on scores c. applicable to simple measurement designs only d. has only one facet
2. Which of the following is true about G theory?
Ans: A
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Analysis
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium a. variance b. universe c. population d. design Ans: B
3. In terms of generalizability, _________ is the whole list of items reflecting a certain factor, while a sample is a few items that are representative of that list.
Learning Objective: 13-2: Discuss how generalizability theory relates to generalizability, universes, and variance components.
Cognitive Domain: Knowledge
Answer Location: Generalizability, Universes, and Variance Components
Difficulty Level: Easy
4. G theory is an extension of a. CTT b. domain-sampling theory c. item response theory d. psychometric
Ans: B
Learning Objective: 13-2: Discuss how generalizability theory relates to generalizability, universes, and variance components.
Cognitive Domain: Comprehension
Answer Location: Generalizability, Universes, and Variance Components
Difficulty Level: Medium a. use data from measurements and estimate variance components b. use data from measurements and estimate generalizability of scores’ various measurement strategies c. use estimate variance components and estimate generalizability of scores’ various measurement strategies d. use estimated variance components and estimate "universe" data from measurement strategies
5. Which of the following is true about G study?
Ans: A
Learning Objective: 13-3: Summarize the two-phase process of generalizability theory: G study and D study.
Cognitive Domain: Comprehension
Answer Location: G Studies and D Studies
Difficulty Level: Medium a. use data from measurements and estimate variance components b. use data from measurements and estimate generalizability of scores’ various measurement strategies c. use estimate variance components and estimate generalizability of scores’ various measurement strategies d. use estimated variance components and estimate "universe" data from measurement strategies
6. Which of the following is true about D study?
Ans: C
Learning Objective: 13-3: Summarize the two-phase process of generalizability theory: G study and D study.
Cognitive Domain: Comprehension
Answer Location: G Studies and D Studies
Difficulty Level: Medium a. G study phase b. D study phase c. a nested design d. a crossed design
7. Archie is a researcher using variance components to estimate data collected through a measurement strategy of interest. The facets most strongly affecting observed score variance are identified, and Archie can estimate the effects. What is Archie most likely working through?
Ans: A
Learning Objective: 13-3: Summarize the two-phase process of generalizability theory: G study and D study.
Cognitive Domain: Analysis
Answer Location: G Studies and D Studies
Difficulty Level: Medium a. G study phase b. D study phase c. a nested design d. a crossed design
8. Archie moves on in his study and begins to estimate the generalizability of various combinations of facets. What is Archie most likely working through at this point?
Ans: B
Learning Objective: 13-3: Summarize the two-phase process of generalizability theory: G study and D study.
Cognitive Domain: Comprehension
Answer Location: G Studies and D Studies
Difficulty Level: Medium
9. If a test taker has a score of "1" for three areas (Hostile, Angry, and Belligerent), they will have a target mean of: a. 1 b. 1.3 c. 3.1 d. 3
Ans: A
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Application
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A OneFacet Design
Difficulty Level: Hard a. ANOVA b. sum of squares c. residual effect d. target effects
10. The _________ reflects the degree to which targets elicit different mean ratings averaged across all items.
Ans: D
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Knowledge
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Easy a. ANOVA b. sum of squares c. residual effect d. target effects
11. This design is most likely to be used during Phase 1 of the G study.
Ans: A
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium a. ANOVA b. sum of squares c. residual effect d. target effects
12. The ___________ is the noise that potentially masks the signal of the target effect.
Ans: C
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Knowledge
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Easy a. participants b. targets c. items d. studies
13. The variance components for ___________ indicates the degree to which some _______ elicit higher mean ratings than other _______. (Answer is the same for all blanks.)
Ans: C
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium a. facets b. design c. reliability d. generalizability
14. To conduct a D study, test users estimate "coefficients of __________" for various measurement strategies.
Ans: D
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 2: D Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium a. facets; reliability b. facets; generalizability c. generalizability; reliability d. generalizability; sample
15. A coefficient of _____________ is analogous to _________ as defined by CTT, in that it represents the degree to which the observed differences among target participants are consistent with the differences that would be obtained in a nearly unlimited number of observations were obtained.
Ans: C
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 2: D Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium a. variance b. signal c. random d. fixed
16. Generalizability coefficient = _________ / ______ + Noise (Both blanks are the same.)
Ans: B
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 2: D Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium a. variance; noise b. signal; noise c. facets; noise d. facets; signal
17. In multiple-facet designs, generalizability coefficients are estimated through an analysis of _________ to __________ as represented by ratios of variance components.
Ans: B
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium a. one b. two c. three d. four Ans: A
18. Three components are most likely to be found in a _____-facet design.
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium a. noise b. signal c. components d. targets Ans: A
19. Additional complexity adds complexity to the _______ or error variance of generalizability coefficients.
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium a. signal b. target c. observer d. component Ans: B
20. A large _________ ´ Item interaction would indicate the items operate somewhat inconsistently across the __________(s). (Answer is the same for both blanks.)
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A Two-Facet Design)
Difficulty Level: Medium a. signal b. item c. observer d. component Ans: C
21. A target ´ __________ interaction reflects the degree to which the _______ provides different rank orderings of the target. (Answer is the same for both blanks.)
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A Two-Facet Design)
Difficulty Level: Medium a. observer b. target c. item d. residual Ans: D
22. The _________ represents elements that might produce additional noise in a measurement.
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A Two-Facet Design)
Difficulty Level: Medium a. random b. fixed c. crossed d. nested Ans: B
23. With ________ factors, units of the facet in the study are of primary interest, not a sample from a universe, and not exchangeable with other units.
Learning Objective: 13-6: Describe the variety of generalizability theory strategies based on measurement situations.
Cognitive Domain: Comprehension
Answer Location: Random Versus Fixed Facets
Difficulty Level: Medium a. random b. fixed c. crossed d. nested Ans: D
24. This kind of design occurs when all possible combinations of two facets are not included in the study.
Learning Objective: 13-6: Describe the variety of generalizability theory strategies based on measurement situations.
Cognitive Domain: Comprehension
Answer Location: Crossed Versus Nested Designs
Difficulty Level: Medium a. relative b. absolute c. crossed d. nested
25. These decisions are often called "criterion-referenced tests."
Ans: B
Learning Objective: 13-6: Describe the variety of generalizability theory strategies based on measurement situations.
Cognitive Domain: Comprehension
Answer Location: Relative Versus Absolute Decisions
Difficulty Level: Medium
True/False
1. Validity in a set of scores can be created by different facets of the measurement strategy.
Ans: T
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Comprehension
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
2. CTT can efficiently differentiate the effects of items and observers as separate sources of error.
Ans: F
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Analysis
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
3. CTT is applicable to complex measurement designs (having one or more facets).
Ans: F
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Comprehension
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
4. When a psychological or behavioral variable is measured, an infinite number of observations can be made.
Ans: F
Learning Objective: 13-2: Discuss how generalizability theory relates to generalizability, universes, and variance components.
Cognitive Domain: Analysis
Answer Location: Generalizability, Universes, and Variance Components
Difficulty Level: Medium
5. In G theory analysis, estimates of generalizability are based on variance components representing the degree to which differences exist in the “universe” for each element of the design.
Ans: T
Learning Objective: 13-2: Discuss how generalizability theory relates to generalizability, universes, and variance components.
Cognitive Domain: Comprehension
Answer Location: Generalizability, Universes, and Variance Components
Difficulty Level: Medium
6. A high level of generalizability would mean that there are minimal differences among items and does not produce much measurement error.
Ans: T
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A OneFacet Design
Difficulty Level: Medium
7. G theory is not an alternative to CTT.
Ans: F
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Analysis
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium
8. The analysis of a multiple-facet design works much like a one-facet design.
Ans: T
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium
9. In multiple-facet designs, generalizability coefficients are estimated through an analysis of signal to noise as represented by ratios of variance components.
Ans: T
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Analysis
Answer Location: Conducting and Interpreting Generalizability Theory Analysis: A TwoFacet Design
Difficulty Level: Medium
10. The interpretation for the T ´ Ia effect means that targets were rank-ordered differently across observers.
Ans: F
Learning Objective: 13-5: Explain the generalizability theory two-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 1: G Study (Conducting and Interpreting Generalizability
Theory Analysis: A Two-Facet Design)
Difficulty Level: Medium
Short Answer
1. Is G theory or CTT more suited for complex measurement strategies in which multiple facets might affect the scores?
Ans: Answer may vary.
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Analysis
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
2. What two components in CTT make up the total variance in a measure’s observed scores?
Ans: Answer may vary.
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Comprehension
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
3. What term is viewed as undifferentiated, amorphous, and monolithic?
Ans: Answer may vary.
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Comprehension
Answer Location: Multiple Facets of Measurement
Difficulty Level: Medium
4. What is the term used for an element of the measurement strategy, as in G theory?
Ans: Answer may vary.
Learning Objective: 13-1: Describe how different measurement strategies affect scores in generalizability theory.
Cognitive Domain: Knowledge
Answer Location: Multiple Facets of Measurement
Difficulty Level: Easy
5. In terms of G theory, measurement quality is evaluated in terms of the ability to make inferences in two ways. Name one of them.
Ans: Answer may vary.
Learning Objective: 13-2: Discuss how generalizability theory relates to generalizability, universes, and variance components.
Cognitive Domain: Application
Answer Location: Generalizability, Universes, and Variance Components
Difficulty Level: Hard
6. Which phase comes first, G study or D study?
Ans: Answer may vary.
Learning Objective: 13-3: Summarize the two-phase process of generalizability theory: G study and D study.
Cognitive Domain: Analysis
Answer Location: G Studies and D Studies
Difficulty Level: Medium
7. What is the range for generalizability coefficients?
Ans: Answer may vary.
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Application
Answer Location: Phase 2: D Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Hard
8. What does signal / signal + noise equal?
Ans: Answer may vary.
Learning Objective: 13-4: Explain the generalizability theory one-facet design.
Cognitive Domain: Comprehension
Answer Location: Phase 2: D Study (Conducting and Interpreting Generalizability
Theory Analysis: A One-Facet Design)
Difficulty Level: Medium
9. The residual or error term reflects which: consistency or inconsistency variance?
Ans: Answer may vary.
Learning Objective: 13-7: Describe how variance components tell us about data.
Cognitive Domain: Comprehension
Answer Location: A Practical, Consistency-Oriented Interpretation of Variance
Components
Difficulty Level: Medium
10. In the one-facet design, the target effects and the item effects are what kind of effects ?
Ans: Answer may vary.
Learning Objective: 13-7: Describe how variance components tell us about data.
Cognitive Domain: Comprehension
Answer Location: A Practical, Consistency-Oriented Interpretation of Variance
Components
Difficulty Level: Medium
Chapter 14: Item Response Theory and Rasch Models
Test Bank
Multiple Choice
1. Item response theory is a psychometric approach emphasizing the fact that an individual’s response to a particular __________ is influenced by qualities of the individual and by qualities of the item.
a. sample b. population c. test item d. theory
Ans: C
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Factors Affecting Responses to Test Items
Difficulty Level: Medium a. item difficulty b. item discrimination c. guessing d. individual’s “trait level”
2. Which of the following is not considered an item characteristic that could affect an individual’s response to a test item?
Ans: D
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Factors Affecting Responses to Test Items
Difficulty Level: Medium a Rasch b. 1PL c. GRM d.
3. Which of the following is not considered an item response theory model?
Ans: D
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Factors Affecting Responses to Test Items
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing Ans: A
4. An employee who has a high level of job satisfaction endorses an item that measures job satisfaction. What is this an example of in terms of determinant of item response?
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Respondent Trait Level as a Determinant of Item Responses
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing
5. A very challenging math item is often answered incorrectly. What is this an example of in terms of determinant of item response?
Ans: B
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Item Difficulty as a Determinant of Item Responses
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing
6. Which determinant of item response is defined as the trait level required for participants to have a .50 probability of answering the item correctly?
Ans: B
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Item Difficulty as a Determinant of Item Responses
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing Ans: C
7. Which determinant of item response tries to differentiate individuals who have high trait levels from individuals who have low trait levels?
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Item Discrimination as a Determinant of Item Responses
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing
8. Which determinant of item response is analogous to an item-total correlation from CTT?
Ans: C
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Item Discrimination as a Determinant of Item Responses
Difficulty Level: Medium
9. What is true about the following question: How many pecks are in three bushels? (a)
12 (b) 24 a. it has a high discrimination value b. it has a low discrimination value c. it does a good job of discriminating students who have a relatively high level of mathematical ability d. we can feel confident discerning the mathematical ability of up to two responders
Ans: B
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Item Discrimination as a Determinant of Item Responses
Difficulty Level: Medium a. respondent trait level b. item difficulty c. item discrimination d. guessing
10. Which component is related to the probability that participants will answer an item correctly, purely based on chance?
Ans: D
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Guessing
Difficulty Level: Medium a. item difficulty and item discrimination b. item discrimination and respondent trait level c. respondent trait level and guessing d. item difficulty and guessing
11. Which components will produce a correct answer 50% of the time?
Ans: D
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Guessing
Difficulty Level: Medium a. partial credit model b. graded response model c. generalized partial credit model d. nominal response model
12. Which of the following is not a polytomous format for difficulty and discrimination parameters?
Ans: A
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: IRT Measurement Models
Difficulty Level: Medium a. 1PL/Rasch Model b. 2PL c. 3PL d. graded response (GRM)
13. Which of the following is the simplest IRT model?
Ans: A
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Knowledge
Answer Location: One-Parameter Logistic Model (or Rasch Model)
Difficulty Level: Easy a. 1PL/Rasch Model b. 2PL c. 3PL d. graded response (GRM)
14. Which IRT model only relates to item difficulty?
Ans: A
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: One-Parameter Logistic Model (or Rasch Model)
Difficulty Level: Medium a. 1PL/Rasch Model b. 2PL c. 3PL d. graded response (GRM)
15. According to this model, a person’s response to a binary item is determined by the individual’s trait level, item difficulty, and item discrimination.
Ans: B
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: Two-Parameter Logistic Model
Difficulty Level: Medium a. 1PL/Rasch Model b. 2PL c. 3PL d. graded response (GRM)
16. Which model states that the probability of a respondent answering an item correctly is conditional on the respondent’s trait level, the item’s difficulty, the item’s discrimination, and a "guessing parameter"?
Ans: C
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: Three-Parameter Logistic Model
Difficulty Level: Medium a. 1PL/Rasch Model b. 2PL c. 3PL d. graded response (GRM)
17. Which model produces several difficulty parameters for each item?
Ans: D
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: Graded Response Model
Difficulty Level: Medium a. compute probabilities b. initial estimates c. compare probabilities d. retain most recent estimates
18. When estimating trait levels and item parameters, what happens after getting the actual responses to the test?
Ans: B
Learning Objective: 14-3: Discuss how to estimate IRT-based parameters.
Cognitive Domain: Analysis
Answer Location: Obtaining Parameter Estimates: A 1PL Example
Difficulty Level: Medium a. model b. parameter c. characteristic d. application
19. Item _______________ curves present and evaluate characteristics of the items on a test.
Ans: C
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Knowledge
Answer Location: Item Characteristic Curves
Difficulty Level: Easy a. lower b. greater c. mixed d. immeasurable
20. Higher information values indicate _________ psychometric quality.
Ans: B
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Analysis
Answer Location: Item Information and Test Information
Difficulty Level: Medium
21. If we compute values at multiple trait levels, we could display the results in a graph called an: a. item characteristic curve b. respondent trait level c. model fit d. item information curve
Ans: D
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Comprehension
Answer Location: Item Information and Test Information
Difficulty Level: Medium a. item characteristic curve b. respondent trait level c. model fit d. item information curve
22. What might we use to illustrate the degree to which a test provides a different quality of information at different trait levels?
Ans: D
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Comprehension
Answer Location: Item Information and Test Information
Difficulty Level: Medium a. item information curve b. model fit c. differential item functioning d. person fit
23. What occurs when an item’s properties in one group are different from the item’s properties in another group?
Ans: C
Learning Objective: 14-6: Discuss how IRT can be applied to a variety of psychological domains.
Cognitive Domain: Comprehension
Answer Location:
Difficulty Level: Medium a. model fit b. computerized adaptive testing c. differential item functioning d. person fit
24. The analysis of __________ is an attempt to identify individuals whose response pattern does not seem to fit any of the expected patterns of responses to a set of items.
Ans: D
Learning Objective: 14-6: Discuss how IRT can be applied to a variety of psychological domains.
Cognitive Domain: Knowledge
Answer Location: Person Fit
Difficulty Level: Easy a. model fit b. computerized adaptive testing c. differential item functioning d. person fit
25. _______________ works by using a very large item pool for which IRT has been used to obtain information about the psychometric properties of items.
Ans: B
Learning Objective: 14-6: Discuss how IRT can be applied to a variety of psychological domains.
Cognitive Domain: Knowledge
Answer Location: Computerized Adaptive Testing
Difficulty Level: Easy
1. CTT is more conceptually complex than IRT.
Ans: F
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Analysis
Answer Location: Item Response Theory and Rasch Models
Difficulty Level: Medium
2 Discrimination can affect an individual’s response to a test item.
Ans: T
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Factors Affecting Responses to Test Items
Difficulty Level: Medium
3. The Rasch model is related to discrimination and guessing.
Ans: F
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Comprehension
Answer Location: IRT Measurement Models
Difficulty Level: Medium
4. The difference between 1PL and 2PL is the inclusion of the item discrimination parameter
Ans: T
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Analysis
Answer Location: Two-Parameter Logistic Model
Difficulty Level: Medium
5. There is statistical software that can do the computational work needed for a variety of response models.
Ans: T
Learning Objective: 14-3: Discuss how to estimate IRT-based parameters.
Cognitive Domain: Comprehension
Answer Location: Obtaining Parameter Estimates: A 1PL Example
Difficulty Level: Medium
6. In an IRT analysis, item characteristics are combined to reflect the characteristics of the test as a whole.
Ans: T
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Comprehension
Answer Location: Item and Test Information
Difficulty Level: Medium
7. Items do not have curves.
Ans: F
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Comprehension
Answer Location: Item Characteristic Curves
Difficulty Level: Medium
8. Item information curves are unable to vary in terms of location.
Ans: F
Learning Objective: 14-5: Describe how IRT provides information about items and tests.
Cognitive Domain: Comprehension
Answer Location: Item Information and Test Information
Difficulty Level: Medium
9. IRT analyses can help create tests with strong psychometric properties across a range of trait levels.
Ans: T
Learning Objective: 14-6: Discuss how IRT can be applied to a variety of psychological domains.
Cognitive Domain: Comprehension
Answer Location: Test Development and Improvement
Difficulty Level: Medium
10. Person fit is a method of computerized test administration that is intended to provide an accurate and very efficient assessment of individuals’ trait levels.
Ans: F
Learning Objective: 14-6: Discuss how IRT can be applied to a variety of psychological domains.
Cognitive Domain: Knowledge
Answer Location: Computerized Adaptive Testing
Difficulty Level: Easy
Short Answer
1. IRT provides procedures for obtaining information about: (name one of three options) Ans: Answer may vary.
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Factors Affecting Responses to Test Items
Difficulty Level: Medium
2. Which factor affecting a person’s probability of responding in a particular way to an item is related to, for example, a person who is good at math being able to respond correctly to a math item?
Ans: Answer may vary.
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Respondent Trait Level as a Determinant of Item Responses
Difficulty Level: Medium
3. Which two determinants of item responses are intrinsically connected?
Ans: Answer may vary.
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Item Difficulty as a Determinant of Item Responses
Difficulty Level: Medium
4. Which two determinants of item responses are scored on a standardized metric so that their means are 0 and the standard deviations are 1?
Ans: Answer may vary.
Learning Objective: 14-1: Describe determinants of responses to test items.
Cognitive Domain: Comprehension
Answer Location: Item Difficulty as a Determinant of Item Responses
Difficulty Level: Medium
5. What term refers to the ability to express the links between an outcome and the components that affect the outcome?
Ans: Answer may vary.
Learning Objective: 14-2: Summarize the measurement models developed from the IRT perspective.
Cognitive Domain: Knowledge
Answer Location: IRT Measurement Models
Difficulty Level: Easy
6. When considering which term is the key question whether the actual responses to a set of items are well represented by a given measurement model?
Ans: Answer may vary.
Learning Objective: 14-4: Identify how to choose the most appropriate model for IRT.
Cognitive Domain: Comprehension
Answer Location: Model Fit
Difficulty Level: Medium
7. How should a test user proceed with interpreting information if the fit indices suggest a good fit between the model and the actual responses?
Ans: Answer may vary.
Learning Objective: 14-4: Identify how to choose the most appropriate model for IRT.
Cognitive Domain: Analysis
Answer Location: Model Fit
Difficulty Level: Medium
