Chapter 7: The Importance of Reliability

Test Bank

Multiple Choice

1. What is the best definition of a “point estimate” of a true score?

a. the range of values within which we strongly believe the person’s true score is likely to be located b. our best guess about a test taker’s actual standing on a particular psychological attribute c. our best guess of the actual reliability of a set of test scores d. the point at which an individual’s true score is greater than the amount of error affecting her/his observed score

Ans: B

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Knowledge

Answer Location: Applied Behavioral Practice: Evaluation of an Individual’s Test Score

Difficulty Level: Easy a. The estimated true score will generally be lower than the individual’s observed score. b. The estimated true score will generally be higher than the individual’s observed score. c. The estimated true score will be less extreme (i.e., closer to the mean of observed scores) than the individual’s observed score. d. The estimated true score will be more extreme (i.e., further away from the mean of observed scores) than the individual’s observed score.

2. When using an “adjusted true score estimate,” how does an individual’s estimated true scores compare to their observed (unadjusted) scores?

Ans: C

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Comprehension

Answer Location: Point Estimates of True Scores

Difficulty Level: Medium a. We are 95% confident that the child’s observed score falls between 950 and 1050. b. We are 95% confident that the child’s true score lies between 950 and 1050 c. The true score is likely to be statistically significant if it is in the range of 950 to 1050. d. The true score is unlikely to be statistically significant if it is in the range of 950 to 1050

3. Say that a confidence interval around a true score was 950 to 1050. Which is the best interpretation of these values?

Ans: B

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Analysis

Answer Location: Point Estimates of True Scores

Difficulty Level: Medium a. 950 to 1050 b. 990 to 1010 c. 910 to 1010 d. 990 to 1050

4. Which is the best (i.e., more precise) confidence interval?

Ans: B

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Comprehension

Answer Location: Confidence Intervals

Difficulty Level: Medium a. There is no systematic association between reliability and the CI (though the CI is heavily affected by validity) b. Better reliability leads to a narrower CI. c. Better reliability leads to a wider CI d. Better reliability leads to a CI that’s more likely to be statistically significant

5. What is the link between test reliability and the confidence interval around an individual’s score?

Ans: B

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Analysis

Answer Location: Confidence Intervals

Difficulty Level: Medium a. observed b. applied c. standard d. normalized

6. According to true score theory, ________ scores are distributed normally around true scores.

Ans: A

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Knowledge

Answer Location: Debate and Alternatives

Difficulty Level: Easy a. estimates b. scores c. reliability d. variance

7. Poor __________ produces scores that are imprecise reflections of individuals’ true psychological traits and skills.

Ans: C

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Knowledge

Answer Location: Summary (Applied Behavioral Practice: Evolution of an Individual’s Test Score)

Difficulty Level: Easy a. attenuation b. Cohen’s D c. correlation d. validity

8. Which term means the effect that measurement error (unreliability) has on the observed score effect size?

Ans: A

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Measurement Error (Low Reliability) Attenuates the Observed Associations Between Measures

Difficulty Level: Easy a. r = 0 b. r = .05 c. r = .20 d. r = .30

9. Imagine you were a psychology researcher and you hypothesized that variable X is correlated with variable Y. You do a study to test that hypothesis you measure both variables, and you compute the correlation between your two measures. Which result would provide the strongest legitimate support for your hypothesis?

Ans: D

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Application

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Hard a. The correlation signifies an important finding with serious practical consequences. b. The correlation indicates a large and robust relationship. c. The correlation is unlikely to have occurred by chance d. There is no way to make a correlation in your analysis.

10. Imagine further that, in your study, your analysis reveals that the correlation is statistically significant. What does statistical significance mean?

Ans: C

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Analysis

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Medium

11 In a research context, low reliability will: a. make the observed effect sizes (e.g., correlations among measures) weaker than they should be b. make the observed effect sizes (e.g., correlations among measures) stronger than they should be c. make the observed effect sizes (e.g., correlations among measures) either weaker or stronger than they should be d. not influence the observed effects, but it will influence the “true” effects

Ans: A

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Analysis

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Medium

12 What is the effect of low reliability on statistical significance? Low reliability will: a. increase the likelihood of getting a “significant” result b. decrease the likelihood of getting a “significant” result c. affect significance in different ways for different effect sizes (making correlations less likely to be significant, but other effect sizes more likely to be significant) d. have no systematic effect on statistical significance

Ans: B

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Analysis

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Medium a. effect size b. true score effect size c. observed score effect size d. statistical significance

13 Which term means a statistical value representing the result of a study as a matter of degree (e.g., the degree of association between variables or the degree of difference between groups)?

Ans: A

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. effect size b. true score effect size c. observed score effect size d. statistical significance

14 Which term means a value representing the result of a study as a dichotomous outcome?

Ans: D

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. effect size b. true score effect size c. observed score effect size d. statistical significance

15. Which term means the result of a study at a psychological level (i.e., that would be obtained if the perfect measurement was possible)?

Ans: B

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. effect size b. true score effect size c. observed score effect size d. statistical significance

16 Which term means the results of a study that is obtained by the research (using imperfect measures)?

Ans: C

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. attenuation b. Cohen’s D c. correlation d. validity

17. Which term means an effect size representing the difference between two groups?

Ans: B

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. attenuation b. Cohen’s D c. correlation d. validity

18 Which term means an effect size representing the degree of association between variables?

Ans: C

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Sizes, and Statistical Significance

Difficulty Level: Easy a. test construction b. item variance c. reliability d. difference scores

19 ____________ affects the observed effect sizes and statistical significance obtained in research.

Ans: C

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Summary (Behavioral Research)

Difficulty Level: Easy a. a general term meaning the degree to which an item differentiates people who score high on the total test from those who score low on the total test b. the reliability estimate that you would obtain if an item were dropped from the test

20 Which is the correct definition for "item discrimination"?

(i.e., the reliability estimated you’d get for a test comprising all the other items) c. the statistical association between an item and the total test score d. the statistical association between an item and the total test score (where the total test score excludes the item in question)

Ans: A

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Test Construction and Refinement

Difficulty Level: Easy a. a general term meaning the degree to which an item differentiates people who score high on the total test from those who score low on the total test b. the reliability estimate that you would obtain if an item were dropped from the test

21 Which is the correct definition for "item-total correlation"?

(i.e., the reliability estimated you’d get for a test comprising all the other items) c. the statistical association between an item and the total test score d. the statistical association between an item and the total test score (where the total test score excludes the item in question)

Ans: C

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy a. a general term meaning the degree to which an item differentiates people who score high on the total test from those who score low on the total test b. the reliability estimate that you would obtain if an item were dropped from the test

22 Which is the correct definition for "alpha if item deleted"?

(i.e., the reliability estimated you’d get for a test comprising all the other items) c. the statistical association between an item and the total test score d. the statistical association between an item and the total test score (where the total test score excludes the item in question)

Ans: B

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy a. a general term meaning the degree to which an item differentiates people who score high on the total test from those who score low on the total test b. the reliability estimate that you would obtain if an item were dropped from the test (i.e., the reliability estimated you’d get for a test comprising all the other items) c. the statistical association between an item and the total test score d. the statistical association between an item and the total test score (where the total test score excludes the item in question)

23 Which is the correct definition for "correlated item-total correlation"?

Ans: D

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy a. interval; effect size b. alpha; omega c. test score; difference score d. mean; variance

24. An item’s _________ and __________ may be related to the consistency between that item and the other items on a test.

Ans: D

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Comprehension

Answer Location: Item Difficulty (Mean) and Item Variance

Difficulty Level: Medium a. effect sizes b. test scores c. variances d. difference scores

25. Items with limited _________ are less likely to have good correlational characteristics than are items with substantial ____________. (Both blanks are the same).

Ans: C

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Comprehension

Answer Location: Item Difficulty (Mean) and Item Variance

Difficulty Level: Medium

True/False

1. The quality and meaningfulness of any research hinges on the quality of the measurement procedures used in that research.

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Analysis

Answer Location: Behavioral Research

Difficulty Level: Medium

2. A fundamental goal of research is to discover how important variables are related to each other.

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Analysis

Answer Location: Reliability, True Associations, and Observed Associations

Difficulty Level: Medium

3. CTT implies that the correlation between two measures is determined by the correlation between psychological constructs and by the reliability of the measures.

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Reliability, True Association, and Observed Associations

Difficulty Level: Medium

4. Cohens d is the effect that measurement error (unreliability) has on the observed score effect size

Ans: F

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Measurement Error (Low Reliability) Attenuates the Observed Associations Between Measures

Difficulty Level: Easy

5 True score effect size is a statistical value representing the result of a study as a matter of degree (e.g., the degree of association between variables or the degree of difference between groups).

Ans: F

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

6 Statistical significance is a statistical value representing the result of a study as a dichotomous outcome.

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

7 True score effect size is the result of a study at a psychological level (i.e., that would be obtained if the perfect measurement was possible).

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

8. Effect size is the results of a study that is obtained by the research (using imperfect measures)

Ans: F

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

9. Correlation is an effect size representing the difference between two groups.

Ans: F

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

10 Correlation is an effect size representing the degree of association between variables.

Ans: T

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Knowledge

Answer Location: Reliability, Effect Size, and Statistical Significance

Difficulty Level: Easy

Short Answer

1. How are confidence intervals computed for various degrees of confidence?

Ans: Answer may vary.

Learning Objective: 7-1: Describe how reliability affects the confidence, accuracy, and precision with which an individual’s true score is estimated.

Cognitive Domain: Comprehension

Answer Location: Debate and Alternatives

Difficulty Level: Medium

2. In terms of psychometrics, what is the most common way of quantifying the association between variables?

Ans: Answer may vary.

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Reliability, True Association, and Observed Associations

Difficulty Level: Medium

3. According to CTT, the correlation between observed scores of two measures is determined by two factors. What is one of them?

Ans: Answer may vary.

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Reliability, True Association, and Observed Associations

Difficulty Level: Medium

4. In terms of observed scores, the correlation between scores from two measures is given by what equation?

Ans: Answer may vary.

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Reliability, True Association, and Observed Associations

Difficulty Level: Medium

5. There are at least three important implications of considering reliability when drawing psychological conclusions from research. What is one of them?

Ans: Answer may vary.

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Implications for Conducting and Interpreting Behavioral Research

Difficulty Level: Medium

6. There are reasons that a researcher may use a low-reliability measure instead of spending the time attempting to develop a highly reliable measure when drawing psychological conclusions from research. What is one reason a researcher may use a low-reliability measure?

Ans: Answer may vary.

Learning Objective: 7-2: Summarize how reliability affects observed effect sizes and statistical significance obtained in research.

Cognitive Domain: Comprehension

Answer Location: Implications for Conducting and Interpreting Behavioral Research

Difficulty Level: Medium

7. What is the general term meaning the degree to which an item differentiates people who score high on the total test from those who score low on the total test?

Ans: Answer may vary.

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Test Construction and Refinement

Difficulty Level: Easy

8 What is the term for a statistical association between an item and the total test score?

Ans: Answer may vary.

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy

9 What is the term for a statistical association between an item and the total test score (where the total test score excludes the item in question)?

Ans: Answer may vary.

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy

10 What is the term for the reliability estimate that you would obtain if an item were dropped from the test (i.e., the reliability estimated you’d get for a test comprising all the other items)?

Ans: Answer may vary.

Learning Objective: 7-3: Discuss the key ways reliability is evaluated and considered in research and practice in behavioral science.

Cognitive Domain: Knowledge

Answer Location: Item Discrimination and Other Information Regarding Internal Consistency

Difficulty Level: Easy

Chapter 8: Validity: Conceptual Basis

Test Bank

Multiple Choice

1. Which of the following is an accurate statement about validity, based on the definition endorsed by AERA, APA, and NCME?

a. Validity is about the interpretations and use of test scores b. Validity is about the precision with which a test’s scores reflect any particular construct. c. Validity is an all-or-none issue, a test is either valid (and thus psychometrically appropriate for use) or not d. Validity is supported by a test user’s experience and personal opinions on the relevant psychological phenomenon.

Ans: A

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Comprehension

Answer Location: What Is Validity?

Difficulty Level: Medium a. “My test is valid.” b. “My test is moderately valid ” c. “My test is valid as a measure of construct X ” d. “My test is moderately valid as a measure of construct X.”

2. Imagine that you develop your own psychological test, and you want to write about validity when describing your analysis of the test. Which of the following statements is phrased in a way that is most consistent with the contemporary definition of validity?

Ans: D

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Application

Answer Location: What Is Validity?

Difficulty Level: Hard a. construct b. content c. criterion d. internal

3. From the traditional perspective, which of the following is not a main type of validity?

Ans: D

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Analysis

Answer Location: What Is Validity?

Difficulty Level: Medium a. A researcher uses a scale that a colleague designed to measure depression. The scale’s scores measure depression but do so very imprecisely (with a great deal of random measurement error). This imprecision harms her ability to detect meaningful psychological effects in her data b. The admissions staff at a university requires applicants to participate in an admissions interview. In this interview process, interviewers rate the applicant’s “capacity for academic achievement.” These interview-based ratings do accept this capacity but do so more precisely for females than for males. This leads the staff to make more well-informed admissions decisions for female applicants than for male applicants. c. A clinical psychologist designs a new scale that he believes to reflect psychopathy, and he uses in his practice and interprets as such when working with clients. In fact, the scale does not measure psychopathy, it instead actually reflects histrionic disorder. This misinterpretation leads to ineffective and potentially harmful work with his clients. d. A lack of construct validity cannot be constructed in one, straightforward example.

4. Which of the following is the most straightforward example of a lack of construct validity?

Ans: C

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Analysis

Answer Location: What Is Validity?

Difficulty Level: Medium a. Evaluate whether test scores are correlated with things that they should be correlated with. b. Evaluate whether test scores are consistent from one time to a later time c. Evaluate whether the different items on a test are consistent with each other d. Evaluate whether test scores do not differ (on average) across groups of respondents

5. Which of the following is the best for evaluating the construct validity of a test or assessment?

Ans: A

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Comprehension

Answer Location: What Is Validity?

Difficulty Level: Medium a. raw scores b. a diverse sample c. reliability d. validity

6. What is perhaps the most crucial issue in a test’s psychometric quality?

Ans: D

Learning Objective: 8-2: Discuss the role and importance of validity in psychological research and practice.

Cognitive Domain: Analysis

Answer Location: The Importance of Validity

Difficulty Level: Medium

7. Loren develops a new test of mathematical ability. She views math ability as a twofactor construct one factor representing calculus ability and one representing noncalculus ability. Her new test includes various types of math items (i.e., calculus, algebra, and geometry). Based on her theory of the “math ability” construct, she hypothesizes that there will be two weakly correlated dimensions to her test a) calculus ability, and b) non-calculus math ability, with all algebra and geometry items loading equally strongly on the non-calc factor. After collecting responses to her test from many students, she finds that the test seems to have a three-dimensional structure (algebra ability, geometry ability, calculus ability) with weakly correlated dimensions. This finding suggests: a. a lack of internal structure validity b. a good level of internal structure validity c. a lack of content validity d. good content validity

Ans: A

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Test Content

Difficulty Level: Medium a. content validity b. face validity c. internal structure validity d. conceptual validity

8 Micha wants to study moral relativism. He designs a new scale to detect who does (and does not) endorse a relativistic moral philosophy. He writes an initial set of items. He then asks experts (professors and students in moral philosophy and moral psychology) to read all of his items and to rate the degree to which each item reflects moral relativism (as opposed to any other construct). Micha is attempting to evaluate __________________.

Ans: A

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Test Content

Difficulty Level: Medium a. content b. face c. internal structure d. conceptual

9. Micha also wants to study moral tolerance, so he develops a new set of items to reflect that construct (to detect who is and is not morally tolerant). He sees moral tolerance as a psychological construct/dimension that is separate from moral relativism (e.g., relativists could either be tolerant or intolerant). He recruits a sample of respondents to answer his moral relativism items and his moral tolerance items. He conducts a factor analysis of the responses to all her items, expecting to find a twofactor structure. Micha is attempting to evaluate __________________ validity.

Ans: C

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Test Content

Difficulty Level: Medium a. you’re evaluating the validity of a test, and you find that the sample of research respondents does not have the construct of interest b. you’re evaluating the validity of a test, and you find that the sample of research respondents does not represent the full range of trait/ability levels of the construct c. a test does not cover the entire range of content that is relevant to its intended construct d. the construct that’s intended to be measured by a test is not well-established as an important psychological attribute worthy of measurement.

10. Construct underrepresentation occurs when _________________________.

Ans: C

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Comprehension

Answer Location: Threats to Content Validity

Difficulty Level: Medium a. you’re evaluating the validity of a test, and you find that the sample of research respondents does not have the construct of interest b. you’re evaluating the validity of a test, and you find that the sample of research respondents does not represent the full range of trait/ability levels of the construct c. a test does not cover the entire range of content that is relevant to its intended construct d. a test covers material that is not part of the construct that the test is intended to measure

11. Construct-irrelevant content occurs when _________________________.

Ans: D

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Comprehension

Answer Location: Threats to Content Validity

Difficulty Level: Medium

12. Let’s say that this class quiz is intended to assess two constructs that we’ll call “knowledge of content validity” and “knowledge of internal structure validity.” Let’s say that the quiz includes an item about discriminant validity. The presence of that item represents: a. content validity b. construct underrepresentation c. construct-irrelevant content d. structural invalidity

Ans: C

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Analysis

Answer Location: Threats to Content Validity

Difficulty Level: Medium a. chart A b. chart B c. chart C d. chart D

13 Robert develops a 12-item test of political attitudes and hypothesizes that it reflects three dimensions conservative/liberal economic attitudes, conservative/liberal social attitudes, and conservative/liberal foreign policy attitudes. He conducts an EFA of responses to his items and evaluates the internal structure validity of his scale. Which of the following scree plots would be the most supportive of his scale as reflecting the proposed construct dimensionality?

Ans: C

Learning Objective: 8-4: Summarize how factor analysis is used to evaluate the internal structure of psychological tests.

Cognitive Domain: Analysis

Answer Location: Factor-Analytic Evidence

Difficulty Level: Medium

14 Mike creates an exam for his Psychological Testing class, and he includes questions about reliability. Some of these questions are intended to assess whether students understand the conceptual and mathematical connections between various concepts (e.g., reliability, true score variance, etc). As part of the exam, he requires students to explain how they arrived at their answers (e.g., to show their computational work, or to explain whether an answer is based upon some basic assumptions of CTT). When grading the test, Mike gives “full credit” only when a correct answer is accompanied by an explanation that reflects students arrived at their answers by recognizing, using, and interpreting the connections correctly. By using this scoring strategy, Mike is most directly attempting to ensure that the exam has: a. response process validity b. convergent validity c. concurrent validity d. discriminant validity

Ans: A

Learning Objective: 8-5: Describe the response process as a type of validity evidence and how some procedures are relatively direct or indirect.

Cognitive Domain: Analysis

Answer Location: Direct Evidence

Difficulty Level: Medium

15 One of the primary ways in which researchers have evaluated the validity of the SAT is by examining the connection between SAT scores and college GPA (under the hypothesis that SAT scores should be correlated with GPA). This procedure reflects an attempt to evaluate: a. criterion validity b. concurrent validity c. predictive validity d. discriminant validity

Ans: C

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Comprehension

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Medium

16. A doctoral student wants to develop a psychotherapeutic treatment for Dependent Personality Disorder (DPD). As part of his dissertation, he develops a new test to measure DPD, but he does not evaluate its psychometric quality very carefully.

He subsequently uses the test in his dissertation. He finds that its scores are indeed affected (i.e., reduced) by a new psychotherapeutic treatment that he has developed. He, therefore, concludes that the new psychotherapeutic treatment is effective at reducing DPD, and he recommends it as a clinical option for DPD.

However, one of his dissertation committee members points to the items on his new measure, and she notes (correctly) that many of them look like they reflect anxiety more so than DPD. Thus, his new measure (which supposedly reflects only levels of DPD) might reflect Generalized Anxiety Disorder (GAD) along with DPD (i.e., the scores likely reflect some messy blend of the two). Based on this, she thus suggests that, unfortunately, his new measure might lack ______________ validity.

Unfortunately, it is thus not clear what his new psychotherapeutic treatment does. Yes, the treatment seems to affect the scores on his new measure. But does that mean that the treatment is a good treatment of DPD or anxiety? Alas, it's not clear, due to the measurement problem that his committee member noticed: a. discriminant validity b. criterion validity c. predictive validity d. consequential validity

Ans: A

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Medium a. large positive correlation b. correlation of about zero c. large negative correlation d. two-factor internal structure

17. Which of the following results would typically be taken as evidence of discriminant validity?

Ans: B

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Comprehension

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Medium a. convergent b. discriminant c. content d. concurrent

18 Suppose a new measure of extraversion possesses a negligible correlation with intelligence (small and non-significant). This would be support for ____________ validity.

Ans: B

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Comprehension

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Medium a. evidence of good internal consistency reliability b. good convergent evidence of validity c. evidence of discriminant validity d. evidence of poor internal consistency reliability

19 Emma creates new a self-report measure of assertiveness and she recruits participants to complete the test. She also asks the participants’ roommates to describe how assertive the participants are. Emma finds that those participants who got high assertiveness scores on her new test were also the ones that were described as very assertive by their roommates. This finding provides ____________________.

Ans: B

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Comprehension

Answer Location: Convergent Evidence

Difficulty Level: Medium a. convergent evidence b. discriminant evidence c. direct evidence d. indirect evidence

20. What is the degree to which test scores are uncorrelated with tests of unrelated constructs?

Ans: B

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Discriminant Evidence

Difficulty Level: Easy a. criterion validity b. concurrent validity c. predictive validity d. discriminant validity

21. Which of the following is the least likely to be linked to the associations between variables?

Ans: D

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Analysis

Answer Location: Criterion, Concurrent, and Predictive Evidence

Difficulty Level: Medium a. Reliability is about a test itself, whereas validity is about test scores b. Validity is about a test itself, whereas reliability is about test scores. c. We can evaluate reliability without regard to the meaning of test scores, whereas validity is primarily about the meaning of test scores. d. Reliability concerns the use of a test, whereas validity concerns the meaning of a test.

22. Which of the following best captures the difference between reliability and validity?

Ans: C

Learning Objective: 8-7: Recall the three key issues regarding consequential validity, including the intended and actual consequences of test use.

Cognitive Domain: Comprehension

Answer Location: Validity Evidence: Consequences of Testing

Difficulty Level: Medium

23. The book described observations made by Shepard (1997) regarding the use of the Medical College Admission Test (MCAT). The MCAT became so central to the medical school admission process, that undergraduate colleges’ “premed” programs began focusing their classes/training only on the narrow types of material covered by the MCAT (biology, chemistry, etc) and ignoring other material that could be crucial in medical training and practice. This shows how a test can shape the broader context in which it is used.

This is an example of: a. intended systemic effects b. unintended differential impact on groups c. unintended systemic effects d. construct Bias

Ans: C

Learning Objective: 8-7: Recall the three key issues regarding consequential validity, including the intended and actual consequences of test use.

Cognitive Domain: Analysis

Answer Location: Evidence Regarding Unintended Systemic Effects

Difficulty Level: Medium

24 Many states in the US require students to take “End of Grade” standardized tests in various subjects, including math. If you read the technical manual for North Carolina’s End of Grade math tests, you will find a description of the test development process. That process involved educators and experts who evaluated potential test items. They read each potential item and determined whether it matched the “standards” (or material that the test should cover, such as geometry, addition, and multiplication). This process of “expert evaluation” of the test items illustrates an attempt to maximize the EoG tests’: a. content validity b. convergent/Discriminant validity c. response process validity d. consequential validity

Ans: D

Learning Objective: 8-7: Recall the three key issues regarding consequential validity, including the intended and actual consequences of test use.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Consequences of Testing

Difficulty Level: Medium a. criterion b. content c. predictive d. construct

25. What is the "hallmark of test validity"?

Ans: A

Learning Objective: 8-8: Summarize alternative perspectives on validity.

Cognitive Domain: Comprehension

Answer Location: Other Perspectives on Validity

Difficulty Level: Medium

True/False

1 Internal structure validity is the degree to which a test covers less material (e.g., items, concepts) than it should if it was intended to reflect a particular construct.

Ans: F

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Test Content

Difficulty Level: Easy

2 Construct-irrelevant content is the degree to which a test covers material (e.g., items, concepts) then it should not cover if it was intended to reflect a particular construct.

Ans: T

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Knowledge

Answer Location: Threats to Construct Validity

Difficulty Level: Easy

3. Validity can, in general, be seen as a concern regarding the meaning/interpretation of test scores

Ans: F

Learning Objective: 8-4: Summarize how factor analysis is used to evaluate the internal structure of psychological tests.

Cognitive Domain: Comprehension

Answer Location: Factor-Analytic Evidence

Difficulty Level: Medium

4. Response process validity is the degree to which the psychological mechanisms that drive individuals’ responses match the mechanisms that should drive their responses.

Ans: T

Learning Objective: 8-5: Describe the response process as a type of validity evidence and how some procedures are relatively direct or indirect.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Response Processes

Difficulty Level: Easy

5 Convergent validity is the degree to which a test’s scores are correlated (either positive or negatively) with other measures that they should be correlated with.

Ans: T

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

6. Discriminant validity is the degree to which a test’s scores are related to a particularly important outcome variable that they should be correlated with.

Ans: F

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

7. Construct validity is the degree to which test scores can be interpreted as reflecting a particular psychological characteristic

Ans: T

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

8 Face validity is the degree to which test scores can be interpreted as reflecting a particular psychological characteristic

Ans: F

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

9. Consequential validity is the degree to which a measure’s actual relationships with other measures correspond to the relationships that the test should have with those other measures.

Ans: F

Learning Objective: 8-7: Recall the three key issues regarding consequential validity, including the intended and actual consequences of test use.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Consequences of Testing

Difficulty Level: Easy

10. Consequential validity is the degree to which the actual effects of using a measure correspond with the effects that should be seen.

Ans: T

Learning Objective: 8-7: Recall the three key issues regarding consequential validity, including the intended and actual consequences of test use.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Consequences of Testing

Difficulty Level: Easy

Short Answer

1. What is one of the key points about the definition of validity?

Ans: Answer may vary.

Learning Objective: 8-1: Describe how the concept of validity has evolved over time.

Cognitive Domain: Comprehension

Answer Location: What Is Validity?

Difficulty Level: Medium

2. What term is the degree to which a test covers material (e.g., items, concepts) that it should cover (if it is to reflect a particular psychological construct)?

Ans: Answer may vary.

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Test Content

Difficulty Level: Easy

3. What term is the degree to which a test has a dimensionality that it should have (if it is to reflect a particular psychological construct)?

Ans: Answer may vary.

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Test Content

Difficulty Level: Easy

4. How is evidence of content validity typically obtained?

Ans: Answer may vary.

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Comprehension

Answer Location: Expert Rating Evidence

Difficulty Level: Medium

5. What term means the degree to which a test appears to be related to a specific construct in the judgment of non-experts?

Ans: Answer may vary.

Learning Objective: 8-3: Identify how an expert rating of test content is an important facet of validity.

Cognitive Domain: Knowledge

Answer Location: Content Validity Versus Face Validity

Difficulty Level: Easy

6. If MSEI scores are validly interpreted and responses exhibit a structure that is consistent with a multidimensional conceptual definition of the construct, should the items appear as separate clusters or one large cluster?

Ans: Answer may vary.

Learning Objective: 8-4: Summarize how factor analysis is used to evaluate the internal structure of psychological tests.

Cognitive Domain: Analysis

Answer Location: Validity Evidence: Internal Structure of the Test

Difficulty Level: Medium

7. What is a form of indirect evidence?

Ans: Answer may vary.

Learning Objective: 8-5: Describe the response process as a type of validity evidence and how some procedures are relatively direct or indirect.

Cognitive Domain: Comprehension

Answer Location: Indirect Evidence

Difficulty Level: Medium

8 What term means the degree to which a test’s scores are related to a particularly important outcome variable that is measured at the same time that the test itself is completed?

Ans: Answer may vary.

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

9 What term means the degree to which a test’s scores are related to a particularly important outcome variable that is measured at a later time (than when the test itself is completed)?

Ans: Answer may vary.

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy

10. What term means the degree to which a test’s scores are uncorrelated (either positive or negatively) with other measures that they should not be correlated with?

Ans: Answer may vary.

Learning Objective: 8-6: Discuss validity evidence’s association between test scores and other variances, including convergent, discriminant, criterion, concurrent, and predictive evidence.

Cognitive Domain: Knowledge

Answer Location: Validity Evidence: Associations With Other Variables

Difficulty Level: Easy