Chapter 6: Empirical Estimates of Reliability

Test Bank

Multiple Choice

1. From the “alternate forms” approach to reliability, how do you estimate reliability?

a. Create two versions of the same test, administer both forms to a single group of participants, and correlate the scores from the two forms. The correlation is your estimate of reliability.

b. Create two versions of the same test, administer both forms to a single group of participants, and correlate the scores from the two forms. Multiply the correlation by two (to reflect the two forms), and that is your estimate of reliability.

c. Create two versions of the same test, administer both forms to a single group of participants, and compute the mean score on each version. The difference between the means reflects the reliability of the test.

d. Create several alternate versions of the same test and determine which one is the best. Estimate the reliability of that test, and it’s the best estimate for the entire test.

Ans: A

Learning Objective: 6-1: Summarize how alternate forms reliability is used for estimating the reliability of test scores.

Cognitive Domain: Comprehension

Answer Location: Alternate Forms Reliability

Difficulty Level: Medium a. the versions must include the same items b. there must be carryover effects (in which error scores from one version are correlated with error scores from the other version) c. there must be *no* carryover effects (in which error scores from one version are correlated with error scores from the other version) d. error scores on each version must be specific

2. For the alternate forms to produce an accurate estimate of reliability, some things have to be true. What must be true for this approach to produce an accurate estimate of reliability?

Ans: C

Learning Objective: 6-1: Summarize how alternate forms reliability is used for estimating the reliability of test scores.

Cognitive Domain: Comprehension

Answer Location: Alternate Forms Reliability

Difficulty Level: Medium

3. If error scores are positively correlated across the versions of a test (e.g., due to carryover effects), then the alternate forms approach will produce an estimate of reliability that is: a. accurate b. too low (compared to the true reliability) c. too high (compared to the true reliability) d. unknowable

Ans: C

Learning Objective: 6-1: Summarize how alternate forms reliability is used for estimating the reliability of test scores.

Cognitive Domain: Comprehension

Answer Location: Alternate Forms Reliability

Difficulty Level: Medium a. it is difficult to create different test “forms” that have identical psychometric properties b. you never know for sure whether different test forms have identical psychometric properties c. it is difficult to have one set of participants take more than one “test” at a time d. you never know for sure whether different test forms are measuring the same true scores (i.e., whether they are measuring the same construct).

4. Which of the following is unlikely to be a problem that might limit the usefulness of the alternative forms approach to estimating reliability?

Ans: C

Learning Objective: 6-1: Summarize how alternate forms reliability is used for estimating the reliability of test scores.

Cognitive Domain: Analysis

Answer Location: Alternate Forms Reliability

Difficulty Level: Medium a. Create two versions of the same test, administer both forms to a single group of participants, and correlate the scores from the two forms. The correlation is your estimate of reliability. b. Create one version of a test, administer it to a single group of participants at two different times, and correlate the scores from the two testing times. The correlation is your estimate of reliability. c. Create one version of a test, administer it to a single group of participants at two different times, and correlate the scores from the two testing times. Multiply the correlation by two (to reflect the two testing occasions), and that’s your estimate of reliability. d. Create one version of a test, administer it twice to a single group of participants, and compute the mean score on each testing occasion. The difference between the means reflects the reliability of the test.

5. From the “test-retest forms” approach to reliability, how do you estimate reliability?

Ans: B

Learning Objective: 6-2: Describe the test-retest method of reliability and how it compares to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Test-Retest Reliability

Difficulty Level: Medium a. respondents’ true scores must change between testing occasions b. scores from the testing occasions must have the same amount of error variance c. the test items must be switched from one occasion to another d. error scores from one testing occasion must be correlated with error scores from another testing occasion.

6. For the test-retest approach to produce an accurate estimate of reliability, some things have to be true. Which must hold true for this approach to produce an accurate estimate of reliability?

Ans: B

Learning Objective: 6-2: Describe the test-retest method of reliability and how it compares to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Test-Retest Reliability

Difficulty Level: Medium a. general intelligence b. extraversion (as a personality trait) c. height d. happiness (as a mood state)

7. Imagine that you are measuring the following characteristics. For which one would the test-retest approach probably not be a good method for estimating reliability?

Ans: D

Learning Objective: 6-2: Describe the test-retest method of reliability and how it compares to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Test-Retest Reliability

Difficulty Level: Medium a. The internal consistency approaches require only one sample of respondents. b. The internal consistency approaches require two versions of a test. c. The internal consistency approaches require only one occasion of testing. d. The internal consistency approaches are appropriate for tests that have more than one item.

8. Internal consistency approaches are the most common way in which people estimate reliability. Why is that? Which factor differentiates internal consistency approaches from alternate forms and test-retest approaches?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Internal Consistency Reliability

Difficulty Level: Medium a. There are multiple ways that you could split a sample of respondents in half, and each way could produce a different estimate of reliability. b. A test must have at least 20 items for this approach to produce an accurate estimate of reliability. c. For most tests, there are multiple ways that you could split the test in half, and each way could produce a different estimate of reliability. d. It’s not very useful to estimate the reliability of a “half-test”, because you want to know the reliability of the full test.

9. What is a challenge that limits the utility of the split-half approach?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Internal Consistency Reliability

Difficulty Level: Medium a. less than .40 b. .40 c. greater than .40 d. no way to tell, based on the information given

10. Imagine that you have an eight-item test, and you want to estimate its reliability via split-half. For each respondent, you create a score for each four-item half-test. You then correlate the two sets of “half test” scores and find that it is .40. What is the estimated reliability of the entire test likely to be?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Application

Answer Location: Internal Consistency Reliability

Difficulty Level: Hard a. .20 b. .40 c. .57 d. 1.12

11. Imagine that you have an eight-item test, and you divide it into halves. For each respondent, you create a score for each half-test. You then correlate the two sets of “half test” scores and find that it is .40. From the split-half approach, what is the estimated reliability of the full test?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Application

Answer Location: Internal Consistency Reliability

Difficulty Level: Hard a. variance of the total test scores b. mean of the total test scores c. number of items d. sum of all the inter-item covariances

12. Which of the following values reflects “consistency” among the parts of a test?

Ans: D

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Internal Consistency Reliability

Difficulty Level: Medium a. raw alpha b. split-half reliability c. alternative forms of reliability d. test-retest reliability

13. What is it called when a test is divided into two parts?

Ans: B

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Knowledge

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Easy a. raw alpha b. split-half reliability c. alternative forms of reliability d. test-retest reliability

14. In which approach must we enter the correlation into a special formula to obtain the estimate of reliability?

Ans: B

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Medium a. raw alpha b. split-half reliability c. alternative forms of reliability d. test-retest reliability

15. Which of the following is an internal consistency estimate of reliability?

Ans: B

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Knowledge

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Easy a. The people who had relatively “elevated” responses to one item tended to also have relatively “low” responses to the other item. b. Individual differences in the responses to one item tend to be consistent with individual differences in the responses to the other item. c. Individual differences in the responses to one item tend to be inconsistent with individual differences in the responses to the other item. d. The two items seem to have nothing in common, in terms of tapping into the same psychological characteristic.

16. What does it mean if two items have a positive covariance or correlation (e.g. if the responses to one item positively co-vary/correlated with responses to the other item)?

Ans: B

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: “Raw” Coefficient Alpha

Difficulty Level: Medium a. In the sample of respondents, we are 95% sure that the raw alpha value is between .866 and .877. b. In the population from which the sample of respondents was drawn, we are 95% sure that the raw alpha value is between .866 and .877. c. We are 86.6% to 88.7% sure that the reliability is .877 in the sample of respondents. d. We are 86.6% to 88.7% sure that the reliability is .877 in the population from which the sample of respondents was drawn.

17. You conduct a reliability analysis of a set of test scores in SPSS and get the results below. Which of the following is the most accurate interpretation of the .866 and .887 values?

Ans: B

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Application

Answer Location: “Raw” Coefficient Alpha

Difficulty Level: Hard a. when there are relatively weak correlations among the items b. when the correlations among the items are too strong c. when some items have much greater variance than do other items d. when some items have much higher means than do other items

18. In which of the following circumstances would you strongly consider standardizing the responses to items before aggregating (i.e., summing or averaging) into a “total score” for a test?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: “Standardized” Coefficient Alpha

Difficulty Level: Medium a. number of “parts” = 1 b. number of “parts” = 2 c. number of “parts” = the number of items on the test d. number of “parts” = the number of factors revealed by an exploratory factor analysis

19. All “internal consistency” methods of estimating reliability are based upon “consistency among the parts of a test.” For the alpha method (raw or standardized), how many “parts of a test” are there?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Application

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Hard a. shorten the test by removing items that are consistent with the other items on the test b. lengthen the test by adding items that are relatively inconsistent with the other items on the test c. removing items that detract from internal consistency and replace them with items that enhance internal consistency d. removing items that increase internal consistency and replace them with items that reduce internal consistency

20. What might you be able to increase the reliability of a test?

Ans: C

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Medium a. alternate forms b. test-retest c. split-half d. alpha Ans: D

21. Which is the most frequently used approach to estimating reliability?

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Medium a. alpha b. omega c. variability d. split-half Ans: B

22. What term is based on the idea that reliability is a ratio of signal and noise?

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Knowledge

Answer Location: Omega

Difficulty Level: Easy a. find a new test b. standardize the test scores c. try test-retest reliability d. use a new sample set

23. What is a solution to the problem of two test scores with very different variabilities?

Ans: B

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Analysis

Answer Location: The Problem of Unequal Variability

Difficulty Level: Medium a. difference b. true c. z d. raw Ans: A

24. ___________ scores are most meaningful if two test scores have some psychological attribute in common.

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Knowledge

Answer Location: The Problem of Unequal Variability

Difficulty Level: Easy a. difference b. true c. z d. raw

25. Which type of scores can have poor psychometric quality and lead to questionable psychological conclusions, especially as they may suffer from high intercorrelations between component tests, poor reliability of the component tests, or unequal variances in the component tests?

Ans: A

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Analysis

Answer Location: Difference Scores: Summary and Caution

Difficulty Level: Medium

True/False

1 Reliability cannot be estimated for state-like constructs such as emotions or moods, which can change rapidly over time.

Ans: F

Learning Objective: 6-2: Describe the test-retest method of reliability and how it compares to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Test-Retest Reliability

Difficulty Level: Medium

2. For an accurate estimate of reliability, two halves of the split-half approach must be parallel tests.

Ans: T

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Medium

3. Split-half reliability is easily obtained from statistical software.

Ans: T

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Medium

4 The alpha approach produces an accurate estimate of reliability in more circumstances than does the omega approach.

Ans: F

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Medium

5 If we compute alpha for a set of test scores, we can be sure that this value accurately reflects the reliability of the test scores.

Ans: F

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Medium

6. We can estimate reliability, but we can never really know for sure what the reliability of a set of test scores is

Ans: T

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: On the Assumptions Underlying Alpha and Omega, the Relative Applicability of Those Indices, and Their Limitations

Difficulty Level: Medium

7. It is probably not a good idea to treat an internal consistency estimate of reliability (e.g., alpha) as a measure of the conceptual homogeneity or dimensionality of the test.

Ans: T

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Internal Consistency Versus Dimensionality

Difficulty Level: Medium

8. A reliability estimate of .63 is considered on the high end of acceptable.

Ans: F

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Factors Affecting the Reliability of Test Scores

Difficulty Level: Medium

9. The greater the variability among respondents with respect to the psychological attribute being measured, the larger the reliability coefficient.

Ans: T

Learning Objective: 6-4: Describe how heterogeneity affects the size of reliability coefficients.

Cognitive Domain: Comprehension

Answer Location: Sample Homogeneity and Reliability Generalization

Difficulty Level: Medium

10. Anything that increases relative to will decrease the RXX

Ans: F

Learning Objective: 6-4: Describe how heterogeneity affects the size of reliability coefficients.

Cognitive Domain: Analysis

Answer Location: Sample Homogeneity and Reliability Generalization

Difficulty Level: Medium

Short Answer

1. What is it called when a test is divided into two parts?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Knowledge

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Easy

2. What is another name for the internal consistency estimate of reliability?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Knowledge

Answer Location: Split-Half Estimates of Reliability

Difficulty Level: Easy

3. What is the most widely used measure of psychopathology?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Raw Alpha for Binary Items: KR20

Difficulty Level: Medium

4. What does KR20 stand for?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Comprehension

Answer Location: Raw Alpha for Binary Items: KR20

Difficulty Level: Medium

5. What is one advantage Omega has over Alpha?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Omega

Difficulty Level: Medium

6. What is one factor affecting an internal consistency reliability estimate?

Ans: Answer may vary.

Learning Objective: 6-3: Discuss internal consistency reliability and how it is a practical alternative to alternate forms reliability.

Cognitive Domain: Analysis

Answer Location: Factors Affecting the Reliability of Test Scores

Difficulty Level: Medium

7. What is one tool that can be used to address psychometric challenges that may arise with difference scores?

Ans: Answer may vary.

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Comprehension

Answer Location: Reliability of Different Scores

Difficulty Level: Medium

8. What would you use to study change or differences?

Ans: Answer may vary.

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Comprehension

Answer Location: Defining Difference Scores

Difficulty Level: Medium

9. What would the following information be useful in estimating?

(1) the estimated reliability of each set of test scores used to compute the difference scores (RXX and RYY),

(2) the variability of the tests’ observed scores and

(3) the correlation between the observed test scores

Ans: Answer may vary.

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Analysis

Answer Location: Estimating the Reliability of Difference Scores

Difficulty Level: Medium

10. What is one factor that determines whether a set of difference scores has good reliability?

Ans: Answer may vary.

Learning Objective: 6-5: Identify the tools to address psychometric challenges that arise with difference scores.

Cognitive Domain: Comprehension

Answer Location: Factors Affecting the Reliability of Difference Scores

Difficulty Level: Medium