/The-Adequacy-of-Tolls-for-Assessing-1

Session Overview The Adequacy of Tools for Assessing Language and Reading Competence Motivation  Identifying children for Early Intervention  Concepts 

Linda Phillips Denyse Hayward & Eleanor Stewart Canadian Centre for Research on Literacy, University of Alberta

 Prevalence,

Examine PLS, CELF, WRMT-R, ELLA



What can we do?

 Scores,

2

Concepts: PREVALENCE 

Prevalence estimates of SLI in a group of 7,000 kindergarteners = 7.4%



Prevalence estimates of language difficulty in preschoolers = 2% - 19%

Concepts: PREVALENCE 

Prevalence estimates - (50% - 73%) of children exhibiting reading difficulties also have impairments in language abilities (Catts, et al, 2001; McArthur, et, 2000).

4

Concepts: PREVALENCE



  

5

   

Will vary over any given year based on a number of factors (age, SES of mother, environment, prenatal health).



Also there will not be an even split between school boards/areas of the city.

Severity is not accounted for in the estimates (problem/no problem).

A smaller percentage have severe language impairment.

ECMERC Symposium 13-02-08

ECMERC Symposium 13-02-08

Concepts: Basic Properties

How do prevalence estimates help us in the quest to identifying the right children for early intervention?

Helps explain the variability in the number of children we identify each year. 

Prevalence estimates of reading/cognitive difficulties in a group of 16,000 kindergarteners = approx 25% (Lee & Burkam, 2002)



(Communication Facts: Incidence and Prevalence of Communication Disorders ASHA, 2008)

ECMERC Symposium 13-02-08

Accuracy, Administration ECMERC Symposium 13-02-08

(Tomblin et al., 1997)

3

Obtained Scores, Accuracy





6

Adequate Sample Representative Sample Appropriate for Use with the Target Population Tests What it Claims to Test (Validity) Reliability – Consistency in Performance across Time Test Results Appropriately

ECMERC Symposium 13-02-08

1

Concepts: True Score (SEM)  

Concepts: True Score (CI)

Trustworthiness of obtained test score



Standard Error of Measurement (SEM)   

Confidence Intervals/Bands

“You would never consider the score obtained on a standardized test as a perfectly precise measure of ability. It is most appropriate to consider a child’s true score as lying within a range of scores. The confidence band.”

Provides estimate of test error around the test score.

Smaller the SEM = greater confidence in obtained test score (10 ± 1).

Larger the SEM = less confidence in obtained test score (10 ± 5).

(Preschool Language Scale-4 Manual pp.127-128)

Example:

7

ECMERC Symposium 13-02-08

8



86 ± 7 (SEM) 95% confidence band = 79 - 93 ECMERC Symposium 13-02-08

Concepts: True Score (CI) 

9

THE TESTS

Comparing Receptive and Expressive scores “Rather than comparing AC and EC scores it is best to compare the AC and EC confidence bands. You can consider the scores to be different only when there is no overlap in the confidence intervals.”

(PLS-4 Manual p.12)  AC  EC SS 71 Percentile 3 (1-8) SS 67 Percentile 1 (1-4) SS 71 Percentile 3 (5-8)

• Preschool Language Scale (PLS new old) • Clinical Evaluation of Language Fundamentals (CELF new old)

SS 69 Percentile 1 (1-3) ECMERC Symposium 13-02-08

PLS-4 AC Standardized Scores Age for 3:0 to 3:5 years Standard Score 50 51 53 55 57 59 61 67 70 76

11

Percentile 1 1 1 1 1 1 1 1 2 5

PLS-4 EC Standardized Scores Age for 3:0 to 3:5 years

Confidence Band

Standard Score

1-1 1-1 1-1 1-1 1-1 1-1 1-2 1-4 1-6 2-13

ECMERC Symposium 13-02-08

12

50 51 52 55 57 58 59 60 61 62 63 68 71 72

Percentile 1 1 1 1 1 1 1 1 1 1 1 2 3 3

Confidence Band 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-1 1-2 1-2 1-2 1-5 1-7 1-8

ECMERC Symposium 13-02-08

2

ELLA – Early Literacy and Language Test (2006)

PLS - new 

Cleave states:



“If adequate psychometric properties could have been achieved using the smaller ceiling, the test developers would have reported their norms and reliability and validity information using the lower ceiling. The fact that they did not suggests that a ceiling of seven was necessary to achieve acceptable psychometric properties and SLPs should use a ceiling of seven if they wish to report standard scores.”

ECMERC Symposium 13-02-08

13

 

14

ELLA     

15

24 Subtests (From PA to Story Retell) Adm time is not specified Provides no information on either classical or differential item functioning analysis Inconsistencies throughout on whether diagnostic/educational assessment Norming data is insufficient.

WRMT-R Woodcock Reading Mastery Tests - Revised   



17

ECMERC Symposium 13-02-08

TERA – Test of Early Reading Ability

ECMERC Symposium 13-02-08



4:6 – 9:11 years of age Intended to be a developmental and educational assessment tool (NOT a diagnostic tool) – “In other words the results can point to developmental, literacy, or language deficiencies, but cannot be interpreted to reflect the presence of a learning disability in reading or a language disorder.” (ELLA, Manual, p.90).

 

 

3 – 8 years Designed to test reading ability by assessing developing skills rather than readiness for learning Test does not represent the entirety of a child’s reading ability profile Buros, “some items are ambiguous, number of items per age range are limited.”

16

ECMERC Symposium 13-02-08

What to Do??

5 years to 75+ Remains unchanged since 1987 even though it advertised as revised.

To identify areas of strengths and weaknesses, to diagnose reading problems, measure gains, program plan, and research.

Buros, “use of norms from smaller sample means interpretations are limited; scores for special education students should be used cautiously; comparisons of old and new data show a pattern of lower performances and higher standard and percentile scores of lower achievers. This effect could result in overestimation of students’ reading levels.

Limited clinical utility.

ECMERC Symposium 13-02-08

3

What to do?

Careful reporting

• Careful reporting of formal test results • Use of other measures with substantial



• Careful reporting • Practice guidelines

“You would never consider the score obtained on a standardized test as a perfectly precise measure of ability. It is most appropriate to consider a child’s true score as lying within a range of scores. The confidence band.”



CI Sensitivity and Specificity

support

(PLS-4 Manual pp.127-128)

19

ECMERC Symposium 13-02-08

20

Other Measures 





Practice guidelines

Language Sampling 



ECMERC Symposium 13-02-08

50 consecutive intelligible utterances in spontaneous interaction: Norms available



Triangulate your measure Do not use composite scores    

Vocabulary 60 Comprehension 30 Composite 90 Mean 45

5+ years PLS-4 3+ years out CELF-P2  Woodcock Reading Mastery!  

Vocabulary 30 Comprehension 60 Composite 90 Mean 45

21

Standard to drop old test after 2 years

ECMERC Symposium 13-02-08

22

ECMERC Symposium 13-02-08

Concluding Remarks

23

The adequacy of tools for assessing language and reading competence are wanting.

Ensure you use adequate information to make informed decisions.

Watch for the Test of Early Language and Literacy Ability (TELLA) – the first to include language and literacy in a robust and defensible form. Goal to be used in National Screening – Ages 3 - 9

ECMERC Symposium 13-02-08

Contact information: Linda M. Phillips: linda.phillips@ualberta.ca Denyse Hayward: dhayward@ualberta.ca Eleanor Stewart: ges@ualberta.ca

24

ECMERC Symposium 13-02-08

4