Textbook Appraisal: Principles, Process, and Lessons Learned

2nd International Conference on Education and Linguistics at Westminster International University in Tashkent, May 21, 2021 Textbook Appraisal: Principles, Process, and Lessons Learned (Room B, 14:45-15:30 UZT) Presenter: Dr. Flavia Ramos-Mattoussi, Florida State University

Uzbekistan Education for Excellence Program Partner E-mail: framos@lsi.fsu.edu Co-Presenter: Azima Toyirova, Senior Technical Officer – EFL E-mail: atoyirova@ueep.rti.org

Oʻzbekiston Barkamollik uchun Ta’lim Dasturi

https://t.me/UzbekistanEducationforExcellence

Abstract Textbooks play an important role in instruction. They support and enrich education by helping to standardize teaching practices and achieve learning outcomes. Additionally, textbooks can help motivate learners and widen their understandings of themselves and others. Understanding a textbook’s strengths and weaknesses is critical to build on what textbooks offer and compensate for any lack of quality. However, how does one evaluate a textbook and compare the merits of one textbook over another? A quality appraisal process using an effective instrument can help in selecting the most appropriate textbook or understand where a textbook already in use may need to be improved. This paper provides an overview of the design of an appraisal instrument used by the Uzbekistan Education for Excellence Program in collaboration with the Uzbekistan Ministry of Public Education to analyze English textbooks. * Audience members will receive the appraisal instrument and resources to assess their own learning materials.

Focus of the Presentation Overview of the design of an appraisal instrument used by the Uzbekistan Education for Excellence Program (UEEP) in collaboration with the Uzbekistan Ministry of Public Education to analyze English language textbooks.

• Discuss the design and purpose of appraisal instruments to assess the relevance, quality, and appropriateness of international series of instructional materials targeting diverse audiences and educational settings.

• Point to how one can utilize similar instruments to appraise textbooks for English language instruction in diverse classroom settings.

Share lessons learned during the process of selecting textbooks for EFL in the context of Uzbek primary and secondary schools supported by the Ministry of Public Education.

• • • •

Instrument Design Data Collection Data Analysis Data-sharing for decision-making

Questions & Discussion

Context of the Appraisal of English Language Textbooks in Uzbekistan • The Appraisal of English language textbooks is part of a series of activities supported by the UEEP and MPE in Uzbekistan under the auspices of the U.S. Agency for International Development.

• One of the goals of the UEEP project is to improve English language instruction, and support EFL teaching and learning across all regions of the country by providing quality EFL materials to Uzbek teachers and students.

• The process is extremely important for EFL education in Uzbekistan as it will serve to inform both the procurement and customization process for student textbooks and teacher’s guides for all grades from one to eleven.

The In-Depth Appraisal of EFL International Series for English as a Foreign Language was carried out virtually by teams in the USA and in Uzbekistan from January 28- February 17, 2021.

5/20/2021

Uzbek English teachers participate in co-creation workshop in Tashkent to set goals for the project in Feb.2020. Photo credit: Flavia Ramos-Mattoussi 4

Why appraising textbooks for English as a foreign language?

A quality appraisal process using an effective instrument can help in selecting the most appropriate textbook or understand where a textbook already in use may need to be improved.

Ministry of Public Education of the Republic of Uzbekistan

Why Appraising Textbooks?

LANDSC APE OF ENGLISH AS A FOREIGN LANGUAGE

Difficult to establish a one-best-way approach

Must consider: Needs of the Population: Why learning English?

International Standards and Language Proficiency Tests

Teacher’s guides, best practices

Learning Environment & Classroom Resources 6

WHY USING STUDENT TEXTBOOKS ?

• Standardize instruction • The use of a textbook in a program can ensure that the students in different classes receive similar content and therefore can be tested in the same way or progressed to the next grade.

• Maintain quality control • If a well-developed textbook is used, students are exposed to materials that have been tried and tested, that are based on sound learning principles, and that are paced appropriately.

7

• Are a key component of most language programs • Provide a structure for teachers and students to follow FOREIGN LANGUAGE COURSE TEXTBOOKS

• Provide attractive, motivational, colorful and diverse content • Serve as a great timesaver for teachers

• Might include print, non-print materials, and electronic communication resources Cunningsworth (1995)

Note: for our purposes, we reviewed print-based textbooks only.

8

HIGH QUALITY TEXTBOOKS

Promote learner engagement Help address and assess different language competencies Focus on both language form and language use Facilitate achievement of learning outcomes Harsono, 2007

9

BENEFITS OF TEACHER’S GUIDES

• Teacher’s Guides provide the basic principles of the student coursebook and the recommended procedures for using the book.

• Provide a source of ideas and methodology for teachers

• Efficiency • Might save teacher's time, enabling them to devote time to teaching rather than on producing their own materials.

• Provide effective language models and input.

• Support teachers with lower subject-matter knowledge (or lower pedagogical content knowledge).

10

What is the best book for teachers and their learners?

The perfect textbook does not exist, however “the best book available for teachers and their learners does exist” (Grant, 1987, p. 8).

Let’s find it!

Hutchinson (1987) defines evaluation as a “matter of judging the fitness of something for a particular purpose” (p.41).

5/20/2021

11

WHAT TO CONSIDER: ELEMENTS OF THE REVIEW & APPRAISAL PROCESS • Basic Concepts of Teaching and Learning English as a Foreign Language • Best Practices in English Language Instruction • Considerations for the Review and Adoption of EFL Textbooks • Content, Design, Layout, Cultural and Linguist Appropriateness • Gender Equity and Social Inclusion • Useful Feedback and Evidence-Based Decision-Making

12

Considerations for appraising and customizing textbooks for a particular setting • Consider infrastructure of school settings and potential limitations and access to resources (i.e., internet connection, computers, audio-visual equipment, etc.).

– What would be the best choice for the specific context?

– Could the Student Textbook and Teacher’s Guide stand alone in print version?

– Which publisher offers more flexibility and less dependency on supplemental materials and multimedia?

• Consider other categories of customization (e.g., design, multicultural representation, layout, photo-art, cultural appropriateness, etc.).

– What resources are more culturally sensitive to multicultural audiences?

– What books are more suitable for the target audience—both in its content, visual layout, age group, and choice of topics?

5/20/2021

13

The Process: From Designing of Appraisal Instrument to Selection of International Series of English Textbooks & Teacher’s Guides

Best Practices research and selection of criteria for evaluating EFL textbooks

Designing the appraisal form as a survey instrument in Excel

Creating a Moodle site to share materials for review, appraisal form, and guidelines for appraisers

Identifying and recruiting a team of appraisers in collaboration with Ministry of Public Education in Uzbekistan

Engaging UEEP/RTI/FSU and MPE appraisers in discussions via Zoom

Completing appraisal forms in Excel, arriving at consensus scores, collating responses

Analyzing, validating, and reporting data gathered from appraisal process

Presenting data and comparative analysis of international series of English textbooks and teacher’s guides

Stakeholders use data for decision making and selection of international series of English books

Sharing Resources https://t.me/UzbekistanEducationforExcellence

All web resources are free to use and share.

Use Diigo to create your own online libraries.

5/20/2021

15

Ministry of Public Education of the Republic of Uzbekistan

Uzbekistan Education for Excellence Program

In-Depth Appraisal (IDA) of International Series of English Student Textbooks and Teacher Guides January 28-February 17, 2021 5/20/2021

16

IDA-EFL Team Members on Zoom February 8, 2021

5/20/2021

17

Overview of the appraisal form for the set of SB and TG Student Textbook (total 13 categories and 39 questions) • • • • • • • • • • • • •

Country & Cultural Appropriateness (4) Structure & Organization (2) Layout & Accessibility (2) Content (7) Pacing of Content (1) Learning (6) Language & Readability (4) Communicative Language Teaching (CLT) (1) Comprehensiveness of Student Resources (1) Universal Design for Learning (3) Gender Equality & Social Inclusion (GESI) (3) Multiculturalism & Interculturalism (2) Assignments & Assessments (3)

5/20/2021

Teacher’s Guide (total 4 categories and 39 questions)

• Overall Content (7) • Design (8) • Instructional Support (18) • Assessment (6) 18

APPRAISAL FORM: ITEMS & SCALE ITEM

Category

SCORE (0)

Absent/ Unusable

Example: Universal Design for No evidence Learning (UDL): an educational framework recognizing that all children learn differently and benefit from differentiated learning techniques in the classroom.

SCORE (1)

Limited

Limited evidence, both in consistency and frequency

SCORE (2)

Satisfactory

Sufficient evidence, both in consistency and frequency

SCORE (3)

Strong

Consistent and frequent evidence

19

1st Section of Appraisal Instrument: Student Textbook

5/20/2021

20

2nd Section of Appraisal Instrument: Teacher’s Guide

5/20/2021

21

3rd Section of Appraisal Instrument: Team Consensus & Customization

5/20/2021

22

Presentation of findings: Data analysis The standardized appraisal form served as a systematic and comprehensive approach to review student textbooks and teacher’s guides.

We reviewed each student textbook against 13 different categories and each teacher’s guide against 4 categories.

We analyzed the data collected during the appraisal process after receiving all consensus forms from appraisers.

We presented the detailed data and comparative analysis of publishers to stakeholders (USAID & MPE) The final selection of textbooks was based on evidence of best practices and appropriateness of the books to the local context.

5/20/2021

23

Example of Overall Averages: Publisher 1 vs. Publisher 2

Publisher 2 sets [student textbook and teacher guide; see in blue] scored slightly higher than Publisher 1 (179.60 points vs. 171.80; 7.8 points or 4.4%).

Example: Student textbook Comparison (by Grade Level) • We also compared textbooks for primary, middle, and high grades.

• Publisher 2 is in orange and Publisher 1 is in blue.

• As you can see here, the differences in points are not significant.

5/20/2021

25

An Uzbek EFL teacher/appraiser’s view of the process of reviewing international series of English language books…

• Transparency • Professional Team – local practicing teachers, MPE representative, International experts and technical team.

• Thoroughly planned appraisal form • I learned how to organize the process, share the content and get everyone equally involved.

• Important points to consider by local teachers • Essential feedback by 1-deputy minister, REC director.

5/20/2021

26

Discussion Questions: What criteria for selection of textbooks matter most?

• Why would you appraise textbooks for English language instruction?

• How to select the criteria or framework to use for appraising the textbooks you currently use?

• What matters most for you and your students?

• If you cannot change the textbooks you have, what can you do to make teaching and learning better, faster, and simpler?

• How could you apply the In-Depth Appraisal Instrument that we developed for appraising and customizing international series of EFL textbooks?

5/20/2021

27

Thank you!

5/20/2021

28

Ministry of Public Education of the Republic of Uzbekistan

Valerie Haugen, Chief of Party Oʻzbekiston Barkamollik Uchun Ta’lim Dasturi Uzbekistan Education For Excellence Program Email Address: vhaugen@rti.org Skype Name: Valerie.Haugen Telephone Number: +1-202-288-8328 Telegram & WhatsApp Name/Number: Valerie Haugen/+1-202-288-8328 Oybek Kurbanov, Deputy Chief of Party Oʻzbekiston Barkamollik Uchun Ta’lim Dasturi Uzbekistan Education For Excellence Program Email Address: okurbanov@ueep.rti.org Skype Name: Oybek Kurbanov Telephone Number: +998-972-247-904 Telegram Number: +998-972-247-904

5/20/2021

29