2nd International Conference on Education and Linguistics at Westminster International University in Tashkent, May 21, 2021 Textbook Appraisal: Principles, Process, and Lessons Learned (Room B, 14:45-15:30 UZT) Presenter: Dr. Flavia Ramos-Mattoussi, Florida State University
Uzbekistan Education for Excellence Program Partner E-mail: framos@lsi.fsu.edu Co-Presenter: Azima Toyirova, Senior Technical Officer – EFL E-mail: atoyirova@ueep.rti.org
Oʻzbekiston Barkamollik uchun Ta’lim Dasturi
https://t.me/UzbekistanEducationforExcellence
Abstract Textbooks play an important role in instruction. They support and enrich education by helping to standardize teaching practices and achieve learning outcomes. Additionally, textbooks can help motivate learners and widen their understandings of themselves and others. Understanding a textbook’s strengths and weaknesses is critical to build on what textbooks offer and compensate for any lack of quality. However, how does one evaluate a textbook and compare the merits of one textbook over another? A quality appraisal process using an effective instrument can help in selecting the most appropriate textbook or understand where a textbook already in use may need to be improved. This paper provides an overview of the design of an appraisal instrument used by the Uzbekistan Education for Excellence Program in collaboration with the Uzbekistan Ministry of Public Education to analyze English textbooks. * Audience members will receive the appraisal instrument and resources to assess their own learning materials.
Focus of the Presentation Overview of the design of an appraisal instrument used by the Uzbekistan Education for Excellence Program (UEEP) in collaboration with the Uzbekistan Ministry of Public Education to analyze English language textbooks. • Discuss the design and purpose of appraisal instruments to assess the relevance, quality, and appropriateness of international series of instructional materials targeting diverse audiences and educational settings. • Point to how one can utilize similar instruments to appraise textbooks for English language instruction in diverse classroom settings.
Share lessons learned during the process of selecting textbooks for EFL in the context of Uzbek primary and secondary schools supported by the Ministry of Public Education. • • • •
Instrument Design Data Collection Data Analysis Data-sharing for decision-making
Questions & Discussion
Context of the Appraisal of English Language Textbooks in Uzbekistan • The Appraisal of English language textbooks is part of a series of activities supported by the UEEP and MPE in Uzbekistan under the auspices of the U.S. Agency for International Development. • One of the goals of the UEEP project is to improve English language instruction, and support EFL teaching and learning across all regions of the country by providing quality EFL materials to Uzbek teachers and students. • The process is extremely important for EFL education in Uzbekistan as it will serve to inform both the procurement and customization process for student textbooks and teacher’s guides for all grades from one to eleven. The In-Depth Appraisal of EFL International Series for English as a Foreign Language was carried out virtually by teams in the USA and in Uzbekistan from January 28- February 17, 2021.
5/20/2021
Uzbek English teachers participate in co-creation workshop in Tashkent to set goals for the project in Feb.2020. Photo credit: Flavia Ramos-Mattoussi 4
Why appraising textbooks for English as a foreign language? A quality appraisal process using an effective instrument can help in selecting the most appropriate textbook or understand where a textbook already in use may need to be improved.
Ministry of Public Education of the Republic of Uzbekistan
Why Appraising Textbooks?
LANDSC APE OF ENGLISH AS A FOREIGN LANGUAGE
Difficult to establish a one-best-way approach
Must consider: Needs of the Population: Why learning English?
International Standards and Language Proficiency Tests
Teacher’s guides, best practices
Learning Environment & Classroom Resources 6
WHY USING STUDENT TEXTBOOKS ?
• Standardize instruction • The use of a textbook in a program can ensure that the students in different classes receive similar content and therefore can be tested in the same way or progressed to the next grade. • Maintain quality control • If a well-developed textbook is used, students are exposed to materials that have been tried and tested, that are based on sound learning principles, and that are paced appropriately. 7
• Are a key component of most language programs • Provide a structure for teachers and students to follow FOREIGN LANGUAGE COURSE TEXTBOOKS
• Provide attractive, motivational, colorful and diverse content • Serve as a great timesaver for teachers
• Might include print, non-print materials, and electronic communication resources Cunningsworth (1995)
Note: for our purposes, we reviewed print-based textbooks only.
8
HIGH QUALITY TEXTBOOKS
Promote learner engagement Help address and assess different language competencies Focus on both language form and language use Facilitate achievement of learning outcomes Harsono, 2007
9
BENEFITS OF TEACHER’S GUIDES
• Teacher’s Guides provide the basic principles of the student coursebook and the recommended procedures for using the book. • Provide a source of ideas and methodology for teachers
• Efficiency • Might save teacher's time, enabling them to devote time to teaching rather than on producing their own materials.
• Provide effective language models and input. • Support teachers with lower subject-matter knowledge (or lower pedagogical content knowledge).
10
What is the best book for teachers and their learners?
The perfect textbook does not exist, however “the best book available for teachers and their learners does exist” (Grant, 1987, p. 8).
Let’s find it! Hutchinson (1987) defines evaluation as a “matter of judging the fitness of something for a particular purpose” (p.41). 5/20/2021
11
WHAT TO CONSIDER: ELEMENTS OF THE REVIEW & APPRAISAL PROCESS • Basic Concepts of Teaching and Learning English as a Foreign Language • Best Practices in English Language Instruction • Considerations for the Review and Adoption of EFL Textbooks • Content, Design, Layout, Cultural and Linguist Appropriateness • Gender Equity and Social Inclusion • Useful Feedback and Evidence-Based Decision-Making
12
Considerations for appraising and customizing textbooks for a particular setting • Consider infrastructure of school settings and potential limitations and access to resources (i.e., internet connection, computers, audio-visual equipment, etc.). – What would be the best choice for the specific context? – Could the Student Textbook and Teacher’s Guide stand alone in print version? – Which publisher offers more flexibility and less dependency on supplemental materials and multimedia? • Consider other categories of customization (e.g., design, multicultural representation, layout, photo-art, cultural appropriateness, etc.). – What resources are more culturally sensitive to multicultural audiences? – What books are more suitable for the target audience—both in its content, visual layout, age group, and choice of topics? 5/20/2021
13
The Process: From Designing of Appraisal Instrument to Selection of International Series of English Textbooks & Teacher’s Guides
Best Practices research and selection of criteria for evaluating EFL textbooks
Designing the appraisal form as a survey instrument in Excel
Creating a Moodle site to share materials for review, appraisal form, and guidelines for appraisers
Identifying and recruiting a team of appraisers in collaboration with Ministry of Public Education in Uzbekistan
Engaging UEEP/RTI/FSU and MPE appraisers in discussions via Zoom
Completing appraisal forms in Excel, arriving at consensus scores, collating responses
Analyzing, validating, and reporting data gathered from appraisal process
Presenting data and comparative analysis of international series of English textbooks and teacher’s guides
Stakeholders use data for decision making and selection of international series of English books
Sharing Resources https://t.me/UzbekistanEducationforExcellence
All web resources are free to use and share. Use Diigo to create your own online libraries. 5/20/2021
15
Ministry of Public Education of the Republic of Uzbekistan
Uzbekistan Education for Excellence Program
In-Depth Appraisal (IDA) of International Series of English Student Textbooks and Teacher Guides January 28-February 17, 2021 5/20/2021
16
IDA-EFL Team Members on Zoom February 8, 2021
5/20/2021
17
Overview of the appraisal form for the set of SB and TG Student Textbook (total 13 categories and 39 questions) • • • • • • • • • • • • •
Country & Cultural Appropriateness (4) Structure & Organization (2) Layout & Accessibility (2) Content (7) Pacing of Content (1) Learning (6) Language & Readability (4) Communicative Language Teaching (CLT) (1) Comprehensiveness of Student Resources (1) Universal Design for Learning (3) Gender Equality & Social Inclusion (GESI) (3) Multiculturalism & Interculturalism (2) Assignments & Assessments (3)
5/20/2021
Teacher’s Guide (total 4 categories and 39 questions)
• Overall Content (7) • Design (8) • Instructional Support (18) • Assessment (6) 18
APPRAISAL FORM: ITEMS & SCALE ITEM
Category
SCORE (0)
Absent/ Unusable
Example: Universal Design for No evidence Learning (UDL): an educational framework recognizing that all children learn differently and benefit from differentiated learning techniques in the classroom.
SCORE (1)
Limited
Limited evidence, both in consistency and frequency
SCORE (2)
Satisfactory
Sufficient evidence, both in consistency and frequency
SCORE (3)
Strong
Consistent and frequent evidence
19
1st Section of Appraisal Instrument: Student Textbook
5/20/2021
20
2nd Section of Appraisal Instrument: Teacher’s Guide
5/20/2021
21
3rd Section of Appraisal Instrument: Team Consensus & Customization
5/20/2021
22
Presentation of findings: Data analysis The standardized appraisal form served as a systematic and comprehensive approach to review student textbooks and teacher’s guides. We reviewed each student textbook against 13 different categories and each teacher’s guide against 4 categories. We analyzed the data collected during the appraisal process after receiving all consensus forms from appraisers.
We presented the detailed data and comparative analysis of publishers to stakeholders (USAID & MPE) The final selection of textbooks was based on evidence of best practices and appropriateness of the books to the local context. 5/20/2021
23
Example of Overall Averages: Publisher 1 vs. Publisher 2
Publisher 2 sets [student textbook and teacher guide; see in blue] scored slightly higher than Publisher 1 (179.60 points vs. 171.80; 7.8 points or 4.4%).
Example: Student textbook Comparison (by Grade Level) • We also compared textbooks for primary, middle, and high grades. • Publisher 2 is in orange and Publisher 1 is in blue. • As you can see here, the differences in points are not significant.
5/20/2021
25
An Uzbek EFL teacher/appraiser’s view of the process of reviewing international series of English language books…
• Transparency • Professional Team – local practicing teachers, MPE representative, International experts and technical team. • Thoroughly planned appraisal form • I learned how to organize the process, share the content and get everyone equally involved. • Important points to consider by local teachers • Essential feedback by 1-deputy minister, REC director. 5/20/2021
26
Discussion Questions: What criteria for selection of textbooks matter most?
• Why would you appraise textbooks for English language instruction? • How to select the criteria or framework to use for appraising the textbooks you currently use?
• What matters most for you and your students? • If you cannot change the textbooks you have, what can you do to make teaching and learning better, faster, and simpler? • How could you apply the In-Depth Appraisal Instrument that we developed for appraising and customizing international series of EFL textbooks? 5/20/2021
27
Thank you!
5/20/2021
28
Ministry of Public Education of the Republic of Uzbekistan
Valerie Haugen, Chief of Party Oʻzbekiston Barkamollik Uchun Ta’lim Dasturi Uzbekistan Education For Excellence Program Email Address: vhaugen@rti.org Skype Name: Valerie.Haugen Telephone Number: +1-202-288-8328 Telegram & WhatsApp Name/Number: Valerie Haugen/+1-202-288-8328 Oybek Kurbanov, Deputy Chief of Party Oʻzbekiston Barkamollik Uchun Ta’lim Dasturi Uzbekistan Education For Excellence Program Email Address: okurbanov@ueep.rti.org Skype Name: Oybek Kurbanov Telephone Number: +998-972-247-904 Telegram Number: +998-972-247-904
5/20/2021
29