Table 11: Experts and Students’ Evaluation of Prototypical Products Data Source Expert Subject Matter Expert
Language Expert Delivery Expert
Pedagogical Expert
Instructional Design Expert One-to-One Evaluation Select learners representative of population. Include: • Learners of high, medium, and low ability. • Males and females. • Young and inexperienced, and mature and experienced learners. Small Group Evaluation Select 10 to 20 learners representative of target population Material is handed to test subjects. Evaluation intervenes only when test subjects need assistance.
Field Test Select one or more groups of learners (30 each) representative of the target population from urban and rural areas
Questions Asked
Methods of Collecting Data
Types of Data Collected
Data utilization
Is content adequate, accurate, up-to-date, and relevant? Is reading level appropriate? Is delivery medium appropriate and costeffective? Does it meet quality standard? Are objectives, content appropriate? Is the material appropriate for the target population? Were instructions designed according to systematic process and based o sound theories of learning?
Check list Panel discussions Interviews
Feedback information in form of: • Narrative reports • Completed checklist • Records of panel discussions Recommendations • Interview transcripts
Feasibility analysis for implementing recommendations. Modify material Discontinue if revision is too involved and/or costly
Is the material learnable? What are the most obvious errors, problems, and weaknesses within the material? What are the learners’ reactions to the material? Are pre-, embedded and post-tests appropriate?
One-to-one learner/evaluator interaction. Think-aloud procedure. Learners’ verbal feedback, body language. Observation, interviews. Unobtrusive measures. Reaction questionnaire. Pre-, embedded, and post-tests. Debriefing interview.
Think aloud protocols Comprehension Appeal Errors Problems Weaknesses Pre-, embedded and post-tests data
Analyse learners’ feedback and make inferences for changes During each try-out implement minor revision on the spot and present modified version to learners for confirmation. After each try-out revise material on the basis of learners’ feedback.
How effective are changes made during one-to-one evaluation? Are there additional problems and errors in the material? Can learners achieve objectives? Were exercises, tests, and feedback appropriate and adequate?
Pre-, embedded and post-tests Observation Unobtrusive measures Attitude questionnaire Debriefing interview
Pre-, embedded and post-tests scores Observation records Interview data Learners’ reactions Non-verbal data
Analyse learners’ feedback data and make inferences for revisions Modify material on the basis of inferences made from learners’ feedback data
Can learners achieve objectives when using material alone? Are previous revisions effective? Will material be accepted by learners, teachers, and administrators?
Evaluation of material in natural setting and context Pre-, embedded and post-tests Attitude questionnaire
Pre-, embedded and post-tests scores Reactions of learners, teachers, and administrators to material
Analysed feedback data and implement any additional revisions necessary to address new problems
144