Celebrating Learning through Evaluation Overview of the Community Health Worker (CHW) Baseline Training Evaluation, Justification, Findings and Recommendations for Next-Steps “While each country’s context will be different when strengthening CHW programs, countries should strive to adhere to ten core principals, including establishing sufficient health worker training and building monitoring and evaluation capacity and structured supervision, to ensure Return on Investment (ROI).”
Evaluation Highlights • 63 CHWs and 19 CHW Peer and Clinical Supervisors participated in an evaluation to assess baseline knowledge retention at a 1 or 4 month follow-up • 54 female, 28 males participated across two counties, seven Health Care Facilities (clinics, health centers, hospitals) • Knowledge Assessment (written questionnaire) and Practical Assessment Tool (checklist) were used in the assessment • CHWs were assessed on a pre-determined set of “Key Health Education and Health Promotion” messages around TB, HIV/AIDS and Leprosy Control, Treatment, Prevention and stigma reduction
Evaluation Justification: Why is evaluating CHW trainings and learning important? The UN Secretary-General’s Special Envoy for Financing the Health Millennium Development Goals released a report titled “Strengthening Primary Health Care through Community Health Workers: Investment Case and Financing Recommendations.” The Ministry of Health of Liberia, Last Mile Health, UNICEF, PIH and other partners contributed to this report which highlighted that investing in the training of CHWs and evaluation of CHW programs is among the ten core principles of sound program implementation.
Overview of PIH/CHT CHW Training Program • In May 2015 the Community Health Worker program began in Maryland and Grand Gedeh Counties • All newly recruited CHWs and Supervisors received an intensive, six-day, 11-Module baseline training of the core competencies in HIV/ AIDS, Tuberculosis, Leprosy prevention, control, stigma reduction, and Voluntary Counseling and Testing • All supervisors received a three-day Supervisor training • Curricula was drawn from existing NACP and NTLCP HIV/AIDS, TB, Leprosy material as well as PIH’s ‘best practice’ CHW/Accompagnateur materials
• All CHWs and CHW Supervisors received a “Personal Evaluation Results Summary” highlighting top three strength areas and three areas for improvement • Learning points included: evaluation methods that allow CHWs to demonstrate their competencies rather than write their competencies seem to be a better assessment of knowledge and abilities among Grand Gedeh and Maryland County CHWs • Average Written Assessment Score: 81.7% (Maryland County) and 78.9% (Grand Gedeh County) • Average Practical Assessment Test score: 80.4% (Maryland County) and 86.3% (Grand Gedeh)
Grand Gedeh CHW Supervisors conducting the Practical Observation Checklist Assessment, October 2015
Objectives of the CHW Training Evaluation • Among the CHWs who received the Baseline, Phase 1 training, what percentage of the CHWs achieved a 65% pass rate on their written Knowledge Assessment questionnaire? • Among the CHWs who received the Baseline, Phase 1 training, what percentage of the CHWs achieved a 65% pass rate on their demonstrated Practical Checklist Assessment in the field? • What can we learn about the best format for assessing CHW knowledge and how evaluation findings can inform new subject and refresher-training subject areas and program improvement decisions moving forward?
Summary Statistics The chart below highlights the summary statistics of the CHW and Supervisor Training Evaluation. Data was analyzed across sub groups (male/female, by county, by CHW and Supervisor and across health care facility). Differences found across sub-groups were not significant. Differences between CHW performances on the baseline knowledge assessment as compared to the post-test at the 1-4 month follow-up was low. Between the baseline knowledge and post-knowledge assessment the measured knowledge assessment never fell below 2.8% percentage points for any sub-group as shown below. Number
Baseline Written Assement (KA)
Post-test Written Assesment (KA)
Practical Assesment (KAA)
Baseline Post Difference
Compostie of KA & KAA
Supervisors (Peer & Clinical)
Summary of Accomplishments by County
Evaluation Strengths • CHT Participation: County Health Team/MOH representatives introduced each training • Strong cross-team collaboration: PIH Liberia Clinical, Program/Logistics and Community Health Team, Boston and Liberia MEQ team, Global Learning & Training Team • Preliminary Program Results: Estimates for LTFU/Default patients returning to Care and Treatment by CHWs between June-October 2015 include 17 ART patients, 40 TB patients, 16 Leprosy patients • Celebrated Learning through Evaluation: This assessment helped to professionalize CHW training and helped to provide CHW Supervisors with a clear follow-up plan for refresher trainings • Established Health Promotion Messages were evaluated: This evaluation was able to assess and demonstrate the specific health education messages 82 CHWs and CHW Supervisors use during community awareness related to TB, HIV, Leprosy and stigma reduction.
Areas for Improvement Challenges
Suggestions for Improvement
1. Some questions were worded poorly and were not understood clearly; this may have contributed to lower test scores for some participants
• Allow adequate time to ‘field test’ all assessment instruments to ensure wording and context is clearly understood • C&T Specialist can coach CH Officer and CH Specialist to design and draft their own evaluation methodology and assessment tools to ensure timeliness and context-appropriate questions
• Community Health Officers (CHO) /Community Health 2. Evaluation results highlighted areas in which CHWs and Specialists (CHS) are encouraged to work together to create Supervisors could benefit from additional trainings from a calendar schedule for ‘mini refresher trainings.’ These mini the Clinical team. These areas include: recognizing urgent and non-urgent side effects among patients on TB, ART and refresher trainings can be conducted at the monthly CHW leprosy medication and recognizing signs and symptoms of meeting to reinforce lower-scoring learning areas opportunistic infections among TB and/or ART patients • CHO/CHSs are encouraged to work with PIH and CHT Clinical leaders to plan for a short series of ‘follow up refresher trainings’ to reinforce some of the learning area that had the lowest scores across CHW and Supervisor participants 3. CHWs and CHW supervisors reported some anxiety around • CHO and CHSs are encouraged to emphasize with their the evaluation from the participants. Some were concerned CHW team that this evaluation is intended to help improve they would be penalized for incorrect answers the training and programming for the CHWs in Maryland and Grand Gedeh County. All professional health workers in the Health Care System are required to participate in on-going learning. 4. PIH Liberia has spent a good amount of money on trainings; • PIH Liberia leadership is encouraged to appoint a “Training among the trainings in FY15 MEQ reports many of those Coordinator” (either a dedicated Training Coordinator or trainings lacked baseline and post-test assessments. For one person who can be the designated “point of contact”) to trainings conducted without any evaluation component the improve training coordination, establish a training calendar, quality of the training is difficult to capture, monitor and report checklist of requirements for training such as defining learning objectives, timeline, budget, pre and post-test requirements, printing and material development plan and budget. 5. Linkages between training outcomes and program outcomes are not causal in this evaluation and can only be inferred
• Strengthen M&E design capacity to begin to collect data on the patient side, to understand the influence of CHW messaging and work on patient behavior or knowledge attitudes and beliefs around the outcome of interest
6. Missing reference point data for Grand Gedeh (Baseline, Phase 1 Knowledge Assessment was administered to Grand Gedeh CHWs, but their names were not recorded.
• CH Specialists and Officers are encouraged to remind their CHWs and Supervisors to write their names on their assessment tests
Key Findings: What did we learn?
Learning, Next Steps + FY16 Plans
• On average, participants scored higher on the Practical Field Checklist (KAA) than on the Written Knowledge Assessment (KA)
• Evaluation methods that allow CHWs to demonstrate their competencies rather than write their competencies seem to be a better assessment of knowledge and abilities among GG and Maryland County CHWs/Supervisors
• When given the opportunity to demonstrate their knowledge CHWs achieve higher scores on their knowledge assessments • Results across sub groups (Male/Female, by county, by clinic or health center or hospital, or among CHWs and Supervisors) were not significantly different • Adequate review and editing of questions is essential to ensure wording is culturally-appropriate and language-appropriate • Multiple choice answers that require the participant to circle “All the above” and “Both B, D” questions are confusing and should not be used in future assessments • Highest performing knowledge areas: ability to describe disease transmission and prevention of TB, HIV, Leprosy, ability to use good communication skills with their families, ability to describe ways to be a ‘buddy’ to a patient • Lower performing knowledge areas: describing signs of opportunistic infections among TB or HIV or coinfected patients, side effects of TB and Leprosy medication and early signs of leprosy
Key Recommendations: What can we apply to sustain training program success? • Community Health Officers and Specialists are encouraged to share findings with their clinical and county health team leaders to celebrate learning and advocate for support in on-going refresher and new subject-area trainings
• Clinical and Community Health Team should strive to incorporate quarterly or half-yearly Field Checklist Evaluations into CHW monitoring and training activities to be able to better track knowledge and messaging used during community awareness and patient education • Establishing a ‘training lead’ on the clinical and community health team would aid in professionalizing and stream-lining training and evaluation activities • Evaluation findings can inform monthly refresher trainings lead by CHW clinical supervisors and clinical team • Evaluation findings can create a case for additional trainings around providing Psychosocial support, Motivational Interviewing (or another Behavior Change Communication counseling methodology) • CHW training program moving towards polyvalence (CEBS, ANC programming) can employ the checklist as an evaluation methodology for new CHW programming • PIH Liberia is currently contributing to the National Community Health Curricula and core competencies; aim to establish standard Key Health Education and Health Promotion messages for TB, Leprosy and HIV
• Community Health Officers should encourage CHW Supervisors to review the CHW Personal Evaluation Results plan to help clarify misunderstandings and ensure on-going study and knowledge refreshment • Community Health leaders are encouraged to continue to work closely with CHT partners to further establish and build consensus around “Key Health Education and Health Promotion Messages for TB, Leprosy, HIV/AIDS” • Developing inexpensive, hand-held visual aids with the agreed upon Key Health Promotion and Health Education Messaging by the National Community Health Curriculum Development working group can serve as a reference for CHWs during Community Awareness and one-on-one patient education
CHW Supervisor Training: Administering the Practical Observation Checklist, Maryland County 4