CLCnumeracy,literacyDTLliteracy,numeracyCLClifeskillChangechampion DTLlifeskillParentworkshopRadioprogrammes

3. Which interventions have been most relevant so far?

Based on the evaluation findings, the Community Learning Classes are so far showing strong relevance in providing literacy and numeracy and life skills coaching for the most marginalised girls. They meet the critical learning needs of these girls and also successfully address the barrier of parents’ fear of girls travelling away from the community. The Community Learning Classes comprise 33% of the budget, the highest cost, which was expected as they were the project’s core aim. The main cost driver was the paid educators rather than infrastructure. The educators were also heavily drawn on by other interventions that need local enablers. The high relevance suggests that this has been a good use of project resources within the implementation context so far.

The endline evaluation found that radio programmes were relevant in response to COVID-19. Radio programmes successfully addressed parental attitudes towards girls’ education and only accounted for 2% of the project expenditure. The evaluation found that half of the girls were aware of the programme and its messages on pregnancy, protection from violence and health.

Life skills classes delivered through the Community Learning Classes and distance teaching activities were very relevant, addressing the social norms barriers relating to girls’ empowerment and family planning and contraception. They comprised 10% of the expenditure to date (6% for Community Learning Classes delivery and 4% through the delivery of distance teaching and learning). But their lack of effectiveness (as discussed below) suggests that some adjustment to the design of this intervention could be necessary to make it more relevant.

In the evaluation, the parental workshops, which comprised 6% of the expenditure so far, displayed mixed relevance. They do target the barriers to schooling but not sufficiently enough, as observed by low enrolment rates in the Community Learning Classes. The project has not yet sufficient time to address the deep rooted barriers of absenteeism during the festive and harvesting seasons and early marriages arranged by parents to avoid higher dowry prices. The project will continue with the parental workshops to address these barriers to schooling, with some adaptations in delivery (introducing oneto-one sessions, for example) to better address these barriers and sharpen relevance.

Other parental barriers regarding education versus vocational training remain. According to qualitative findings, parents thought that providing girls with vocational cash grants for income-generating opportunities was more relevant than providing opportunities for formal learning. This raises the question of whether a project focus on formal education is appropriate for older girls within these communities and whether better use of project resources would be to focus on a more flexible blend of skills development opportunities for older girls to transition access employment opportunities successfully. Also, encouraging girls to move into nontraditional vocational pathways may help change traditional gender stereotypes. Younger girls were more likely to transition to formal schooling, which is relevant for the project in the future.

4. Which interventions have been most costeffective so far?

Literacy and numeracy through Community Learning Classes

The evaluation found significant positive impacts on literacy and numeracy, driven by the fact that the classes were run in the community and parents did not have to worry about the daughters travelling long (potentially dangerous) distances. In addition, the evaluation found that the treatment group demonstrated higher learning gains than the control group and that an extra 25% of girls met the minimum reading proficiency level.

However, some girls were still unable to identify words, read, and comprehend the passages of text. In terms of numeracy, some girls could not add, subtract, multiplicate and divide. The evaluation suggested that the project should adapt this intervention by further improving the curriculum and holding separate teaching sessions for groups with different abilities. The project could also consider running longer classes or providing group-based support to improve reading skills and show the practical applications of the learning to market trading and other life skills. This would sharpen relevance and effectiveness. The project could also benefit from continuing the hybrid delivery of learning through Community Learning Classes and distance teaching and learning, which has worked well so far.

The cost per girl for the literacy and numeracy classes for the first year was £106, comparable to the Nepal formal school annual figure (£155), suggesting good cost-effectiveness. However, the evaluation found that re-enrolment to formal education was not effective, especially for older girls. As explained earlier, qualitative evidence from the evaluation suggested that this was due to girls and parents being less interested in formal education and more interested in vocational training due to its focus on income generation opportunities.

Literacy and numeracy through distance teaching and learning

The evaluation reported that almost all girls had attended literacy and numeracy classes through distance teaching and learning activities. Girls said they were satisfied with the distance teaching and learning support provided by the project during the lockdown and classes improved their learning. Girls also reported that they got the necessary support from parents during home-based learning. The cost per girl was £20 for literacy and numeracy. This suggests good cost-effectiveness in comparison with the unit costs of the Community Learning Classes. But it must be noted that the distance teaching and learning component benefited from the sunk costs of investment coming from the creation of and enrolment in Community Learning Classes. In the future, even without lockdowns, the project could benefit from continuing the hybrid model that combines Community Learning Classes and distance teaching and learning activities to maximise cost-effectiveness and relevance.

Life skills classes through Community Learning Classes and distance teaching and learning

The cost per girl for life skills classes was £20 for Community Learning Classes and £18 for distance teaching and learning. According to the evaluation, the adolescent sexual and reproductive health classes delivered through the Community Learning Classes led to shifts in attitudes and behaviour for matters related to family planning and contraception. However, the coaching activities aimed at increasing girls’ self-esteem did not change behaviour, suggesting that girls still had no voice within the household on school enrolment or starting their own business and that elders have the final say on those matters. It takes time to address deep-seated social norms, so after just one year of project activities, it is not surprising that these barriers still exist.

Longitudinal research that tracks young beneficiaries is needed to understand the longer-term impact of project activities on attitudes and practice towards education, early marriage and empowerment.

Change Champions

The project appointed 50 community leaders as Change Champions to help break down deep-seated social norms. Change Champions conducted household visits aimed to raise awareness of the importance of educating girls amongst parents. They also engaged with local government officials advocating for girls’ education. The resistance to change in community attitudes and behaviours suggests that this intervention has not yet had a significant impact. This activity comprised 5% of the budget, at the cost of £526 per Change Champion. This relatively high cost reflects the high fixed project inception costs, which will be spread over the following years. Thus the unit cost is expected to decline in future years.

In the future, the project could consider blending the life skills components and the Change Champions. More meetings with the Change champions could also improve effectiveness by reinforcing the messages and reaching a larger number of parents to effect greater change.

Parental workshops

Whilst the quantitative evidence from the evaluation found emerging changes in parental attitudes towards girls’ education, qualitative evidence suggested that shifts in attitudes have not yet changed actions or behaviours. Parental workshops have not yet shown strong effectiveness. As above, it may be too early to expect behaviour changes in the first year of implementation. The project could benefit from adapting the content or delivery of this intervention, for example, experimenting with alternative behaviour change models.

Radio programmes

Radio programmes were effective in changing parents’ attitudes towards girls’ education during the COVID-19 pandemic. There is some tentative evidence from the evaluation to suggest that they were driving the decrease in time spent on household chores. Beyond that, it is unclear whether they have shifted attitudes. Again, it may be too early to tell. Their cost was low at only £3 per beneficiary (based on 3,259 girls and their families), which was very economical. Nevertheless, it is recommended to continue such programmes not only to respond to the current pandemic but as another way to address barriers presented by social norms. Moreover, assessing change in knowledge, attitudes and practices might be more relevant in judging the results generated by the combined (complementary and mutually reinforcing) radio programmes and parental workshops.

5. Which interventions have been showing sustainability so far?

There has been an encouraging commitment by the local government towards sustaining the project activities. Some municipalities have provided written commitments to continue the Community Learning Classes in the future. Since the project is working across various municipalities of the same district, it should continue with its systemlevel activities within the next phase. The project is also involved in the Sisters’ for Sisters education network, a girl-led initiative supporting girls’ education across the GEC projects in Nepal.

Partnerships with the government could be strengthened for more cost-effective components, such as radio programmes. The project could also work with the government for hybrid approaches of distance teaching and learning and Community Learning Classes. This may lead to employment opportunities for the local facilitators trained by the project by the Sisters’ for Sisters Education network. Political will and available budget do, however, remain a challenge and a potential barrier.

At the household level, challenges remain for sustaining learning as no parents are happy to send their daughters away for higher studies. The project could address this challenge by ensuring girls can learn within their community using technology or distance learning approaches to continue and sustain learning. Alternatively, the project could focus more extensively on working with parents to assuage their concerns, encouraging younger siblings to start school, offering vocational activities for Aarambha graduates and helping to reduce the prevalence of child marriage.

6. Conclusions

Findings from the VfM review suggest that the project has shown good VfM so far, providing a strong foundation. To illustrate the VfM of the specific interventions achieved so far, Figure 2 plots the intervention costs against the benefits delivered, as proxied by effectiveness, sustainability and relevance. The top left quadrant, where a high value of benefits is delivered through a low-cost investment, shows the highest VfM. Radio programmes have the highest VfM so far, followed by the blended approach of distance teaching and learning and Community Learning Classes. However, the project overall shows good potential and there is scope for interventions to veer to the top left as the project progresses.

The benefits are further broken down and rated against each of the VfM review criteria.

Annex 1: GEC VfM review methodology

The aim of the GEC VfM framework is to offer a quick pragmatic methodology to review the VfM of a GEC project by using existing evaluation findings. The framework uses the OECD DAC criteria (relevance, efficiency, effectiveness and sustainability) and evaluation findings, and reframes them through a VfM “lens” drawing out the key features of the findings that point to strong efficient value generation for the right people against optimal costs and resource allocation.

Effectiveness

The GEC approach to VfM analysis relies on the extraction of effectiveness data from the evaluation reports (midline and endline). This should include all the different types of outcomes assessed (learning, transition and sustainability) and the intermediate outcomes, such as wellbeing and life skills, self-esteem, and social norms and behaviour changes. It should also include data on how effective interventions have been for different targeted subgroups. All types of data used to demonstrate effectiveness are relevant for VfM purposes (quantitative or qualitative data, including the beneficiaries’ voices on what they found most valuable).

Effectiveness can either be assessed for a GEC project as a whole, or for separate components. Some projects’ midlines or endlines may be able to disentangle the impact and causality of certain interventions on outcomes over and above others. This likely will only be feasible for evaluations with a comparison group.

Cost-effectiveness

With activity-based budgeting, specific interventions can be assessed on cost-effectiveness. Costs can be presented in cost per girl format, with narrative attached to it, explaining what the overall cost per girl achieved in terms of outcomes observed. The number of girls reached by interventions can differ, thus giving rise to very wide-ranging figures. These variances should be discussed within the VfM analysis. Benchmarking to similar projects within the same context would be useful.

If there are strong, statistically significant findings, with a control group of girls displaying the counterfactual ‘without project’ learning achieved in a year of schooling, the analysis can be taken further to estimate the Cost-Effectiveness Ratio (CER): additional years of schooling per $100 spent.

Relevance

The GEC approach to VfM and relevance is to use evaluation report findings around relevance to understand whether resources were allocated to meet the needs and objectives identified upfront, and whether optimal resources were allocated across activities. If such needs changed over time, the VfM review should consider whether resources were reallocated to reflect this to maintain relevance of the programme.

An equity angle to relevance would determine whether or not the right beneficiaries were targeted by the project according to needs, and if enough or optimal resources were allocated to various targeted groups. The analysis should consider whether, in retrospect, the budget would have been carved differently across activities to reflect relevance better, and whether there was a correct allocation of funds for technical functions, monitoring and evaluation, and management etc.

Sustainability

Sustainability within the GEC is measured by:

1. A long-term continuation of outcomes for the direct beneficiaries themselves (including targeted schools and communities) and

2. Replication and/or scale-up or adoption of project activities without the need for FCDO funding.

A project may have sound input costs (Economy), have a demonstrated ability to translate its activities into quality outputs (Efficiency), and achieve its targets with regard to learning and attendance (Effectiveness), but may not have a strong sustainability case. Sustainability is not always covered in the conventional measures of VfM. But it is another factor to justify expenditure. The evidence of evaluation findings on contributions to sustainability should be integrated into a VfM narrative. Evidence of replication or scale up beyond project funding would point to very strong VfM. Sustainability intent may have been present from the start in the form of specific design features or plans. But over time, as contexts have changed, contributions to sustainability may not have materialised. This may require projects to undertake additional activities targeting sustainability that increase their costs but do not necessarily improve their efficiency or effectiveness in the short term. These should be considered in a VfM assessment. Another angle to considering costs and sustainability is defining the minimum spend for activities required to achieve sustainable outcomes. For example, determining the cost of a minimum amount of project exposure/duration or intensity necessary to achieve sustained outcomes based on findings.

Efficiency

Taking a narrative approach, assessing efficiency involves understanding how smoothly processes and interventions have been delivered (speed, quality, cost). There are four aspects to efficiency:

1. Whether the project as a whole was delivered on time and on budget

2. Assessment of the speed, quality and cost of the operating models for each intervention and the project as a whole

3. Assessing the efficiency of processes and management of the project as a whole

4. Assessing the efficiency of targeting girls (inclusion or exclusion errors).

Limitations of the GEC VfM approach

The VfM analysis is dependent on the evaluation findings, so its efficacy depends on the efficacy of the evaluation findings. It is also dependent on the ability of projects to producing expenditure data in relevant formats (activity-based budgeting). VfM assessments such as these are not often appropriate for making comparisons with other projects, due to differing contexts, cost structures and activities.