Feinstein: Use of Evaluations and the Evaluation of their Use influence both on supply and demand, and the same is true with capacities. Though in some cases it might be useful to discuss use issues in terms of supply and demand, it might be more fruitful (and rigorous) to focus on incentives and capacities to produce, disseminate and use relevant evaluations.
Evaluation of Use There are some typical pitfalls in the evaluation of the use of evaluations. One of them is due to the existence of lags, to a ‘gestation period’ for the occurrence of use. It might seem that there is no evidence of use and therefore no use. But this may just mean that the process leading from the production of the evaluation to its use takes time, and that the evaluation of evaluation use might have been premature. There are two risks: the first is of waiting sine die, always refraining to pass judgement because it might be that the evaluation will still be used (apocalyptic fallacy); the other risk is ‘killing’ an evaluation, arguing that it has not been used and that therefore it is useless, whereas it might be that it will be used in the future (premature killing). Another source of pitfalls is the attribution problem: one can find things that have been done after the evaluation was completed in a way consistent with the evaluation’s recommendations. Is this evidence of use? It seems so, but it might be that there were other reasons why things were done in such a way and that this is merely a case of apparent use. The fact that there is consistency between the evaluation findings and recommendations and what was done after the evaluation is not necessarily an indication of use (post hoc fallacy). However, it is also possible to completely neglect the role a specific evaluation had in the decision making process and in achieving results. A particular evaluation could possibly have played a contributing role, perhaps helping the decision makers reach the ‘tipping point’ (Gladwell, 2000) through the cumulative effect of evaluations. Finally, in evaluating the use of evaluation it is worthwhile to refer to the factors mentioned before, such as relevance and dissemination and their determinants (timeliness, credibility, quality of presentations and means of dissemination, as well as incentives and capacities); low levels of those factors or of incentives and/or capacities can act as barriers to use. Furthermore, when evaluating use it is important to consider changes in knowledge, attitudes and behavior, bearing in mind lags and the attribution problem.
A ‘Possibilist’ Approach to Evaluation Use It is generally recognized that evaluation activities generate knowledge that is significantly under-used. The distinction between actual and potential use results in greater focus on possible uses of evaluation that are more intensive and beneficial. One of these possible uses would be to identify what worked and what did not work in specific contexts (applying a sort of ‘realist’ approach to evaluations) in order to identify opportunities for effective interventions and to avoid those that are ineffective. 437 Downloaded from http://evi.sagepub.com by Osvaldo Feinstein on February 15, 2007 © 2002 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
Use of Evaluations and the Evaluation of their Use