The Current Column

Evidence for Learning – How to use impact assessments more effectively

Rudolph, Alexandra / Armin von Schiller / Christoph Strupat
The Current Column (2017)

Bonn: German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE) (The Current Column of 2 October 2017)

Bonn, 2 October 2017. The relevance of impact assessments is not a new topic in development cooperation. There is an increasingly strong consensus in development policy that it is crucial to assess interventions by using rigorous methods. In Germany, the value and relevance of independent rigorous impact assessments has grown in recent years, although impact assessments are far less institutionalised than in other major donor countries.

There are many good reasons to support more impact assessments. These include, for instance, holding implementing agencies accountable and legitimising development cooperation by showing positive impacts where these are demonstrable. In the policy discussion, however, the goal to learn dominates the debate. Impact assessments, in theory, create knowledge that leads practitioners to change their strategies to achieve better results.

Unfortunately, donors and implementing agencies severely underutilise impact assessments, particularly in the German context. Several factors hinder the proper implementation of impact assessments and the proper use of their results. For instance, first, the external validity of results is restricted. As a result, any insights about what worked (or did not work) cannot be easily transferred to other contexts. Second, the perception that the assessments’ main purpose is not to learn but to appraise the performance of programmes or even individuals increases scepticism among practitioners. It is natural that under these circumstances, the problem of bias towards positive reporting becomes more prominent and in some cases there is even open resistance towards rigorous impact assessments.

Many of the challenges that the development community faces in using evidence from impact assessments could be mitigated by building more constructive relationships between researchers and practitioners. Unfortunately, impact assessments conducted by external researchers are increasingly seen as parallel and peripheral to the day-to-day work of practitioners. The cause of this trend is not only a problem of incentives to support rigorous impact assessment by practitioners, but also a problem of communication between practitioners and researchers. On the one hand, researchers need to make results accessible and useful for practitioners. It is especially crucial to explain the intuition behind methodological approaches in order to get the practitioners on board. On the other hand, practitioners need to make clear what their learning interests are, and to discuss their constraints and fears openly. Project planning must also ensure that practitioners have enough time available to be meaningfully involved in the impact assessment process.

We consider that setting up impact assessments in the form of accompanying research can help to achieve a more constructive relationship between practitioners and researchers. At the German Development Institute (DIE) we are collecting positive experiences using this approach in countries like <link en research projects details wirkungsinitiative-governance>Benin, <link en research projects details social-protection-food-security-and-rural-development>Malawi and <link en research projects details wirkungsinitiative-governance>Mozambique. The cornerstone of this model is regular advisory exchange on the design and implementation of impact assessments in close cooperation between the researchers engaged as independent evaluators and the agencies implementing the development interventions under evaluation. In contrast to short-term consultancy-based assessments, this model aims to develop a trustworthy, long-term working relationship between all actors. This facilitates a joint learning process while ensuring an independent impact assessment. In the best-case scenario, the programmes that are implementing the interventions do not have to cover the financial costs of the impact assessment because external donors provide this.

This model can increase the quality and usefulness of impact assessments as well as improving discussions about their results. In addition, there are benefits for researchers and practitioners beyond the specific impact assessment. Researchers learn more about the challenges of implementing interventions in the field as well as how to present their results beyond the research community, while practitioners learn about the advantages and limitations of rigorous impact assessments and increase their capacity to understand and use evidence generated outside their project.

This is not to say that all impact assessments should be set up in this fashion. There are multiple ways to set up independent impact assessments and all of them have their benefits. Our experience indicates that impact assessments based on accompanying research are particularly good at improving the quality and use of the assessment’s results as well as brings benefits that go beyond the specific assessment. As the policy community pushes to increases the use impact assessments, we should think more carefully about how to assure a better use of generated evidence. Impact assessment based on accompanying independent research is a model that has a relevant contribution to make in this context.

Alexandra Rudolph is working at the Federal Ministry for Economic Cooperation and Development (BMZ), Christoph Strupat und Armin von Schiller are researchers at the German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE).