Monday 29 March 2021

Finally! Thoughtful guidance on applying the DAC criteria

Long-awaited new guidance on applying the evaluation criteria defined by the Development Assistance Committee of the Organisation for Economic Cooperation and DevelopmentD) (OECD-DAC) is finally available in this publication! Long-awaited, because evaluators and development practitioners have grown desperate with assignments that are expected to gauge every single project against every single OECD-DAC criterion, regardless of the project's nature, and of the moment & resources of the evaluation. This new, gently worded document is a weapon evaluators can use to defend their quest for focus and depth in evaluation.

Those who commission evaluations, please go straight to page 24, which states very clearly: "The criteria are not intended to be applied in a standard, fixed way for every intervention or used in a tickbox fashion. Indeed the criteria should be carefully interpreted or understood in relation to the intervention being evaluated. This encourages flexibility and adaptation of the criteria to each individual evaluation. It should be clarified which specific concepts in the criteria will be drawn upon in the evaluation and why."

On page 28, you will find a whole section titles Choosing which criteria to use which makes it clear that evaluations should focus on the OEC-DAC criteria that make sense in the view of the needs and possibilities of the specific project, and for the evaluation process. It provides a wonderful one-question heuristic: "If we could ask only one question about this intervention, what would it be?" And it reminds readers that some questions are better answered by using other means, such as research projects or a facilitated learning process. The availability of data and resources - including time - for the evaluation helps determine which evaluation criteria to apply, and which not. Page 32 reminds us of the necessity to use a gender lens, with a handy checklist-like table on page 33 (better late than never).

About half of the publication is dedicated to defining the six evaluation criteria - relevance, coherence, effectiveness, efficiency, impact, and sustainability - with plenty of examples. This is also extremely helpful. Each chapter comes with a table that summarises common challenges related to each criteri on - and what evaluators and evaluation managers can do to overcome them. It also shows very clearly that lack of preparation on the evaluation management side makes it very hard for evaluators to do a decent job - see for example table 4.3 (p.55) on assessing effectiveness. 

The document is a bit ambiguous on some questions: The chapter on efficiency still defines efficiency as the conversion of inputs (...) into outputs (...) in the most cost-effective way possible, as compared to feasible alternatives in the context" (p.58), which makes it extremely hard to assess the efficiency of, say, a project that supports litigation in international courts - interventions that may take decades to yield the desired result. However, the guidance document states that resources should be understood in the broadest sense and include full economic costs. On that basis, one can indeed argue, as Jasmin Rocha and I have on Zenda Ofir's blog, that non-monetary costs, hidden costs and the cost of inaction must be taken into account. Yet, table 4.4 on efficiency-related challenges remains vague (p.61). Has anyone read the reference quoted in the table (Palenberg 2011)? I did and found it very cautious in its conclusion. My impression is that in many cases, evaluators of development interventions are not in a position to assess efficiency in any meaningful manner.

On the whole, I would describe the new OECD-DAC publication as a big step forward. I warmly recommend it to anyone who designs, manages or commissions evaluations.


Für Deutschsprachige: Online Moderieren - Lebendig und Produktiv

Mein online-Workshop zu online-Workshops ist am 26.Mai online! Mehr Informationen und Anmeldungsmöglichkeiten gibt es beim PME-Campus

Apologies to those who don't speak German - my first workshop on online facilitation will be in German. But if it works out nicely, I might offer sequels in English and in French! A bon entendre, Michaela

Friday 12 March 2021

Join my workshop on online facilitation in PME

After a year of lockdown-induced life in cyberspace, web-based workshops have become a routine in planning, monitoring and evaluation (PME). Workshops are about exchange, about developing something together. But often, I have witnessed online workshops that were so virtual you hardly noticed the participants. Seemingly endless pages of screen-shared text being read out, word by word, in a soothing voice. No breaks. Confusion about technical refinements, links posted to inaccessible clouds. The loneliness of the person who finds herself alone in the main channel, without any pre-assigned breakout group... 

But there are also online workshops that actually work, that engage us in an exhilarating process, and that produce results. They can be more efficient than real-life workshops. And they surely save enormous amounts of CO2 and travel costs. Even though many miss the informal encounters at the coffee machine, over lunch, in the bathroom line (we're in 2021 and people identified as male still enjoy better acess to toilets than the rest of us) -  I suspect that online workshops are here to stay. 

It takes deliberate planning, adaptive pacing and plenty of participation to make an online workshop work. After more than a year of developing, facilitating and documenting a range of workshops on different platforms, I am distilling key insights into a short workshop for German speakers - online, of course! Not the technical stuff - the providers' video tutorials take care of that - but key principles and ways to apply them. I'd be delighted to meet you. More details in German) are available here