Monday, 29 March 2021

Finally! Thoughtful guidance on applying the DAC criteria

Long-awaited new guidance on applying the evaluation criteria defined by the Development Assistance Committee of the Organisation for Economic Cooperation and DevelopmentD) (OECD-DAC) is finally available in this publication! Long-awaited, because evaluators and development practitioners have grown desperate with assignments that are expected to gauge every single project against every single OECD-DAC criterion, regardless of the project's nature, and of the moment & resources of the evaluation. This new, gently worded document is a weapon evaluators can use to defend their quest for focus and depth in evaluation.

Those who commission evaluations, please go straight to page 24, which states very clearly: "The criteria are not intended to be applied in a standard, fixed way for every intervention or used in a tickbox fashion. Indeed the criteria should be carefully interpreted or understood in relation to the intervention being evaluated. This encourages flexibility and adaptation of the criteria to each individual evaluation. It should be clarified which specific concepts in the criteria will be drawn upon in the evaluation and why."

On page 28, you will find a whole section titles Choosing which criteria to use which makes it clear that evaluations should focus on the OEC-DAC criteria that make sense in the view of the needs and possibilities of the specific project, and for the evaluation process. It provides a wonderful one-question heuristic: "If we could ask only one question about this intervention, what would it be?" And it reminds readers that some questions are better answered by using other means, such as research projects or a facilitated learning process. The availability of data and resources - including time - for the evaluation helps determine which evaluation criteria to apply, and which not. Page 32 reminds us of the necessity to use a gender lens, with a handy checklist-like table on page 33 (better late than never).

About half of the publication is dedicated to defining the six evaluation criteria - relevance, coherence, effectiveness, efficiency, impact, and sustainability - with plenty of examples. This is also extremely helpful. Each chapter comes with a table that summarises common challenges related to each criteri on - and what evaluators and evaluation managers can do to overcome them. It also shows very clearly that lack of preparation on the evaluation management side makes it very hard for evaluators to do a decent job - see for example table 4.3 (p.55) on assessing effectiveness. 

The document is a bit ambiguous on some questions: The chapter on efficiency still defines efficiency as the conversion of inputs (...) into outputs (...) in the most cost-effective way possible, as compared to feasible alternatives in the context" (p.58), which makes it extremely hard to assess the efficiency of, say, a project that supports litigation in international courts - interventions that may take decades to yield the desired result. However, the guidance document states that resources should be understood in the broadest sense and include full economic costs. On that basis, one can indeed argue, as Jasmin Rocha and I have on Zenda Ofir's blog, that non-monetary costs, hidden costs and the cost of inaction must be taken into account. Yet, table 4.4 on efficiency-related challenges remains vague (p.61). Has anyone read the reference quoted in the table (Palenberg 2011)? I did and found it very cautious in its conclusion. My impression is that in many cases, evaluators of development interventions are not in a position to assess efficiency in any meaningful manner.

On the whole, I would describe the new OECD-DAC publication as a big step forward. I warmly recommend it to anyone who designs, manages or commissions evaluations.

 



Für Deutschsprachige: Online Moderieren - Lebendig und Produktiv

Mein online-Workshop zu online-Workshops ist am 26.Mai online! Mehr Informationen und Anmeldungsmöglichkeiten gibt es beim PME-Campus

Apologies to those who don't speak German - my first workshop on online facilitation will be in German. But if it works out nicely, I might offer sequels in English and in French! A bon entendre, Michaela

Friday, 12 March 2021

Join my workshop on online facilitation in PME

After a year of lockdown-induced life in cyberspace, web-based workshops have become a routine in planning, monitoring and evaluation (PME). Workshops are about exchange, about developing something together. But often, I have witnessed online workshops that were so virtual you hardly noticed the participants. Seemingly endless pages of screen-shared text being read out, word by word, in a soothing voice. No breaks. Confusion about technical refinements, links posted to inaccessible clouds. The loneliness of the person who finds herself alone in the main channel, without any pre-assigned breakout group... 

But there are also online workshops that actually work, that engage us in an exhilarating process, and that produce results. They can be more efficient than real-life workshops. And they surely save enormous amounts of CO2 and travel costs. Even though many miss the informal encounters at the coffee machine, over lunch, in the bathroom line (we're in 2021 and people identified as male still enjoy better acess to toilets than the rest of us) -  I suspect that online workshops are here to stay. 

It takes deliberate planning, adaptive pacing and plenty of participation to make an online workshop work. After more than a year of developing, facilitating and documenting a range of workshops on different platforms, I am distilling key insights into a short workshop for German speakers - online, of course! Not the technical stuff - the providers' video tutorials take care of that - but key principles and ways to apply them. I'd be delighted to meet you. More details in German) are available here

Wednesday, 30 December 2020

Gender equality in organisations: a good resolution for 2021

Gender equality is a key element of sustainable development – as illustrated in the Sustainable Development Goals (SDGs), which weave gender across virtually all 17 SDGs. It makes sense that 'mainstream' organisations, which are not specialised in promoting gender equality, have developed gender policies and related activities. Where are they at, and what should come next?

Friday, 25 September 2020

More handy tips for videoconferences

An addendum to yesterday's post - ICA:UK, a reliable source of materials and training on highly participatory facilitation, has summarised 10 principles to prevent online fatigue. I've been using all of them. They work. 

At any rate, avoid text-filled slide shows with voices droning on in the background! Visual aids are great, but if you just show pages and pages of text that you read to your audience, they'll end up muting you and joining a different event on their other computer. I admit that was what I did in a recent conference full of half-hour text-rich presentations by invisible voices. I couldn't help it. Since the different conference followed the same mode, I still felt I would have been better off reading an article, in my own time, at my own (rather energetic) pace.

Tuesday, 22 September 2020

Easy socialising in tight video conferences

How to recreate a sense of a "real life" team event in a video conference? In real life (IRL, as nerds put it), people usually linger near the coffee/tea kitchen or in the hallway for a quick chat - one reason why it tends to be so hard to get participants back from "real" breakout rooms. 

"Random" virtual breakout rooms - if they don't come with too burdensome assignments - can recreate this atmosphere.

Like most facilitators I know, I have facilitated more video conferences in 2020 than ever before. I have discovered that participants tend to hijack virtual breakout rooms: Before getting started on the small group assignment, they'd have an informal chat on totally different subjects. Or, in other cases, they'd get the assignment done as fast as possible so as to spend the rest of the small group chat on their own agendas.

Thursday, 10 September 2020

Know what you need to know

Evaluations often come with terms of reference (TOR) that discourage even the most intrepid evaluator. A frequent issue are long lists of evaluation questions that oscillate between the broadest interrogations – e.g. “what difference has the project made in people’s lives” – to very specific aspects, e.g. “what was the percentage of women participating in training sessions”. Sometimes I wonder whether such TOR actually state what people really want to find out.

I remember the first evaluation I commissioned, back in the last quarter of the 20th century. I asked my colleague how to write TOR. She said, “Just take the TOR from some other project and add questions that you find important”. I picked up the first evaluation TOR I came across, found all the questions interesting and added lots, which I felt showed that I was smart and interested in the project. Then I shared the TOR in our team and others followed suit, asking plenty more interesting questions.

I wonder whether this type of process is still being used. Typically, at the end, you have a long list of “nice to know”-questions that'll make it very hard to focus on questions that are crucial for the project.  

I know I have written about this before. I can’t stop writing about it. It is very rare that I come across TOR with evaluation questions that appear to describe accurately what people really want and need to find out. 

If, as someone who commissions the evaluation, you are not sure which questions matter most, ask those involved in the project. It is very useful to ask them, anyway, even if you think you know the most important questions. If you need more support, invite the evaluator to review the questions in the inception phase – with you and all other stakeholders in the evaluation – and be open to major modifications.

But please, keep the list of evaluation questions short and clear. Don’t worry about what exactly the evaluator will need to ask or look for to answer your questions. It is the evaluator’s job to develop indicators, questionnaires, interview guides and so forth. She’ll work with you and others to identify or develop appropriate instruments for the specific context of the evaluation. (The case is somewhat different in organisations that attempt to gather a set of data against standardised indicators across many evaluations - but even then, they can be focused and parsimonious to make sure they get high quality information and not just  ticked-off boxes.) 

Even just one or two evaluation questions is a perfectly fine amount. Anything more than ten can get confusing. And put in some time for a proper inception phase when the evaluation specialists will work with you on designing the evaluation. Build in joint reflection loops. You’ll get so much more out of your evaluation.


Monday, 13 July 2020

My first hackathon #EvalHack

The International Programme for Development Evaluation Training (IPDET) has moved its 2020 summer school on-line. As an extra, it has opened up its online evaluation hackathon to external people - like me. I have taken up the invitation, registering for my first evaluation hackathon. Our group challenge was to develop a prototype for efficiency measurement. This is what we have come up with: Evaluation United. If you like it, scroll to the bottom of that web page and LIKE it; we're in a competition! 

Sunday, 10 May 2020

Five tips for remote facilitation

Despite the risks and uncertainties associated with independent consulting, I have never felt as privileged as I do now, living in a country with a highly developed, accessible health system, working from my customary home office, and equipped with a decent internet connection and the hardware needed to stay in touch with friends and colleagues. The crisis has been an opportunity to develop my remote facilitation skills. Before, I facilitated the occasional "real-life" workshop in a video conference room with participants in other locations joining us via Skype or the like. I have shared that type of hybrid experience on the Gender and Evaluation community pages. Now I have gone one step further, facilitating fully remote workshops from my home office. I mean interactive workshops with some 5-20 people producing a plan, a strategic review or other joint piece of work together - not webinars or explanatory videos with hundreds of people huddling around a lecturer who dominates the session. To my delight, virtual facilitation has worked out beautifully in the workshops I have run so far. Good preparation is a key element - as in any workshop. I have distilled a few tips from my recent experience and from the participants' feedback.

Sunday, 29 March 2020

Facilitating with care in lock-down situations

In the current situation of (near-) lock-downs, interviews, meetings and group discussions are moving on-line. And so are workshops, with tutorials and useful advice on running on-line workshops popping up all over the place. If, as a facilitator, you want to make sure all workshop participants make the best of a virtual workshop, the challenges of moving on-line are not just technical. When everybody has to work from home, your workshop enters the participants' households. To express it in terms of systems thinking, your workshop is not confined within the borders of office spaces and working hours anymore; it intersects with a number of household systems:

Wednesday, 11 March 2020

International Evaluation in Times of the New Coronavirus

What does the surge of SARS-CoV-2 (the scientific name of the new coronavirus) infections in parts of Europe mean for international evaluation? Can we, as evaluators, join the soothing voices of those who say, the current common flu epidemic has killed many more people and there is no reason to change anything in our lives? I don't think so. I would like to remind all of us of the Do No Harm principle: Research ethics require us to carefully weigh the potential benefits of undertaking research (at a given time) against the potential harm associated with it. We can relax about ourselves but we must not endanger others. International evaluations can also be done without international travel.

Monday, 17 February 2020

Proud to be Rosa Marina Flores Cruz's mentor

Rosa Marina is the third mentee I present on these pages. Although our mentorship has formally ended, we're still in touch. Rosa has kept me up-to-date on her engagement for indigenous people's rights around the world - most recently at the Indigenous Women in Community Leadership gathering in Antigonish, Canada (November 2019). As with the previous posts, I am presenting Rosa in a written interview. 

(1) What are the main issues that you are currently working on?

I am an Afro-indigenous activist and researcher from the Isthmus of Tehuantepec in the State of Oaxaca, Mexico. I work on topics like rural feminism, environment and energy, and the autonomy and rights of indigenous peoples. I have worked in training projects for community health and human rights promoters, and for the defence of the Right to Free and Informed Prior Consultation. 
Photograph by Shirley Kimmayong


Tuesday, 10 December 2019

Less is more - also in evaluation questions

Writing evaluation terms of reference (TOR) - that is, the document that tells the evaluators what they are supposed to find out - is not a simple exercise. Arguably, the hardest part are the evaluation questions. That section of evaluation TOR tends to grow longer and longer. This is a problem because: Abundant detailed evaluations questions may lock the evaluator into the perspective of those who have drawn up the TOR, turning the evaluation into an exercise with quite predictable outcomes that limit learning opportunities for everyone involved. 

Thursday, 5 December 2019

Proud to be a mentor for Nurshaim Tilenbaeva

Nurshaim Tilenbaeva was my second mentee under the Global Change Leaders Programme. Mentoring Nurshaim, I embarked on a learning journey about about Kyrgyzstan, where Nurshaim lives. I do hope I'll get a chance to travel there! 
Photograph by Kusekhaya

There was one day when I was moved to tears by Nurshaim's honesty in describing difficulties she was facing at the time, at a different work place. Reading her narrative of that conversation now, I remember interviews I have carried out as part of a recent study on gender policies and practice in four German organisations. A couple of persons told me about a leader in their organisation shedding tears in public - which they described as a display of emotional intelligence and an empathetic form of leadership. Interesting! Meanwhile, as I have accumulated more experience in mentoring and coaching, I have learned to empathise differently - but I still get visibly moved when a mentee tells me about her success! And here are Nurshaim's answers to my questions: 

Monday, 4 November 2019

Proud to be a mentor for Kateryna Kravchuk

Five years ago, the Coady International Institute at St. Francis Xavier University in Nova Scotia (Canada) contacted me with a request to mentor a woman who had just graduated from its Global Change Leaders Programme (GCL). The idea was that we would have a one-hour Skype call every two weeks over a six-month period, to support the graduate in her efforts to translate learning from GCL into practice at home. I loved the idea! My first mentee was Kateryna Kravchuk from Ukraine, very close to my home time zone in Berlin.

Mentorships are great learning processes for everyone involved, I believe. I have happily agreed to mentor further GCL graduates since that first experience. Some mentees have answered a short set of questions to present themselves on my blog. I will post a mini-series of menteesin the coming weeks, starting today with my first GCL mentee, in her own words: 
Photograph by Andriy Maksymov
Kateryna Kravchuk - the economist, cultural researcher, focused in strategy, monitoring and evaluation of the projects aimed at improving the quality of life. Her main areas of expertise are community development, responsible entrepreneurship, cultural and creative industries, and cross-sector cooperation. 

(1) What is the main issue - or are the two or three main issues - that you are currently working on?

Currently I am busy with three projects: Creative Enterprise Ukraine (workshops and mentring for young entrepreneurs; keynote speaker for the Gender equality in culture and creative industries research), the evaluation of the UNDP/ADA project “Sustainable local development in rural areas” (two regions in Ukraine) and strategy development for Ivan Honchar Museum – centre for traditional culture in Kyiv. The main issues that are calling me since my childhood and more or less predefine everything I do are: