Thursday, 10 September 2020

Know what you need to know

Evaluations often come with terms of reference (TOR) that discourage even the most intrepid evaluator. A frequent issue are long lists of evaluation questions that oscillate between the broadest interrogations – e.g. “what difference has the project made in people’s lives” – to very specific aspects, e.g. “what was the percentage of women participating in training sessions”. Sometimes I wonder whether such TOR actually state what people really want to find out.

I remember the first evaluation I commissioned, back in the last quarter of the 20th century. I asked my colleague how to write TOR. She said, “Just take the TOR from some other project and add questions that you find important”. I picked up the first evaluation TOR I came across, found all the questions interesting and added lots, which I felt showed that I was smart and interested in the project. Then I shared the TOR in our team and others followed suit, asking plenty more interesting questions.

I wonder whether this type of process is still being used. Typically, at the end, you have a long list of “nice to know”-questions that'll make it very hard to focus on questions that are crucial for the project.  

I know I have written about this before. I can’t stop writing about it. It is very rare that I come across TOR with evaluation questions that appear to describe accurately what people really want and need to find out. 

If, as someone who commissions the evaluation, you are not sure which questions matter most, ask those involved in the project. It is very useful to ask them, anyway, even if you think you know the most important questions. If you need more support, invite the evaluator to review the questions in the inception phase – with you and all other stakeholders in the evaluation – and be open to major modifications.

But please, keep the list of evaluation questions short and clear. Don’t worry about what exactly the evaluator will need to ask or look for to answer your questions. It is the evaluator’s job to develop indicators, questionnaires, interview guides and so forth. She’ll work with you and others to identify or develop appropriate instruments for the specific context of the evaluation. (The case is somewhat different in organisations that attempt to gather a set of data against standardised indicators across many evaluations - but even then, they can be focused and parsimonious to make sure they get high quality information and not just  ticked-off boxes.) 

Even just one or two evaluation questions is a perfectly fine amount. Anything more than ten can get confusing. And put in some time for a proper inception phase when the evaluation specialists will work with you on designing the evaluation. Build in joint reflection loops. You’ll get so much more out of your evaluation.


Monday, 13 July 2020

My first hackathon #EvalHack

The International Programme for Development Evaluation Training (IPDET) has moved its 2020 summer school on-line. As an extra, it has opened up its online evaluation hackathon to external people - like me. I have taken up the invitation, registering for my first evaluation hackathon. Our group challenge was to develop a prototype for efficiency measurement. This is what we have come up with: Evaluation United. If you like it, scroll to the bottom of that web page and LIKE it; we're in a competition! 

Sunday, 10 May 2020

Five tips for remote facilitation

Despite the risks and uncertainties associated with independent consulting, I have never felt as privileged as I do now, living in a country with a highly developed, accessible health system, working from my customary home office, and equipped with a decent internet connection and the hardware needed to stay in touch with friends and colleagues. The crisis has been an opportunity to develop my remote facilitation skills. Before, I facilitated the occasional "real-life" workshop in a video conference room with participants in other locations joining us via Skype or the like. I have shared that type of hybrid experience on the Gender and Evaluation community pages. Now I have gone one step further, facilitating fully remote workshops from my home office. I mean interactive workshops with some 5-20 people producing a plan, a strategic review or other joint piece of work together - not webinars or explanatory videos with hundreds of people huddling around a lecturer who dominates the session. To my delight, virtual facilitation has worked out beautifully in the workshops I have run so far. Good preparation is a key element - as in any workshop. I have distilled a few tips from my recent experience and from the participants' feedback.

Sunday, 29 March 2020

Facilitating with care in lock-down situations

In the current situation of (near-) lock-downs, interviews, meetings and group discussions are moving on-line. And so are workshops, with tutorials and useful advice on running on-line workshops popping up all over the place. If, as a facilitator, you want to make sure all workshop participants make the best of a virtual workshop, the challenges of moving on-line are not just technical. When everybody has to work from home, your workshop enters the participants' households. To express it in terms of systems thinking, your workshop is not confined within the borders of office spaces and working hours anymore; it intersects with a number of household systems:

Wednesday, 11 March 2020

International Evaluation in Times of the New Coronavirus

What does the surge of SARS-CoV-2 (the scientific name of the new coronavirus) infections in parts of Europe mean for international evaluation? Can we, as evaluators, join the soothing voices of those who say, the current common flu epidemic has killed many more people and there is no reason to change anything in our lives? I don't think so. I would like to remind all of us of the Do No Harm principle: Research ethics require us to carefully weigh the potential benefits of undertaking research (at a given time) against the potential harm associated with it. We can relax about ourselves but we must not endanger others. International evaluations can also be done without international travel.

Monday, 17 February 2020

Proud to be Rosa Marina Flores Cruz's mentor

Rosa Marina is the third mentee I present on these pages. Although our mentorship has formally ended, we're still in touch. Rosa has kept me up-to-date on her engagement for indigenous people's rights around the world - most recently at the Indigenous Women in Community Leadership gathering in Antigonish, Canada (November 2019). As with the previous posts, I am presenting Rosa in a written interview. 

(1) What are the main issues that you are currently working on?

I am an Afro-indigenous activist and researcher from the Isthmus of Tehuantepec in the State of Oaxaca, Mexico. I work on topics like rural feminism, environment and energy, and the autonomy and rights of indigenous peoples. I have worked in training projects for community health and human rights promoters, and for the defence of the Right to Free and Informed Prior Consultation. 
Photograph by Shirley Kimmayong