Thursday, 9 August 2018

Still some places available at our EES professional development workshop

Please join us at the biennial conference of the European Evaluation Society in Thessaloniki in early October!  Jasmin Rocha and I will be there, participating in lots of inspiring sessions and delivering a paper on using the DAC criteria in evaluations in the field of human rights. 

On October 1st, we'll run a pre-conference professional development workshop with the title "How to deal with 'mission impossible' evaluations"

The workshop is designed to enhance participants’ skills and knowledge in 
  1. Identifying pitfalls in evaluation terms of reference (TOR), so that evaluators can 
  2. Make well-founded choices with regard to submitting an offer/expression of interest (or decide it is not worth the effort), 
  3. Develop robust proposals for evaluations, especially of programs and project portfolios, and 
  4. Communicate the proposals effectively to the (potential) client. We will use concrete examples from real evaluations and distil learning from participants’ experience, to develop a framework that can guide participants in making the best of sometimes confusing, over-ambitious or excessively laconic TOR.
You can register for the workshop here. In case you experience any technical issues in the registration process, there are e-mail addresses of the conference secretariat who can help you.

We hope to see you in Thessaloniki!

Wednesday, 30 May 2018

Recent resource on human rights funding

The Foundation Center has published an appealingly designed, informative resource on human rights grant making around the world, mapping tens of thousands of grants by hundreds of foundations. 

You can search the database by region, by issues/rights, by "populations" (meaning, for example, women & girls, human rights defenders, migrants & refugees) and by strategies (for instance "grassroots organising", "arts and culture", "litigation and legal aid"). The resource is reasonably up to date; data on the grants is from 2011 to 2015. More "qualitative" information has been posted up to late 2017, with promises for new updates in 2018. 

Obviously, this resource is very interesting for those who who seek prospective donors. But its structure can also inspire those who wish to gain a structured overview of international human rights work - with the caveat, though, that this resource does not reflect the kind of work that does not get international funding.

Free e-prints of our article on effective evaluation in the field of violence against women and girls

A couple of years ago, Wolfgang Stuppert and I conducted a review on behalf of the UK Department for International Development (DFID) of evaluation methods in the field of violence against women and girls. We used qualitative comparative analysis (QCA), a relatively new method, that has yielded useful insights. You can find plenty of information and links on our dedicated site

Meanwhile, Development in Practice has published our article on this review. The article is geared to evaluation practitioners - people who commission, conduct and use evaluations. We can share a few free e-prints; if you wish to have one, use this link. First come, first served. When the free lot is used up, you can still view the article on the Taylor and Francis site here and, of course, in the printed publication. 

Our full report is still available from the UK government, for free.

Join us at the EES Conference in Thessaloniki - October 2018

The European Evaluation Society (EES) convenes its biennial conference in Thessaloniki from 1-5 October 2018. Join us there! Social scientist Jasmin Rocha (Camino gGmbh, Berlin) and I will run a pre-conference professional development workshop on 1 October. The working title is: How to deal with 'mission impossible' terms of reference. And this is what our workshop is about:

Evaluation TOR tend to come with more and more questions and detailed methodological requirements, while deadlines and budgets are often tight. The mismatch between ambition and resources appears to be growing; commissioning agencies often list high numbers of evaluation questions to be answered within unrealistic time frames and budgets. Not only young evaluators but also experienced ones can feel overwhelmed by the ensuing challenges. In dynamic and sometimes rapidly changing environments, narrowly defined evaluations equipped with insufficient resources often fail to fulfil their purpose to support learning. 

We intend to bring together young evaluators with more seasoned ones to create a space where we can jointly improve our capacities to identify and address potential challenges in evaluations with ‘mission impossible’ ToR. Key themes will be appropriate evaluation design, methods and communication. 

Have a look at the conference website! Preferential "early bird" registration is available until 15 June. We'd be delighted to see you in our workshop! 

Tuesday, 13 June 2017

Friendly feed-back on our quasi-experimental evaluation

The German national centre for crime prevention (NZK) has published a synthesis of studies on efforts to prevent violent extremism among youth. Our 2015 study on the HEROES project in Berlin (Raab & Stuppert 2015 - unfortunately Wolfgang Stuppert is misspelled in the article) has been lauded for (1) our  definition of a highly relevant central evaluation question which the evaluation could actually answer with methods that we described transparently, and (2) the strengthening of the internal validity of our findings by the use of propensity score matching in quasi-experimental research.

Those who wish to read the full NZK text (in German only) can find it here. A summary of our study is available from this page of the Berlin Senate (in German, too); just download publication BFG Nr.57/2015.

Tuesday, 6 June 2017

Choosing appropriate evaluation methods

Barbara Befani, a reliable source of high quality information, has held a talk at the Centre for Development Impact on choosing appropriate evaluation methods. Listen to her presentation here; it comes with a slide show that is best watched while listening to the talk.

The spreadsheet which helps choosing appropriate evaluation tools is available from the BOND website here.  Enjoy! And contain your disappointment if you realise that most methods won't work with the weak data and skimpy resources available for your evaluation.

Wednesday, 10 May 2017

Evaluation Norms and Standards - now in six languages!

The United Nations Evaluation Group norms and standards for evaluation are now available in all UN languages - Arabic, Chinese, English, French, Russian and Spanish. 

Read them - spread them - use them! You can find them here.

In recent years I have been involved in large reviews of evaluations in the development sector, studying a couple of hundreds of evaluation reports. Many of the evaluations would have been vastly better if they had complied with, or at least been inspired, by evaluation norms and standards. I don't say that every evaluation must attain the quality of rigorous scientific research - in the opposite, time and resource constraints usually don't permit rigorous research in the context of evaluations, anyway. But you can still try to make it useful. 

At least, evaluators must make sure they respect the rights and safety of people who participate in the evaluation - i.e. ethics. Otherwise, the harm caused by an evaluation might outweigh its benefits.

Tuesday, 9 May 2017

A must-read for evalutors, campaigners and everyone: Beware of confirmation bias

Followers of this blog know that bias is a recurring topic on these virtual pages. Evaluators must be aware of bias to produce robust work. After this year's elections and the disturbing success of "alternative facts" and "post-factual" phenomena, the topic's importance has been acknowledged way beyond research circles, entering public debates.

Bias affects everyone's judgment - and that includes ourselves! A good way to deal with bias is to know about it, to seek out opinions that conflict with ours (i.e. mine, yours, everybody's), and to keep thinking.

A February 2017 issue of the New Yorker aptly summarises a few recent books on confirmation and "myside" bias. Find it here and read it - it may change your way of seeing things, and help you understand why others may sometimes seem terribly obtuse.

And if you like the idea of a song about the types of bias that may occasionally cloud your mind, have a look at this earlier post.


Sunday, 30 April 2017

Getting those evaluation reports right

The other day I tried to remember how many evaluation reports I reviewed or used over the last ten years - probably a couple of hundreds. An astonishing number of reports felt like a a waste of time (and money). Anyone who is in the international development business knows how extremely diverse evaluation reports tend to look in this sector - from clear, concise and comprehensive to inintelligbile, interminable and still incomplete.

It is super-frustrating when - for example - you carry out a meta-analysis of evaluations and you have to discard half of the reports because of their poor quality.
Now, the great Better Evaluation site has published a checklist for evaluation reports, which lays down the characteristic of the ideal evaluation report. Find a link to download it here (a new page opens and there you'll click on "View Resource" to get the checklist). It is an annotated checklist with tips on how to write well. A wonderful resource. Use it and spread it to everyone who writes evaluations!

By the way - and I believe I have posted that earlier - Better Evaluation also offers guidance for managing evaluations and TOR writing. An important resource for those who commission evaluations. Because even the best-written report will be of little use if it answers the wrong questions, or if it can't answer the TOR questions because the resources at hand don't match the expectations. 

Thursday, 23 February 2017

All that research for NOTHING?

A very accessible video by 3ie starts with evidence from a recent study by the World Bank. It has found that a third of the research reports published by the World Bank on its website have never been downloaded by anyone and 87% never referred to in any subsequent research. This is extremely wasteful. Remember, our tax money pays for such research.

So the big problem is maybe not that too few research reports and evaluations are published - the big problem is that people don't use them, because they don't look for such evidence, or they find the reports too hard to understand. The latter can be dealt with by writing in a more accessible style. Finding time to think and explore may be more complicated.

Monday, 20 February 2017

New Resource on Women, Land and Corruption

Transparency International has just published a guide which is the fruit of a three-year process I have had the privilege to support with my facilitation and writing. 

The guide builds on the experience of Transparency chapters in Ghana, Uganda and Zimbabwe, who have recognised that they can work more effectively on land and corruption issues if they involve more women and take into account the needs and interests of women and men. It is called Gender-Responsive Work on Land and Corruption: A Practical Guide; if you click on the title you'll be taken to the link where you can download it. (At the time of writing this post there is a typo under a caption; try to spot it before they correct it!)

Sunday, 25 December 2016

Resolutions for 2017

Dear reader,
if you drop by regularly, you may have noticed that this blog has been in a state of semi-hibernation for a few months. I have been wildly busy working on a complex evaluation with a wonderful team; this will keep me busy for many more months to come. So, things will stay relatively quiet on these pages. But I'll add posts every so often. (At the moment, I am awaiting copyright clearance to repost the "sidekick manifesto" on development - summarised in a lovely infographic. Watch this space.) 

If you are looking for inspiration on evaluation - one of my favourite browsing pastures in that field is, which currently advertises a special holiday package of free gadgets, including - inter alia - the world clock meeting planner that identifies the best possible moment for meetings across time zones, the GeneraTOR software that helps you to draw up TOR, and a few graphics programmes for logical models (slightly more comfortable than designing diagrams with Power Point).

Meanwhile, all my best wishes for a healthy, happy and successful 2017!

Monday, 22 August 2016

Getting those indicators right

In the world I work in – social development – the terms "theory of change" and "indicator" have become essential. You can't just try doing something that might improve people's lives. You need to explain why and how the things you (plan to) do will contribute to the improvements you want to happen, i.e. you need to develop a theory of change. A common way to visualise a theory of change is the logical framework, which ideally shows how interconnected activities and their immediate products (“outputs”) are supposed to contribute to further-reaching changes, i.e. goals, objectives, outcomes, impact and so forth.

Your theory of change can be more or less abstract, more or less detailed and restrictive,

Thursday, 26 May 2016

Let's evaluate together

This is the time of the year when I would like to be able to clone myself, to respond to all those requests for evaluation proposals (RFPs) while busily working away on on-going jobs that need to be completed before the Northern hemisphere summer break breaks out. List servers publish new RFPs every day; as July approaches, the deadlines become increasingly adventurous. In late May, RFPs ask for offers to be submitted by the first week of June; the selected evaluation team would start working right away. It seems many of those who publish those last-minute quick-quick RFPs assume evaluation consultants spend their days sitting in their offices, twiddling thumbs, chewing nails or randomly surfing the web, waiting for that one agency to call them up and get them to work right away, tomorrow! Drop everything and work for us!

Monday, 18 April 2016

Work to be done: Ending violence against children

A recent report on the global prevalence of violence against children in the past year has shown that  more than half of the children in 96 countries across the world —1 billion children aged 2–17 years—experienced violence in the past year. Violence against children is a human rights violation. It makes people more likely to fall ill, and to  experience and perpetrate violence in their adult lives. In other words, violence is passed on through generations - even genetically, as it can alter a child's genes.

Sustainable Development Goals (SDG) call for an end to “abuse, exploitation, trafficking and all forms of violence against and torture of children” (SDG 16.2) and to “eliminate all forms of violence against all women and girls in the public and private spheres, including trafficking and sexual and other types of exploitation” (SDG 5.2). SDG 4 on education refers to the importance of promoting non-violence in several sub-goals, e.g. by calling for a non-violent environment for education (SDG 4.a).

With probably more than half of the world's children experiencing violence, major efforts are needed to attain the SDGs. For inspiration, have a look at UNICEF's Six Strategies for Action. If you know of any useful resources to share, please post a comment and share.