Tuesday, 15 July 2014

We're still alive!

This is the first time I have not posted anything on my blog for 2 months in a row. It has been a very, very busy time. My associate Wolf Stuppert and I have completed our review of evaluation approaches and methods for interventions related to violence against women and girls. If you want to find out more about it, please visit the dedicated blog www.evawreview.de, where you can also download the full review report, as well as the scoping and inception reports. 
Watch this space for more posts in the very near future. Thank you for your patience!

Wednesday, 14 May 2014

Another nice resource on bias

The Center for Evaluation Innovation has published a 15-page booklet called How Shortcuts cut us Short - Cognitive Traps in Philanthropic Decision Making. Click on the title to get to the site where you can download the publication. It explains what "confirmation bias", "escalation of commitment", "availability bias", "bounded awareness" and "groupthink" are about, and how they risk clouding the judgment of grant makers and other decision makers in well-meaning ventures. Anyone who has been involved in grant making will recognise familiar situation.

Hindsight bias, Mr.Wray would say. He has recorded the catchy tune embedded above which summarises the different types of bias that can obstruct intelligent decision-making. You'll find a few extra ones that you may have come across at work, too.

Monday, 14 April 2014

Resources for facilitators

A friend has asked me about resources on facilitation of planning workshops. This makes me realise that my approach in facilitation is fed by many different streams. Some favourite resources in English:
  • One is the "Technology of Participation", developed over a period of about half a century by the Institute of Cultural Affairs. Apparently a recent ToP handbook has been published (late 2013 or early 2014).
  • Another favourite source: "Time to Think" by Nancy Kline. Maybe a bit wordy (pleasant for people who enjoy reading stories about the author's many clients), but definitely worthwhile.
  • On effective low-tech visualisation, "The back of the Napkin" by Dan Roam is full of lovely ideas.
  • Also nice, "Game Storming" by Dave Gray, Sunni Brown and James Macanufo and "Visual Meetings" by David Sibbet.
  • If you're into systems and complexity, "Systems Concepts in Action" by Bob Williams and Richard Hummelbrunner offers inspiration.
  • And then there is that fat compendium of facilitation methods alled "The Change Handbook" compiled by Peggy Holman, Tom Devane and Steven Cady. The descriptions are rather short, but if you don't look for step-by-step guidance, it is quite adequate.
I recommend you start with the first 3 items on the list. More time-tested facilitation tips are on the page "Planning, Strategies, Tools" on this blog. I particularly recommend the posting "Tips for Multi-Everything Facilitation". (Apologies for the messy layout of that page - this blog is an personal unpaid spare time initiative and I find too little time to make it all look neat and smart.)

Friday, 4 April 2014

Review of Evaluations - Inception Report ready

It has been quiet again on this blog - this is because we have been busy producing the Inception Report and pursuing our research on approaches and methods in evaluations of interventions on violence against women and girls. (Apologies I still haven't found a shorter way of saying this!)

You can download our full inception report and find the link to an interesting discussion of our work by Rick Davies on our dedicated review blog www.evawreview.de.

Friday, 7 March 2014

Gentle evaluations for huge projects?

Some weeks ago we – Wolf Stuppert (my associate and co-author of this posting) and I – noticed a call for evaluation proposals that seemed exciting. It was about a 50-million-USDollar initiative in a field where both of us have substantive experience. 

At first sight, the terms of reference (TOR) looked exciting: an ambitious, nationwide programme that would have to be asessed for its replicability in other contexts. But our level of excitement dropped dramatically when we studied the TOR in more detail. Although all DAC criteria were listed – relevance, effectiveness, efficiency, impact, sustainability -, the key questions under those headings seemed suprisingly modest. They focused chiefly on programme process and results among direct programme stakeholders, i.e. the non-governmental organisations that had received grants and free training under the programme. That is, the evaluators would ask those who have obtained those goodies whether they felt the programme was effective. 

The reputed international accounting firm that has run the initiative (and drafted the TOR?) should know that a certain amount of bias might cloud the judgement of people who have drawn such immediate benefits from the programme. 

No mention whatsoever, in the TOR, of the ultimate beneficiaries – the citizens of that country who are expected to enjoy more responsive and accountable governance as a result of the programme. Although the programme has been going on for several years, the only evaluation question about impact is fairly abstract; inviting the evaluators to speculate about the extent to which the outcomes achieved might contribute to longer-term changes.

If the evaluation is supposed to test the replicability of the initiative, it would seem important to scrutinise the theory of change underlying the programme, the way it has been translated into action and the changes it has contributed to. What the TOR calls for falls short of that – by a long way. 

For instance, a special TOR section on potential risks and limitations explicitly rules out quantitative data collection and counterfactuals. Instead, the prospective evaluators are invited to rely primarily on their own judgment, “backed by qualitative evidence” that would be drawn from statements and reports produced by the organisations running the programme. It is unclear what specific risks such restraint is supposed to address – not the risk of discovering the programme has passed unnoticed in the society it is supposed to strengthen, we hope?

We love qualitative research and we feel it is important to gather stakeholders’ views on the programmes they are implementing. And we do not say that every development project needs an impact assessment. In many cases, an exercise that focuses on process and immediate outcomes can be perfectly sufficient. (For instance, many smaller initiatives are so grossly understaffed or underfunded that one can’t expect them to produce any significant results anyway – in such a situation, a combination of in-depth conversations and an experienced evaluator’s own judgment may help to draw attention to necessary adjustments.) 

But if you want to gauge the replicability of a multi-year multi-million-dollar initiative, then you’d better do it with the thoroughness and transparency it takes to produce robust findings. The firm that runs the programme convinces clients around the world to invest massive amounts of money into accountability. It should know how to run the kind of evaluation you need to find out whether a programme works. Are transparency and external scrutiny less important when it comes to one’s "own" programmes? 

Who owns those programmes, anyway? But that opens a different discussion...

Tuesday, 18 February 2014

Tell a story: Evaluations that Make a Difference

The research project "Evaluations that Make a Difference" is looking for stories about evaluations. The idea is to explain what has made evaluations influential or successful in an emotionally more engaging way than scientific publications. A lovely idea! Find more information by clicking on the following link: Call for Stories | Evaluations that Make a Difference

Wednesday, 12 February 2014

Evaluations of interventions related to violence against women and girls

A quick update on our review of evaluations - the full scoping report, a link to dozens of published evaluation reports in the field of violence against women and girls, and more are now available on our dedicated blog www.evawreview.de
We have started the blog because we use a particularly interesting method in our review - Qualitative Comparative Analysis. Our inception report, to be posted on the blog in March, will provide more detail. Come and have a look!

Thursday, 9 January 2014

A quip on women & maths

Yesterday I bought a block of 20 stamps à €0,02. (The German postal services have increased the postage cost for letters from €0,58 to €0,60.)
While the man behind the cashier started to work his calculator, I said "that'll be €0,40 I suppose". The salesman said, "I always use the calculator... erm... that'll be €0,40 indeed. No surprise our Chancellor is a woman."
The German Chancellor, Angela Merkel, is a woman indeed, and she has a PhD in Sciences. (I shall refrain from discussing her politics here.) 
Germans have realised that women are good at maths. Q.E.D.

Tuesday, 7 January 2014

Happy 2014 - quieter times ahead here

My best wishes for a happy 2014! For me, 'happiness' includes health, peace, a reasonable degree of prosperity and all other good things we wish each other in this season. Plus plenty of sunshine. (Unfortunately I can't insert any sunshine picture with this text today because Google seems to have introduced a bug - or some evil new twist - into its Blogger software.)

Early 2014 will be relatively quiet on this blog, as my attention is quite focused on our project with the UK Department for International Development (DFID), which is accompanied by a dedicated blog: www.evawreview.de But I will do my best to continue posting at least once a month on www.developblog.org

Maybe this year I'll manage to revamp the blog, making it more searchable and updating the design? We'll see.
If you wish to receive new posts by e-mail rather than looking for them on this page, feel free to use the "subscribe" function, to the right of this page.

Saturday, 21 December 2013

Resources on gender, and gender & corruption

In 2012 and 2013, I carried out short series of gender sensitisation and gender mainstreaming training for organisations that work on development, human rights and governance issues. Some of the materials that have inspired the training are in the public domain. My favourites are:

  • The gender training manuals by the UK Department for International Development (DfID) and the Swiss Development Cooperation (SDC). Click on this link to find both resources; the SDC toolkit is available in English, French and Spanish. You will also find a folder with a guide on 'mainstreaming' work to end violence against women into development interventions which I have written for Oxfam International (in Arabic, English, French and Spanish).
  • The OECD DAC Guiding Principles for Aid Effectiveness, Gender Equality and Women's Empowerment, as well as OECD Guidelines for Gender Equality and a handy sourcebook on Concepts and approaches can be found in this folder - as well as on the OECD Gender Equality pages.
  • Basic concepts in gender analysis are nicely explained in this presentation by the Dutch Royal Tropical Institute (KIT). For more KIT resources on gender and development (including a recent course on gender in value chains), see the KIT site. 
  • I have also compiled a set of resources on gender-sensitive communication.
  • Ten studies and other documents on gender and accountability and gender and corruption are available here; I have compiled them from various web sources.
  • I have also built a folder with good gender and development resources in French.That folder includes a few training modules I prepared in 2012, inspired chiefly by the excellent series (in French) by Le Monde Selon les Femmes, a Belgian NGO specialised in gender and development. Unfortunately, most of their materials are available as hard copies only.
One caveat: the vast majority of these resources focus on gender in terms of the familiar male-female dichotomy. Arguably, discrimination and violence against women are gender issues that affect the largest numbers of people. But discrimination and violence against sexual minorities, transgender and intersexual persons are also gender issues. I hope that soon, gender manuals will include references and practical tips around the Yogyakarta Principles - an application of international human rights law in relation to sexual orientation and gender identity. Click on "Yogyakarta Principles" to find the doucment in all six United Nations languages.