Gentle evaluations for huge projects?

Some weeks ago we – Wolf Stuppert (my associate and co-author of this posting) and I – noticed a call for evaluation proposals that seemed exciting. It was about a 50-million-USDollar initiative in a field where both of us have substantive experience. 

At first sight, the terms of reference (TOR) looked exciting: an ambitious, nationwide programme that would have to be asessed for its replicability in other contexts. But our level of excitement dropped dramatically when we studied the TOR in more detail.

Measuring time with a yardstick?

Much of my work on monitoring and evaluation is with organisations that work on human rights and governance. It is harder for them to show they make a difference than for people who build roads or wells. Building a road is complicated, but the builders have reasonable control over the process and at the end they can say, "we have built a 50-kilometre road". Human rights and governance work is more indirect: it is not the intervention of  human rights organisation "X" that frees a prisoner - it is a prison guard, the last piece in a huge puzzle of actors and actions. It is not campaign Y that ends domestic violence in a woman's life - it is herself, when she leaves an abusive relationship, or the abusive partner when he stops battering, for reasons that are far beyond the reach of campaign "Y". And you can't blame, say, Amnesty International if the US government fails to close down the Guantánamo detention camp.

Quantitative is qualitative, too...

Yesterday I read an advertisement for a body lotion scientifically proven* to better the skin of 80% of lotion users. The *footnote explained that, in a trial bringing together twenty women, 80% stated the lotion made their skin feel smoother. Does that sound scientific enough? In any case, it illustrates how you turn qualitative judgements (respondents' reported feelings) into "hard" figures, a procedure which as such is not "manipulative" but established scientific practice.

The Listening Project

CDA Collaborative Learning Projects, best known for Do No Harm, its peace and conflict impact assessment tool, launched LISTEN in 2007. Under this initiative, the "receipients" of international aid, rather than its "providers", voice their views on international aid. Reports on listening exercises with wide ranges of informers in 13 countries can be downloaded from the CDA site. They are useful background information for appraisal or strategic planning exercises - and they remind us of the importance of seeking and keeping in contact with people beyond the microcosm of international aid systems.