Wednesday, 24 September 2014

EES presentation: Evaluation methods and approaches

Next week in Dublin, at the 11th Biennial Conference of the European Evaluation Society (EES), I'll present the findings of our review of methods and approaches in evaluations of interventions on violence against women and girls (written by Michaela Raab and Wolfgang Stuppert, commissioned by the UK Department for International Development DFID).

If you happen to be at the conference, you are warmly invited to drop by at one of the two sessions where our findings will be presented:

  • Poster presentation on Wednesday 1 October at 2pm, Poster area Liffey A.
  • Session on  Thursday 2 October at 9:45am at Wicklow, Meeting Room 2b. The session, facilitated by Zoe Stephenson (DFID), will focus on effective evaluation in the field of violence against women and children. Sally Neville (UK), Alexandra Chambel and Krishna Belbase (USA) and I will present papers.
If you wish to read the full review report, or find out about the methodology we have used (chiefly Qualitative Comparative Analysis -QCA- and Process Tracing), please visit the dedicated site The site includes updates from our research process, links to the full scoping, inception and review reports and (in the posting "Presentations at the EES Conference") a link to a handy 2-page document that summarises our findings. 

The QCA specialists on our review team, Wolfgang Stuppert and Julian Brückner, can't make it to the EES conference this year. If you have any technical questions on specific aspects of QCA in our review, feel free to contact Wolfgang Stuppert via the e-mail address contained in our reports - or rummage through, which includes our responses to questions raised by other evaluators.

Monday, 22 September 2014

To my readers, especially those in Ukraine

Six years ago, in October 2008, I started this blog. I had planned on developing a website to advertise for my consulting services. I kept pondering what would need to be on that site, what the design would need to look like to attract a wide range of potential clients, and so forth. At the same time I was quite busy with my consultancies, learning exciting things and meeting interesting people. I had started a couple of mailing lists which just kept growing, because I wanted to share so many things with so many professionals around the world. In the end, I decided to stop sending mass mailings and to start blogging instead. It turned out to be very easy, and a bit addictive.

I never found time to design that other website which would present my firm. But I still enjoy using this blog to share resources, impressions and opinions with people engaged in similar work. I try to post at least once a month. I make efforts to keep my link lists reasonably up-to-date, even though I have not yet found any moment to add extra pages that would deserve to be on those lists. 
I still resist the temptation to accept advertisements on the blog. This is an entirely voluntary, not-for-profit blog, written in the free moments I find between "normal" work and "normal" life.

The blog has proven quite successful - at the moment of writing, it has recorded more than 130000 page views. Thank you for the massive interest! The analytical data I get from the blog host show that my readership is spread across the entire world. Between 2008 and today, the top ten countries where most page views have originated from are the USA, Ukraine, France, Turkey, Germany, China, Netherlands, Romania, Russia and the United Kingdom. There are visitors from all continents; much depends on the topics, among other factors. For instance, when I posted a few pages in French, the blog received many visits from French-speaking African countries. After having met professionals from the Pacific at a workshop in New York, I started to record visits from Fiji and neighbouring nations.

But what I find most intriguing is that the stream of page visits from Ukraine remains consistently high. I know many people in or from the other countries on the list. But my small number of Ukrainian acquaintances are not involved in development and human rights work at all - as far as I know. That makes it particularly interesting for the blog to receive so much attention from Ukraine.  

Regardless where you are based - I would find it interesting to hear what you have found useful on this blog, or any other comments or recommendations you would like to make. Feel free to use the "comment" function at the bottom of this post to let me know . Clicking on "[no] comment" will take you to a dedicated pop-up window where you can type your comment. Anonymous comments are possible and welcome (as long as they don't advertise for a particular product). 

When reading comments, be careful about opening any links contained in those comments: Although I do my best to filter our spam, I cannot screen every single link displayed on this site.

Looking forward to hearing from you!

Wednesday, 10 September 2014

A gender-sensitive Human Development Index (HDI)

The latest Human Development Report (HDR) comes with interesting news on gender inequality. This is because, in addition to the gender inequality index (GII), UNDP has calculated male and female human development indices (HDI) separately. Comparing the "female" and the "male" HDI yields a new index called the Gender-related Development Index (GDI).

There is a huge difference between the GII and the GDI. 
The familiar GII has focused on three dimensions regarding the situation of women and girls: reproductive health (maternal mortality ratio and adolescent birth rates), empowerment (measured in terms of women's presence in parliament seats and in secondary education), and economic participation (percentage of females and males aged 15 and older in the labour market and labour force). Constructed in a very complex manner, the GII has been difficult to interpret and a bit clumsy to use.  

The new GDI is radically different. Rather than examining specific aspects that are suspected to contribute to (or hamper) gender equality, it takes into account all data that forms the human development index - data on health, education, and command over economic resources. 

The GDI looks at the bottom line: what are the differences between women's and men's human development indices in a country? UNDP points out that on average, the female HDI value is 8% lower than male HDI. In four countries, the female HDI is 30-40% below the male HDI: Afghanistan (40%), Niger, Pakistan and Yemen. Conversely, in Slovakia, the difference between men's and women's HDI is exactly zero.
In 15 countries, women's HDI is above men's: Sweden, Finland, Slovenia, Estonia, Lithuania, Poland, Latvia, Argentina, Uruguay, Belarus, Russia, Barbados, Kazakhstan, Ukraine, and Mongolia (HDI rank: 103), listed here in the order of their HDI rank. Intriguing! This unusual form of gender inequality can be caused by poor male HDI data, often due to below-average male life expectancy. Men and boys tend to be disproportionally involved in violence, alcoholism and other forms of risky behaviour.

Countries that rank high in terms of HDI are not necessarily near the top in terms of GDI: for instance, Switzerland ranks 3rd in terms of its HDI, but 76th in terms of gender-related development. Conversely, Mongolia is 87th on the HDI list, but 8th on the GDI ranking. Lesotho is 162nd regarding its HDI, but 43rd for the GDI. Some countries on the HDI list do not yet have a GDI rank - it is probably still difficult to get fully sex-disaggregated data from all countries. That makes it a bit complicated to draw accurate comparisons between HDI and GDI ranks, espcially near the lower end of the HDI list. But it is still exciting.

Tuesday, 12 August 2014

3ie comment on video tutorials

3ie, the International Initiative for Impact Evaluation, has sent a comment explaining why you need to register if you wish to take the useful tests that come with their video tutorials on impact evaluation. For some technical reason their comment cannot be displayed under my original post (below), so I am taking the liberty to post it right here:

>>Thank you for reviewing the Asian Development Bank-3ie video lecture on quasi-experimental methods. We appreciate your feedback. We would be happy to have you review the other videos in our series.

Your point on the log-in for the quiz is well taken. We are requesting for basic information right now to help the Asian Development Bank screen staff participants for the Making Impact Evaluation Matter conference in early September. From a technical perspective, such a feature also helps us keep spammers at bay. We may however decide to disable this at a later date.<<

Thursday, 7 August 2014

Video tutorials on impact evaluation

The Asian Development Bank (ADB) and the International Initiative for Impact Evaluation (3ie) have published a set of six video lectures on impact evaluation. They are available here. The lectures are presented by different specialists; the slides accompanying the lectures can be downloaded from the same website.

I have taken a look at Dr. Jyotsna Puri's lecture on quasi-experimental methods. Dr. Puri illustrates the use of such methods with a real example she has worked on. The lecture is quite clear, but be prepared for technical jargon and high information density. Also, a basic understanding of mathematics helps - you need to know where to find the x axis and the y axis on a graph. This is not for absolute beginners, but arguably, impact evaluation, in particular the kind of rigorous evaluation 3ie promotes, is not for beginners (nor for the financially meek, by the way).

A nice feature is the quiz that one can take after each lecture. What I like less about it is that you need to log in to take the quiz - what for, I wonder? There are no prizes to gain or diploma to obtain. Only the user's privacy to lose. In my view, that is an unnecessary hurdle which might put off some prospective learners.

Monday, 4 August 2014

Ending violence against women - what works?

In case you have not come across this yet: the UK Department for International Development (DFID) has published a whole series of "How To Notes" and "Evidence Digests" to guide work on violence against women and girls. There is a dedicated web page (click on the link to get there) where you can download the guides. 

The web page includes a link to Violence against Women and Girls Newsletters, which are published at quarterly intervals. The newsletters are rich in information on a wide range of interventions and tools.

Quality and quantity

In this holiday season I visited my sister, who has been passionate about gardening. I brought a beautifully illustrated book for her, about pear orchards in Prussia. Prussia, a belligerent kingdom that ceased to exist in 1918, was mainly known for its military dominance in the region, and for an obsession about order and discipline. So I was hardly surprised to find, in that book, a table showing drawings of differently shaped pears, arranged in neat rows and columns. The roundest pears were displayed near the top left corner, the thinnest, longest ones near the bottom right, with dozens of intermediary states in-between. Every pear came with a drawing of its appearance, as well as a transversal cut, which was criss-crossed by lines and dots dividing it into neat circles and measurements. "Look," I exclaimed, "the Prussians developed a system to classify pears!" My sister took a quick glance and responded, somewhat bitterly, "according to size, of course". Arguably, size was the only aspect of pears that could be reliably and replicably measured in those days. But what does its size say about a pear? It doesn't tell me whether the fruit is ripe or green, hard or soft, sweet or bitter, juicy or dry.  

The Prussians were great simplifiers. They were (in)famous for organising forestry in a way that measured effectiveness only in terms of its marketable yield. Basically, a forest was an amount of timber, nothing else. In "Seeing Like a State - How Certain Schemes to Improve the Human Condition have Failed", 1998, James C.Scott explains how that reductionist approach produced those sad monocultures of same-age trees in rank and file which have disfigured German landscapes ever since. The model was adopted throughout the world, a disaster for peasants and others who had used the many fruits and animals of rich, diverse forests. Rather than learning to produce maps and measurements faithful to the phenomena they found in nature, the Prussians and their followers aligned reality to their systems of measurement, cutting down forests to turn them into easily manageable and measurable plantations.

You have guessed why these musings appear on a blog on development and evaluation. Counting things can be useful, and often it is necessary. If you sell timber, you want to know how many cubic metres you can get out of those trees. But if you're into human development you need to know when quantity ceases to be the only important dimension, and where you have to look for quality to understand a phenomenon. If you do your qualitative homework well, you might even find ways to accurately describe the characteristics you are looking for, and turn those aspects of quality into scales that allows you to put a number to them. 

But it is a risky idea to try and bend reality until it can be counted. (Please, no chopping down of forests just to count the trees.) 

It is a good idea to think about monitoring and evaluation as soon as you start designing a programme. But let's resist the temptation to build only programmes that are geared to produce the kind of change that can easily be measured. Following that temptation would generate a very barren development landscape.

Tuesday, 15 July 2014

We're still alive!

This is the first time I have not posted anything on my blog for 2 months in a row. It has been a very, very busy time. My associate Wolf Stuppert and I have completed our review of evaluation approaches and methods for interventions related to violence against women and girls. If you want to find out more about it, please visit the dedicated blog, where you can also download the full review report, as well as the scoping and inception reports. 
Watch this space for more posts in the very near future. Thank you for your patience!

Wednesday, 14 May 2014

Another nice resource on bias

The Center for Evaluation Innovation has published a 15-page booklet called How Shortcuts cut us Short - Cognitive Traps in Philanthropic Decision Making. Click on the title to get to the site where you can download the publication. It explains what "confirmation bias", "escalation of commitment", "availability bias", "bounded awareness" and "groupthink" are about, and how they risk clouding the judgment of grant makers and others working on well-meaning ventures. Anyone who has been involved in grant making will recognise familiar situations.

Hindsight bias, Mr.Wray would say. He has recorded the catchy tune embedded above which summarises the different types of bias that can obstruct intelligent decision-making. You'll find a few extra ones that you may have come across at work, too.

Monday, 14 April 2014

Resources for facilitators

A friend has asked me about resources on facilitation of planning workshops. This makes me realise that my approach in facilitation is fed by many different streams. Some favourite resources in English:
  • One is the "Technology of Participation", developed over a period of about half a century by the Institute of Cultural Affairs. Apparently a recent ToP handbook has been published (late 2013 or early 2014).
  • Another favourite source: "Time to Think" by Nancy Kline. Maybe a bit wordy (pleasant for people who enjoy reading stories about the author's many clients), but definitely worthwhile.
  • On effective low-tech visualisation, "The back of the Napkin" by Dan Roam is full of lovely ideas.
  • Also nice, "Game Storming" by Dave Gray, Sunni Brown and James Macanufo and "Visual Meetings" by David Sibbet.
  • If you're into systems and complexity, "Systems Concepts in Action" by Bob Williams and Richard Hummelbrunner offers inspiration.
  • And then there is that fat compendium of facilitation methods alled "The Change Handbook" compiled by Peggy Holman, Tom Devane and Steven Cady. The descriptions are rather short, but if you don't look for step-by-step guidance, it is quite adequate.
I recommend you start with the first 3 items on the list. More time-tested facilitation tips are on the page "Planning, Strategies, Tools" on this blog. I particularly recommend the posting "Tips for Multi-Everything Facilitation". (Apologies for the messy layout of that page - this blog is an personal unpaid spare time initiative and I find too little time to make it all look neat and smart.)