Wednesday, 23 September 2015

My data

A couple of days ago a colleague working on an interesting new e-learning tool invited me to test an initial, yet unofficial version of that tool. I clicked on the link they had sent to me. A screen appeared which asked me for my full name, my e-mail address and my company. Every single field was mandatory, that is, I could not move to the subsequent screen without providing my name, my e-mail address and a company name.

That is a threshold. When you open a book or a newspaper, no-one asks you to send your name, your e-mail address or other personal data. You open the thing and you read it. The publisher can track the number of sold books - to some extent - the places where they have been sold, and that's it. Has anyone ever complained about that?

Wednesday, 16 September 2015

Interesting debate on evaluating human rights work

Who is evaluation of human rights work for? How about "strategic plausibility" as an evaluation criterion? How do we measure success when protecting civilians in conflict? These are the kinds of questions discussed in this web debate on evaluating human rights work. Very commendable!

Workshops that work: Six essential tips for facilitators

It is delightful to get plenty of positive feed-back on the workshops I design and/or facilitate. A few weeks ago one participant even described the workshop I facilitated as a "once in a lifetime experience"! Since I would love all workshops people attend to be useful, I have started asking participants to tell me what exactly they like about "my" workshops, so that I can share it here. Some of the points below have been made in earlier posts on this blog, others have come up in recent conversations.
  1. A good workshop starts with careful preparation. Spend at least as many days on preparing the workshop as you will spend in the workshop. Key points here: (a) Make sure the purpose of the workshop is clear and known to everyone involved: If there is no explicit purpose, don't waste people's time - scrap the workshop! (b) Get to know all participants ahead of time. A short web-based survey (keep it really short - a handful of questions!) is a good opportunity to introduce yourself to the participants and find out what they know about the topics to be addressed. (c) Encourage the organisers to invite the persons who really want and need to be there, to send out the invitations in good time, and to book a venue that is easy to reach and a pleasant (but not overly luxurious) environment for working together. Avoid basement meeting rooms! Daylight is important.
  2. Workshops are about people interacting with each other - not about funnelling ready-made packages of information into other people's minds. If people can get all the necessary information from reading a couple of articles, send them the articles, don't ask them to sit through boring presentations. Instead of bombarding the participants with dozens of power point slides filled with text, mobilise participants' own knowledge by asking them how they deal with the issues the workshop is about. "Buzz groups" of 4-6 persons discussing topics and then sharing them with the plenary are a good way of doing this. You can always provide extra information near the end of the discussion, if an important point is missing. But you'll find that in many cases, the knowledge shared by the participants in the room surpasses the facilitators'. I love using aspects the Technology of Participation (ToP) developed by the Institute for Cultural Affairs (ICA) for a quick succession of individual reflection, small group work and plenary discussion. (By the way, several countries boast their own ICA who provide ToP training and certification; there are also a couple of books about the methodology.)
  3. That is also why it is so important people have time to get to know each other. Use friendly, inclusive, non-competitive ways of introducing all participants to each other at the beginning of the workshop. Arrange the room in a way so that everyone can see everyone's face. "World Café" style seating is great for that - just keep reminding people that they can move their chairs around to look at others, for instance when someone behind them asks a question. Also, encourage them to sit next to participants they don't know well. Between sessions and after breaks, initiate short games to help participants memorise each other's names. By the end of the first day, you should be able to address every participant by their name. (In large groups - say, more than 30 - name tags may be necessary. In smaller groups I'd rather avoid them, or use them only on the first day.)
  4. Facilitation means making things easier. Your job is to help participants feel good and work productively. I believe that one can't be productive if one feels bad. Adapt your facilitation to the participants' needs and interests, also during the actual workshop if needed. If you are not sure what they want and need, ask them.
  5. Keep your poise. There is no reason to feel stupid or offended if people don't understand what you ask them to do or don't want to do it. Find out from the participants what it is that needs adjusting or more explaining, and adjust and explain.
  6. Involve the participants in monitoring the workshop. In multi-day workshops, I distribute short evaluation forms at the end of every day, asking participants to rate the quality of the participation in the workshop and the usefulness of every single session (multiple-choice questions with scales). In addition, I also like asking one or two participants per day to be "participant observers" who (a) can give me quick feed-back during breaks if they feel something is wrong in the workshop dynamics (e.g. if the facilitator has not noticed that certain people don't participate sufficiently), and (b) summarise the workshop process to the plenary at the end of the day, with recommendations as to what should be done differently on the following day.
For more tips, I recommend my earlier posts "Tips for multi-everything facilitation" and "The learning contract" (which is more about my broader philosophy of learning)

Monday, 14 September 2015

10 things to know about evaluations

The UK Department for International Development (DFID) has produced this wonderful short guide for everyone who uses evaluations. Have a look at it and spread it around! It'll make evaluations more useful. The guide also refers to the Better Evaluation site, also highly commendable for evaluators and everybode interested in the topic. 

Thursday, 10 September 2015

Emerging Evaluators meeting virtually and in real life

My occasional associate Wolfgang Stuppert is part of the Emerging Evaluators' Network of the European Evaluation Society (EES). He has invited me to broadcast this invitation to the First Virtual Conference for Emerging Evaluators. Here it is:

On 19 September 2015, the First Virtual Conference for Emerging Evaluators will take place. On that day, from 3 pm to 8 pm [Berlin time, I presume], more than 100 emerging evaluators will gather on-line to discuss the bright and not-so-bright sides of their profession.

Tuesday, 25 August 2015

What is a sound theory of change?

The term "theory of change" (ToC) has established itself in the development world. Agencies invite consultants (including your blogger) to facilitate workshops which would help them develop the theory of change for a particular programme; donors ask prospective grantees to come with a sound theory of change; evaluators (including your blogger) bemoan the absence thereof. There are companies which have developed theory of change software and dedicated websites that propose to help you build your own ToC. The picture below shows a fraction of what I get when I ask a popular search engine to find images of "theory of change": 

Monday, 20 July 2015

Value for money in training?

In an evaluation of a large initiative designed to help changing social norms on gender-based violence (GBV), I found out that each of the many different organisations involved carried out training workshops. The training participants were mainly for police, staff in health services, religious leaders, social workers and other people who tended to spend no or very little time at universities or other academic venues. That is, the trainees were people who were not used to sitting and listening attentively to complicated presentations. 

Yet, most of the training workshops were organised the "academic" way: The trainer would take the audience, seated in neat rows, through a more or less lengthy set of "power point" slides with plenty of text. Often, the content of the slides was highly theoretical, presenting definitions with plenty of terms people would never use in everyday language. The audience would sit and listen and ask the occasional question, if they dared to. People don't want to look stupid - so if a presenter uses lots of big words the audience are not familiar with, changes are that there won't be many questions.

Arguably, that type of training is a waste of time and money. There is a large and growing body of experience on adult education - and education in general - which shows that one of the most effective ways to acquire new knowledge and skills is learning by doing, by solving problems.

Tuesday, 26 May 2015

Too quick, too dirty

Today I came across the (virtual) file of a "quick and dirty" evaluation I carried out a while ago. That made me feel a bit queasy - because I should have turned down that offer, or advised my clients to go for something radically different. Quick and dirty can be quite wasteful.

Someone had approached me for the job at very short notice - the expectation was that the consultant would start travelling within a couple of weeks. I happened to have time - some other assignment had been postponed - and I found the subject matter interesting. That's why I accepted the job. I was worried about the terms of reference, though. They came with plenty of specific ideas as to how the evaluation would have to be carried out - including requirements to visit three countries, to conduct a survey and to complete the job within some 30 person-days spread across two months. Within that time frame, the relevance, effectiveness, efficiency and sustainability of a project covering several rural areas across three countries was to be assessed.  Oh, impact, too, but I talked them out of that.

Saturday, 21 March 2015

Rape is not about sex

Every so often, I facilitate gender training. One of my favourite ways to get discussions started is a quiz, inspired by the 1994 classic, The Oxfam Gender Training Manual (Suzanne Williams with Janet Seed and Adelina Mwau). I read a set of statements - for instance, recent research findings related to the field the participants work in - about sex and gender. For each statement, the participants are asked to determine whether it is about sex (as in biologically male, female or a bit of both - not sexual activity) or gender (roughly speaking, the behaviour societies expect from people according to their sex).

The discussions are always interesting. In a recent quiz of that type, I read the statement:  "A survey in Botswana revealed that 67% of school girls interviewed had been subjected to sexual harassment by teachers; 10% had consented to sex for fear of reprisals." (That was about a decade ago.)
One participant argued the statement was about sex.

Tuesday, 24 February 2015

Mystery moaners

Today a message arrived in my inbox from an address called DevBalls, with links to a blog called "DevBalls - Exposing the absurdity of the aid industry". The mystery author must be someone who knows people who know my e-mail address, but at some distance - because apparently I have missed out on a whole year of DevBalls. The spring and autumn 2014 issues can be downloaded from that blog.

Everything you find there is harsh criticism of UK government aid policy and its implementation. The style is rather aggressive, but the questions the authors raise are pertinent: for example,

Thursday, 29 January 2015

A written survey with people who don't read and write

Last year - ah, no, in 2013 - my colleague Wolfgang Stuppert and I carried out an evaluation of services for survivors of violence against women and girls in Mozambique. We felt it was important to gather feedback from many women and girls who used the services. We had only little time in Mozambique and no resources to train enumerators who would interview large numbers of service users.

We decided to organise a written survey. But some service users, we were told, could not read and write well enough.

Monday, 12 January 2015

Why Ukraine?

A reader has asked whether I have found out why this blog gets many hits from Ukraine, where I know few people. His blog records many hits from Ukraine, too.
I have no idea!

Could it be that people who experience difficulties accessing websites with such words as "human rights" hide behind a Ukrainian identity to escape censorship in their countries? Or is Ukraine where all those people are located who keep trying to post comments with dubious links?

Will post soon: written surveys with respondents who don't read and write.

Thursday, 25 December 2014

Moral judgment

It is somewhat late in the year to share observations from the 11th Biennial Conference of the European Evaluation Society which took place in early September. Apologies! This year has been a bit breathless for me - more on that in a different, future post.

Meanwhile, one particularly gripping topic at the EES conference was moral judgment, and the question whether evaluators should exercise it even if the evaluation terms of reference were only about, say, value for money. As one speaker put it, “you’re going to crash into [ethics] in the course of your normal trafficking around as an evaluator”. Not only because you need to do your best to avoid hurting anyone’s rights or anyone’s safety, but also because you are bound to bump into things that are bad, or that you might find bad.

Examples abound.

Saturday, 15 November 2014

Nice movies on coffee and gender in Uganda

I have promised posts about the recent European Evaluation Society conference in Dublin. Apologies that'll have to wait for a few weeks! I have been too busy preparing for my next assignment: a multi-country evaluation of a project that promotes gender equity and value chain development. A couple of beautiful short videos illustrate the approach, in two parts: Coffee Value Chain Uganda, Part 1, and part 2. Very commendable! Versions in French and Portuguese are available as well.

Saturday, 18 October 2014

Best Poster Award at EES Conference

Our poster "Paths to Effective Evaluation" won the Best Poster Award at the 11th biennial conference of the European Evaluation Society in Dublin. We - co-researcher Wolfgang Stuppert and I - are delighted, because we did put in much effort to find ways to translate complicated findings from Qualitative Comparative Analysis into something comprehensible to an audience of development practitioners. Find linkes to the poster, as well as to the full documentation of the research, on our dedicated blog