A rough guide to building monitoring systems

Today, I wrote up a few lines for a colleague to summarise the key steps of building monitoring systems. For the case the same question pops up again, I have saved my response here:

Indicators and data collection instruments are just one aspect of a good monitoring system. Most importantly, early in (or ideally before) the actual project, people should sit down and think: 

• What is the information / are the types of data that we need most to make sure we implement our project as well as we can with the resources we have mobilised/ can mobilise?
• What is the information that we must share with project participants/ beneficiaries, so that they can participate fully and be empowered through the project?
• What ist the information our donors need from us, at what moments?

When these questions are answered, the logical next step is to determine:

• Where can we find the information?
• Who gathers it, at what intervals? To maximise capacity building, I would encourage a maximum of participatory monitoring, i.e. project participants/ "beneficiaries" check whether the project progresses as planned and what its results are. One can  even gather robust statistics in a highly participatory manner - Jeremy Holland has just published a book on the topic.
• How will we do it? Ideally, there should be some counting ("quantitative" data collection and analysis, using well-defined indicators) and some narrating ("qualitative" data collection and analysis, which looks more into the specific people and situations you work with). 
• How will the information be discussed, stored and shared with others? (Also called "knowledge management".)

When all that is reasonably clear, it is time to make a concrete monitoring plan, for example with the use of a calendar that covers the full project duration. It should show:

• Who collects what information, how and at what moments?
• What are the routine moments where information is shared and discussed within the project (for example, monthly monitoring meetings among key staff; quarterly feed-back meetings with representatives of project participants/ "beneficiaries")?
• Which specific events which warrant specific monitoring, for example major campaign events that could be discussed in "after action reviews" to make sure fresh impressions are recorded?
• How knowledge generated in these discussions will be shared (a report? a blog post?)
• How knowledge will be shared with people outside the project ?
• Who makes sure this monitoring plan is implemented?

The last bullet point is important - monitoring systems need to be monitored, too.

Practical tips: note-taking during interviews

Taking notes in "qualitative" interviews, for example at the occasion of evaluations, seems a fairly simple and straightforward thing to do. Over the years, I have discovered that people take notes in many different ways. The way that works best for me is demanding on the interviewer, but quite efficient: no/ little need for extra transcription time after the interview!
A principle: Try to capture everything the person says, in the original wording, so as to avoid distortion. In an ideal world, you would audio-record the conversation and transcribe every single word later - a process likely to take 2-3 times longer than the actual interview. Unfortunately, the time-frames and budgets for most evaluations do not allow such a luxurious use of time.

My solution is touch-typing. There are plenty of touch-typing tutorials - including free ones, such as TIPP10 (click on the name to reach the TIPP10 site). Once you touch-type at a decent speed, you can transcribe the interviewee's answers while you conduct the interview, and maintain eye contact most of the time!

Meanwhile, regardless of the way in which you record the conversation (does anyone still know anything about stenography?), it helps to remember a few key things:

• Write down as much as you can, including repetitions - they'll show, when you'll analyse your data, where the interviewee places importance. If you can't write as fast as the interviewee speaks, avoid asking her to wait until you are ready - that may block the conversation flow. For the case you need to verify something after the interview, make an audio recording of the conversation.
• Write down what the interviewee says, in the words she/ he uses. In your notes, avoid paraphrasing or summarising what she/ he says - summarising what other people say means that you filter their words through your own mind, which may create distortion. (But it may of course be useful, in your actual conversation with the interviewee, to paraphrase what they say in a probing question, so as to increase chances you understand what they mean.)
• Don't worry about orthography, typing mistakes, formatting - you can run a "spell-check" after the interview, if you need to share it with anyone
• Just make sure you separate your questions (or any remarks you may find time to jot down) clearly from what the interviewee has said.  
• When you ask (probing) questions, make sure you jot them down as well - but mark them as your questions, so that you recognise them as such when you read your notes.
• If you lead a group discussion, write the name of the person next to what she says.

This is an open-ended list - I am currently working on a guide for qualitative interviews (in German, for the time being). Suggestions are welcome! (Comments box or message to me :-)

Pursuing Elusive Justice

Vahida Nainar and Saumya Uma have engaged in advocacy for bringing standards set in the International Criminal Court (ICC) statute to the criminal laws in India through the ICC-India campaign. They have published a collection of essays that examine, in the light of international and domestic law, impunity in mass crimes in India - torture, enforced disappearences, sexual and gender-based violence. The book is called "Pursuing Elusive Justice - Mass Crimes in India and International Standards", a brief description is available here. The price is in Indian rupees!

Free webinars on evaluation

...now available on the YouTube Channel of the American Evaluation Association.
The webinars are in the first row of the page that opens when you click on "YouTube Channel" above. The second row is filled with other interesting videos, such as Julia Coffman's speech on evaluator's mistakes and the importance of sharing them with others so that they don't repeat them. Enjoy!

What men can do to end gender violence

An impressive TED talk, not only for beginners - click HERE to see it. Jackson Katz encourages "bystanders" to stop "laughing along or pretending not to hear it". The speech is not totally respectful of feminist organisations (shame on you if you're more interested in the video now ;-) but it is definitely worth watching.

Quotes from the "Politics of Evidence" Conference

In late April, the "Politics of Evidence" conference brought together (at the IDS in Brighton) development and evaluation practitioners from a rich mix of professional backgrounds and countries. Some of the papers presented have been available on the internet, such as Rosalynd Eyben's article on the The power of results and evidence artefacts. Eyben's paper is about tools initially designed to stimulate critical thinking which have mutated into instruments of confusion: For example, the log-frame was built to encourage people to reflect on the assumptions behind their "project logic" - but in most contemporary versions, the "assumptions" column has disappeared.
The push to demonstrate quick results may encourage development agents to privilege actions which yield easily achievable results - over more complex work that aims for social transformation.
A few stories and "soundbites" from the conference (no attribution - Chatham House rules):

• The move from a "trust me" to a "show me" culture may not be entirely unjustified. But: External validity is often sacrificed on the altar of internal validity - i.e., people sometimes feel pushed to conduct "super-rigorous" research even if it is not particularly interesting or useful for their work.
• One participant described how several external consultants, in successive rounds, modified "her" project to match (perceived) donor expectations as to what was a proper proposal - "suddenly the whole project was a complete stranger to our themes and our organisation". I would suspect that was not the donor's intention. But in a situation of unequal power, organisations at the receiving end of development funding may be tempted to bend their projects to what they consider to be donor preferences.
• A participant with extensive experience in evaluations using randomised controlled trials (RCT) said that RCTs worked well in certain situations: for example, to find out whether people would use mosquito nets more systematically if they paid for them (rather than getting them for free). But scientific institutes that use experimental designs turn down requests for RCT-based when the questions are complex - for example, whether the availability of toilets for girls at schools improves girls' educational achievement. That participant stated that in most cases (97% of those his institute was approached with), RCTs were inappropriate and systematic monitoring would be the most effective way to assess impact. Utility (what development practitioners need most) vs. accuracy (what academic researchers need most).  
• The positive effects of evidence artefacts depend on the user - not on the artefact. Anyone who genuinely wishes to create opportunities for critical reflection and discussion will find a way to do that, regardless of the specific tool they may use. I would say, though, that some tools are more conducive to promoting reflection than others. "Don't start with the tools - start with the thinking!"

About one day at the (2-day) conference was devoted to experience exchange in small groups. The conclusions the groups came up with similar conclusions:

• A constructive way to deal with the "politics of evidence" is to create systems that foster critical reflection - in the way one's own organisation or programme needs it. For example, one participant explained his (large) organisation measured "outputs" only - but each of its initiative was well documented as part of a "jolly good knowledge management system", which could easily respond to queries by donors or other stakeholders.
• What is happening right now: scientists are brought into the discussion to reveal that there is more to science than just experimental designs (such as RCT), and to show the limits and risks (ethical issues, inter alia) of experiments.
• Some "results artefacts" tend to be more adaptive and more likely to answer "how" and "why" questions than others. Participatory action research or the "action learning cycle" were quoted as efective ways of gathering, analysing and disseminating key information.
• Creative compliance, rigorous relevance, relevant rigour - watch this space for more on that!

Evidence artefacts - an example

This is a "case study" prepared for the "Big Push Forward" conference next week (for links to the conference, see the extra post below). 

The example is from a real organisation, a group working on human rights. 

The document which defined the organisation’s 3-year programme described the types of work it would carry out, within its mission and values.

The appraisal document and contract which governed the funding relationship between the human rights group and the funding organisation (donor) included a list of intended results (“results list”). All results came with a form of quantification. Many were very indirectly connected to the advocacy and legal aid carried out by the human rights group. For example (from memory – inaccurate, but close to the original):

-          At least 10,000 persons (30% female) will have basic knowledge of their civil rights.

-          The percentage of rape cases brought to court will increase to at least 30%.

-          No more extrajudicial killings will happen.

When I asked the human rights group as to how they recorded progress against those targets, defined upon the donor’s request, it took days until their copy of the “results list” was located. Annual reports to the donors did not refer to that list. Instead, they contained dozens of pages of log-frame like matrices, which presented every activity as a mini-results chain. For example (from memory i.e. inaccurate):

• Activity: 200 awareness raising sessions
• Output:  212 awareness raising sessions conducted
• Outcome:     12,034 participants (31.4% female)
• Objective:    80% have basic knowledge on civil rights
• Goal:            Rights awareness has increased

When I asked why “they” sliced up their experience in that way, I was told that a different donor had asked them “to do results-based management”. Yet another organisation - a third donor - had dispatched a consultant for results-based management (RBM) training, which focused on that particular type of results chain. The course materials I saw were all about the correct use of terminology – for example, the difference between an "outcome" and an "objective", according to that particular agency (or that particular consultant). In those days, every single donor seemed to develop their own results chain.

All senior and middle managers of the human rights group had to participate in the week-long (5 full days) training course – a huge transaction cost for an organisation where the managers were involved in urgent human rights work every day. The only visible results of the course were annual progress reports full of matrices which I found hard to understand. The English was excellent, though. 

Independently from any donor intervention, the organisation ran its own, parallel knowledge management system around monthly meetings, where all professionals of that group came together for a full day to discuss external and internal developments. That is where the group made sure that (i) learning was shared, and (ii) plans would be made on the basis of experience and… evidence the way they saw it. The reports produced for the donors did not play any role in those meetings.

Apparently, no donor had ever asked the group about its own ways of planning and monitoring – let alone tried to organise reporting so that it would build on (or further improve) the existing system, which appeared quite well suited to the group's needs.

Coming soon: Critique of results and evidence artefacts

Next week I'll attend the Big Push Forward conference at the Institute of Development Studies (IDS) in Brighton. For those who can't make it there, a few interesting papers are already available on the web (for example Carrots and sticks: Results and evidence artefacts and their effects | The Big Push Forward).
The final plenary session will be "live streamed" on 24 April at 15:45 (3:45 pm) UK time, under this LINK. Watch this space for post-conference musings in mid-May.

Beware of bias - a song

For evaluators and other people who collect data, it is always good to be aware of the risks of cognitive bias distorting our observations - presented beautifully in this video!

Ressources en français - formation sur le genre

(English text below)
En 2012 j'ai eu le privilège d'animer une formation genre pour des professionels du développement en Afrique subsaharéenne. Mes clients m'ont gracieusement autorisée à partager les fiches d'information développées à l'occasion. Une liste de littérature fait partie de ce kit, que vous trouvez ici - pourvu que SlideShare fonctionne!: 
Last year I facilitated a gender training workshop with development professionals in Subsaharean Africa. The clients kindly authorised me to share the hand-outs I developed for the workshop. Full references are included in the kit published on SlideShare (see above). I hope this works!