Documenting qualitative comparative analysis and process tracing

I am delighted to announce that I have started a dedicated blog on Wolfgang Stuppert's and my work on a DFID-commissioned review of evaluations of interventions related to violence against women and girls. We will use qualitative comparative analysis (QCA) and process tracing to examine the evaluation approaches and methods used in up to 100 evaluations completed since 2008. 

Since QCA and process tracing are still relatively new in evaluation, we have decided to document our work on the new blog www.evawreview.de, which is supported by the UK Department for International Development (DFID) We will post the main steps of our work, every few weeks. Just like this blog here, it includes a "follow by e-mail" function, if you wish to be alerted when new posts come up.

Exciting new resource pack for social movements

BRIDGE has published a rich multi-media resource pack on gender and social movements. Find the full set on http://socialmovements.bridge.ids.ac.uk/, a beautifully designed, dedicated website. I particularly recommend the overview report and, for busy people, the brief bulletin that summarises key points.

WANTED: evaluation reports and contacts

The UK Department for International Development (DFID) has commissioned a review of evaluation approaches and methods for violence against women and girls (VAWG) interventions in development and humanitarian contexts ('the Review').  The Review will provide recommendations as to how evaluations in this complex field can be designed and implemented to yield maximum use. It will use qualitative comparative analysis and process tracing to detect the drivers of effective evaluations of VAWG interventions, and describe good practice. Review findings will be widely disseminated to stakeholders and made publically available.

The Review team, led by international consultants Michaela Raab and Wolf Stuppert, plans to analyse up to 100 evaluations. Since relatively few evaluations have been published, we need your support in identifying and sharing those that you are aware of – including unpublished evaluations. The evaluation reports will be read by the Review team only, so please do not hesitate to share evaluations even if you have doubts about their quality or about their findings. Findings will be presented in an aggregated form that will prevent readers from identifying specific evaluations.

If you have access to evaluation reports that meet the criteria outlined below, we would be grateful if you could forward them to this address. The criteria are:

• Published and unpublished evaluations of development or humanitarian programmes that (i) either focus on preventing or responding to any form of violence against women and girls (VAWG), (ii) or include a VAWG component as part of a wider initiative (for example, on human rights, humanitarian protection, peace building, post-conflict reconstruction or HIV/AIDS)
• Evaluations that have been completed between January 2008 and October 2013
• Report in English.

Furthermore, we would be grateful for your identifying persons the review team could interview to find out about their experience with VAWG-related evaluations. You are kindly requested to provide the name and contact details (e-mail address and/ or phone number) of

• Evaluation commissioners (for example, programme officers who have commissioned VAWG-related evaluations)
• Evaluators (both external consultants and internal evaluators who have carried out VAWG-related evaluations
• Leading members of VAWG-related initiatives or organisations 

The Review report will individually credit everyone who contributes evaluation reports and data (unless you wish not to be mentioned). We would be grateful if you could send us your response by 20 October. In addition, you are encouraged to forward this message to others who may have access to VAWG-related evaluations and professionals in the field. 

We are looking forward to learning from the evaluations you will share!

Update on OECD/ DAC Evaluation standards and advice

The OECD Development Assistance Committee appears to have moved part of its evaluation pages to new sites. That means that some links in older OECD/DAC-related posts here may be broken - my apologies! Some day I will get help to update all links since 2008, when I started blogging. In the meantime, here are a few important links that work:

Avis aux francophones - good news for French readers

Le guide Oxfam « Eradiquer les violences faites aux femmes » vient de paraître en version française. Vous le trouvez, en version PDF, sur le site d’Oxfam international que voici ; la version anglaise est disponible là. Si vous avez une connection un peu lente, vous pouvez aussi regarder la version anglaise directement sur ce site. Bonne lecture !

Conducting qualitative interviews

I am working on a publication on qualitative interviews – but it’ll be in German and it’ll take ages to complete because it is one of those spare-time not-for-profit projects. Since a friend is about to embark on a round of interviews we have designed together, I have decided to summarise some bits in English. Here they are! 

Evaluating peace building activites

Just in case this has escaped your attention: Last year the OECD Development Assistance Committee published comprehensive guidelines on Evaluating Peacebuilding Activities. Click on the title to download the document, or else read it on-line here: Evaluating Peacebuilding.

Politics of Evidence - an abstract

I have submitted a conference report on the "Politics of Evidence" Conference in Brighton (April 2013) to a German evaluation journal. Check this blog for an announcement of its publication date! Meanwhile, here's an abstract in English:
More than 100 specialists participated in the ‘Politics of Evidence’ conference at the Institute of Development Studies in April 2013. The main theme was the growing tendency to tie funding for international development and transformative initiatives to the use of pre-defined instruments (‘evidence artifacts’) for effectiveness and impact assessment – and how to deal with this trend.
The participants were in favour of well-informed planning and robust monitoring as a source of information for effective project steering, and as way to expand the knowledge base. However, when the use of a narrow spectrum of specific instruments becomes an obligation, distortion and loss of effectiveness may occur. A key conclusion was the call for rigorous relevance and relevant rigour, i.e.: the main criterion for the choice of instruments for project monitoring should be the extent to which they generate practical use.
---
Here is the link to the conference and to the papers delivered by Rosalind Eyben and Brendan Whitty respectively. (Click on the names to get to the paper.) Additional resources are available on the Politics of Evidence site under "Resources".
By the way, the notes of a somewhat smaller but equally interesting event at the Institute of Development Studies (IDS) in Brighton are also available on the web now. The Centre for Development Impact has hosted an "Impact Innovation and Learning Event: Towards a Research and Practice Agenda for the Future". Click on the title to get to their web-site, which includes the keynote speakers' power point presentations as well as a list of 18 action points. I also recommend the site of the IDS Centre for Development Impact, a fairly new group that brings together academic research and evaluation practice. To be continued...

Recruitment by Quiz?

An international agency has launched a new web-based "experts' roster". The organisation has an important mission and I have the kind of expertise they look for, so it seems natural to register.
The opening page of the roster looks friendly and clear. I complete a few text boxes with basic data on my life and work.
The following step is tedious -

A rough guide to building monitoring systems

Today, I wrote up a few lines for a colleague to summarise the key steps of building monitoring systems. For the case the same question pops up again, I have saved my response here:

Indicators and data collection instruments are just one aspect of a good monitoring system. Most importantly, early in (or ideally before) the actual project, people should sit down and think: 

Practical tips: note-taking during interviews

Taking notes in "qualitative" interviews, for example at the occasion of evaluations, seems a fairly simple and straightforward thing to do. Over the years, I have discovered that people take notes in many different ways. The way that works best for me is demanding on the interviewer, but quite efficient: no/ little need for extra transcription time after the interview!

Free webinars on evaluation

...now available on the YouTube Channel of the American Evaluation Association.
The webinars are in the first row of the page that opens when you click on "YouTube Channel" above. The second row is filled with other interesting videos, such as Julia Coffman's speech on evaluator's mistakes and the importance of sharing them with others so that they don't repeat them. Enjoy!

What men can do to end gender violence

An impressive TED talk, not only for beginners - click HERE to see it. Jackson Katz encourages "bystanders" to stop "laughing along or pretending not to hear it". The speech is not totally respectful of feminist organisations (shame on you if you're more interested in the video now ;-) but it is definitely worth watching.

Quotes from the "Politics of Evidence" Conference

In late April, the "Politics of Evidence" conference brought together (at the IDS in Brighton) development and evaluation practitioners from a rich mix of professional backgrounds and countries. Some of the papers presented have been available on the internet, such as Rosalynd Eyben's article on the The power of results and evidence artefacts. Eyben's paper is about tools initially designed to stimulate critical thinking which have mutated into instruments of confusion: For example, the log-frame was built to encourage people to reflect on the assumptions behind their "project logic" - but in most contemporary versions, the "assumptions" column has disappeared.

Evidence artefacts - an example

This is a case study prepared for the "Big Push Forward" conference next week (for links to the conference, see the extra post below). 

The example is from a real organisation, a group working on human rights.