Taking the guesswork out of aid

Esther Duflo explains in a TED talk how we can bring aid evaluation from the “middle ages” to the 21st century.

It is extraordinary how much resistance there is within development agencies to rigorous evaluation of development interventions.

11 comments on “Taking the guesswork out of aid”

  1. Not really when you consider human nature. So many interventions and assistance projects are b.s. and if you try calling the kettle black the holier than thou and, let’s face it, self-interested members of the aid community (not all, I know but a high percentage) who have dollars at stake will chafe and complain and whine as loudly as they can cause they might be out of a job.

    It’s sometimes hard to see what, if any, benefits aid actually brings to those whose money is being used to support it. Sure it feels nice not to have to hear about starvation or people suffering from disasters and to know you’ve helped alleviate someone’s problems, but how much impact does that have on one’s own well-being? That may sound cruel but that’s why people wonder where there money is going in tight economic times. And why too many aid agencies chafe at transparency.

    Of course, calling for transparency in recepient countries is quite another thing.

  2. “It is extraordinary how much resistance there is within development agencies to rigorous evaluation of development interventions.”

    That’s…wildly inaccurate. Do you know how much emphasis there is in the development sector on impact evaluation and monitoring? Aid and development agencies are audited just as rigorously as other private agencies.

  3. Rose

    I’m afraid it is not inaccurate, nor even wildly inaccurate.

    I sat on a panel on Wednesday accompanied by two speakers – one talking about Millennium Villages, the other talking about conditional cash transfers in Cambodia – both advocating their interventions on the basis of so-called evaluations based on comparing “before” and “after” data. This is not rigorous evaluation. (There is some rigorous evaluation of conditional cash transfers, but none of Millennium Villages).

    The aid industry – public and private – thinks that because it spends a lot of time and money on evaluations, that there a lot of “emphasis” on evaluation. But the quality of these evaluations is pitiful. We should do many fewer, much more rigorous evaluations.

    Owen

  4. We should do many fewer, much more rigorous evaluations.

    Agree. But not sure about the dismissal of evaluations based on before and after data. I understand the issue (spurious correlation) but very few interventions can be RCT’d (a health SWAp, for example). In those cases what are we do to other than (good) before and after studies?

  5. This is great stuff. You’ve got to applaud Esther Duflo and JPAL’s work and their commitment to using aid dollars better. Better informed policy must surely make a huge difference to achieving more and making it harder to divert funds to other agendas. Roll on the twentieth first century!

    The argument applies at every level. Not every project can be evaluated, or should be, as JPAL’s Rachel Glennerster said on Development Drums episode 21. RCTs are expensive. Good polices need to be well formulated – and they also need to be effectively implemented. So every project needs reliable monitoring. The same emphasis on careful measurement to test key assumptions, focused on operational priorities (rather than ultimate ‘impact’ questions or just performance compared to original plans) has a crucial role to play in day-to-day and month-to-month management. It can boost both results and accountability – specially when it uses feedback systems to stay responsive to what local people think.

    Bjorkman and Svennson’s paper “Power to the People” [http://people.su.se/~jsven/PtP_QJE.pdf] brilliantly brings these two sides of the measurement coin together. They used a randomised control trial to test the effect of community monitoring on primary health care provision in Uganda. Among other effects, it decreased under 5 child mortality by 33%. Impressive!

    I wonder if there’s a risk that ‘evaluation’ could sometimes dominate the ‘monitoring and evaluation’ space? It seems like there’s a lot to gain from improving measurement during implementation as well as design, driven by the same commitment to learn, improve and use every dollar as effectively as possible.

  6. I’m intellectually sympathetic to Duflo and her crew. However, JPAL & the mantra of “fewer better evaluations” have always struck me as the academic non-answer to field realities. In the field, my observation is that we actually don’t urgently need more research into narrow, other-things-being-equal, controllable questions that fewer but more rigorous true evaluations could address. There are many egregious project failures, which simpler methods could prevent or at least mitigate — if the fashion weren’t to dismiss everything that fails a (usually poorly-understood and abstract, or simply wrong) definition of the rigorous, RCT, randomized, quasi-experimental, ‘gold standard’.

    Wrt the TED talk, let’s also keep in mind that population health outcomes are beyond a doubt the easiest phenomenon to measure in the field of development. That is not to say they are easy to measure correctly and meaningfully for a given intervention, and it is not to take away from the excellent and valuable contribution of sound, affordable health M&E and evaluation to development in other sectors. (Humanitarian assistance is a substantively different challenge, unfortunately muddled with development in the above TED talk.)

    Yet we must honestly acknowledge that the complex systems in which behavior change and system interventions are embedded, which across all sectors constitute the overwhelming majority of development interventions, add multiple layers of challenges (intervening or confounding variables, if you prefer) to practical and ethical RCT design, and thus to valid, robust, generalizable findings.

    Far cheaper and far more useful for sustainable in-country capacity and engagement is strengthening transparent, shared M&E of each and every program and project. Not the misconception of M&E as project management — what did the project do — but substantive and challenging M&E orthogonal to standard management: What happened *as a result* of what the project did? Pushing this “So what?” question, using empirical evidence, as far as reasonably possible provides tailored, grounded, relevant facts that project managers can use to direct resources more efficiently, thus enhance impact, /while the project is ongoing/.

    To Alex Jacobs: ‘Evaluation’ definitely is drowning out the M&E space, at least in the US. And you’re right on the money (so to speak) that improving measurement incorporated in design and thoughtfully applied throughout implementation can yield knowledge that is not only immediately useful but when shared or further tested can improve methods and execution far beyond a single intervention.

    Nowadays, alas, we are living through proof that “a little knowledge is a dangerous thing”. As noted in comments above, contemporary rhetoric flows loudly with /emphasis/ on evaluation, yielding no more — and probably fewer — well-done evaluations being executed. At the same time, incalculable costs in lost opportunities and real damage pile up through the increasingly diminished attention and care to the nuts and bolts of valid and reliable field measurement, tied to a program’s theory of change and hypothesized results.

  7. Monitoring and Evaluation seems to have lost its way.

    While I applaud all efforts to improve evaluation and strengthen monitoring, I do think there is some confusing the nail for the hammer. Are development professionals using M&E tools to help strengthen their programs or is M&E the nail that’s supposed to hold a mix of good intentions and well thought out policy together?

    Implementation capabilities are needed to wield the hammer of Monitoring and Evaluation at the right time and for the right purpose. I think that’s the secret ingredient to making sure all the ‘hard’ stuff Ms. Duflo describes in her last 30 seconds happen.

  8. M&E correctly integrated into program design and implementation helps ensure an intervention is NOT just a “bundle of activities” but is instead operating according to coherent hypotheses and development expectations based on evidence and tailored to context (in other words, an empirically-grounded theory of change).

    Measuring incremental progress (or lack thereof) according to activities on the ground is cost-effective in any project or program, and enables critical course corrections through the life of the intervention to enhance overall results (impact). The biggest missed opportunity in learning more effective and meaningful development approaches is the proprietary attitude most implementers hold with respect to M&E methods, data, and nitty-gritty project feedback.

Leave a Reply

Your email address will not be published. Required fields are marked *

Published by

Owen Barder

Owen is Senior Fellow and Director for Europe at the Center for Global Development and a Visiting Professor in Practice at the London School of Economics. Owen was a civil servant for a quarter of a century, working in Number 10, the Treasury and the Department for International Development. Owen hosts the Development Drums podcast, and is the author Running for Fitness, the book and website. Owen is on Twitter and