I recently traveled to Bangkok with two MIF colleagues to attend the second annual Impact and Policy Conference, sponsored by the Asian Development Bank, Innovations for Poverty Action (IPA), the Citi Foundation and the Abdul Latif Jameel Poverty Action Lab (J-PAL). The conference brought together over 200 researchers, policy makers, and international development specialists to go over the most recent impact findings on improving programs and policies for the poor. It was clear from the diverse list of participants, that donors and policy makers alike are actively seeking evidence with which to assess welfare outcomes on beneficiaries. There was consensus that financial outcomes are merely intermediary effects, and that what is ultimately sought are increases in things like consumption, health, asset building, housing investment, income smoothing, ability to withstand shocks, etc. Given these goals, research into what products and mechanisms lead to these outcomes is very important to ensure that resources go into programs that can deliver the desired impacts.
(A small aside about impact evaluation: Its use is part of a wider discussion on the need to make evidence-based decisions when using public funding for development outcomes. Impact evaluation is different from other evaluations in that it compares populations that benefitted from a particular program with similar populations that did not have the program (control groups or comparison groups), thereby effectively isolating the effects of the program and convincingly demonstrating causality.)
I was most eager to hear the panel on the Impact of Microcredit, since the MIF has been a key player in the evolution of that industry in Latin America and the Caribbean. In fact, my MIF colleague Claudia Gutierrez recently put together a literature review on the existing impact evaluations in the LAC region and found substantial gaps in research and knowledge. Yale economics professor Dean Karlan (founder of IPA and author of the recent book “More than Just Good Intentions) started out this discussion with brand new findings from his randomized control trial of Compartamos. During a panel at last year’s FOROMIC, held while Karlan’s evaluation was still underway, Carlos Labarthe, one of Compartamos’ founders, spoke honestly about how difficult it can be for an operating, for-profit company to undertake an experiment without disrupting ongoing operations. Karlan is not ready yet to publish his paper, but indicated that that the findings showed some modest results. There was a measurable impact on formal credit use as Compartamos expanded to new regions. Clients seemed to use loans in a mix of business investment and debt management. Although revenues of borrowers increased, so did expenses, resulting in no net growth in profits. On average, there was no noticeable pattern of growth, rather the effects seems to be dispersed differently among clients. Interestingly, the best impact he identified was related to subjective wellbeing and female empowerment.
Britta Augsburg of the Institute for Fiscal Studies attempted to address the question of the relative impacts on borrowers of individual versus group lending with two studies, one in Bosnia and one in rural Mongolia. In general, her studies had similar findings to Karlan’s. The poverty alleviation effects were limited and heterogeneous. There were two findings that I found surprising and somewhat counter-intuitive. In Mongolia, her data showed that group borrowers were less likely to make informal transfers to families and friends, implying that the group liability might enforce the use of proceeds for business investment, rather than diversion for other uses. Her study in Bosnia discovered a slight reduction in teenage school attendance, as microcredit borrowers invested more in their business growth, thereby requiring more labor from teens in the business.
There were a variety of other panels on business training programs, credit officer incentives, transnational household financial management behaviors, self-help groups in India, village savings and loans in Africa, and a variety of other interventions. After recovering from jet lag, I can report the following take-aways from the conference:
- Impact evaluation and evidence based programming are here to stay in the development community - both as a product of accountability to donors, and as a tool to shape interventions so as to maximize their impact and understand their effects on beneficiaries.
- The debate over randomized control trials versus other forms of evaluation has been moderated, as both have proven their value. RCTs, when done well, show definitive causality and are often used to perfect elements of an intervention, by “tweaking” the treatment or intervention and assessing these against a control group. Other kinds of assessments are important to “tell the story,” assess the model, or gain an understanding of how or why certain interventions work.
- Practitioners, policy makers and funders need to be part of the conversation to shape the research questions and decide which interventions are most worth expending efforts on. As Alexia Latourtue from CGAP commented during her panel, “If we can’t agree on impact, we are in trouble.”