Technical shortcomings with IMF analysis

There has been quite a bit of interest today in two new studies by Raghuram Rajan and Arvind Subramanian from the IMF. I apologise for this rather technical posting, but it is important to understand that, while the authors are eminent economists, the econometrics underlying these particular papers is seriously flawed. Correctly specified, the model in fact produces the same correlation between aid and growth that countless other studies have found.

There was a press report about the study in today’s Financial Times, and also on Newsnight tonight. It has also been picked up elsewhere in the blogosphere. The first paper, Aid and Growth: What Does the Cross-Country Evidence Really Show?, challenges evidence of a link between aid and economic growth. The second, What Undermines Aid’s Impact on Growth, argues that the benefits of aid may be offset by an appreciation of the exchange rate.

The problems with the first paper are to do with the way the econometrics have been specified. The criticisms that follow are derived entirely from expertise of my colleagues – especially Michael Clemens. (Any errors are my own.)

The authors’ own results show that a number of small and sensible changes to the way they have specified their model would change their results. Each change would, by itself, only lead to a small and possibly statistically insignificant increase in the correlation between aid and growth; but if they are taken together, as they should be, the result would get a large and statistically significant positive relationship, as found in all the other studies of this topic.

Here are three examples of how the authors have missspecified the model and ignored the evidence in their own results. First, when they regress growth on aid, assuming a linear relationship between the two, they get a zero or negative relationship. They do this several times. But every growth model since the late 1950s and nearly every aid-growth study since 1995 has recognised that aid would have a nonlinear impact: that is, there would be diminishing returns to aid. If you don’t allow for this effect explicitly in your model, then an upside-down U-shape can look like a flat line. When Rajan and Subramanian allow for this (in part of one table): not surprisingly, the aid effect becomes positive in all cases and comes close to being statistically significant in many cases. Clearly, a non-linear specification is needed: but Rajan and Subramanian take no notice of this whatsoever. Second, much aid is either not aimed at growth (e.g. humanitarian) or is not expected to cause growth on the medium term, not even a few decades (e.g. aid to support democracy). When Rajan and Subramanian eliminate these types of aid from their measure and focus on "economic sector" aid: not surprisingly, the aid coefficient is positive and statistically signficant (though sensitive to the instruments chosen, as all instrumented variables are). Again, this is to be theoretically expected, but Rajan and Subramanian do not adjust their model to take account of it. Third, theory leads us to expect that it is absolutely crucial which other variables you control for, since many country characteristics are correlated with both aid and growth, and could lead to spurious negative correlations between those two variables if they aren’t accounted for. Rajan and Subramanian don’t meaningfully test the sensitivity of their results to which other variables are controlled for, they leave out variables which have been demonstrated in the literature to be correlated with both aid and growth (e.g. civil warfare and the occurrence of natural disasters) and wrongly include other variables (e.g. they control for government spending — presumably the very channel through which much aid acts!). Now while each of these effects is small by itself, if they were to make all of these changes simultaneously, as their own tables make clear, the effect of aid would be positive and significant. All of the changes are justified by theory and by 30 years of literature (to which they scarcely refer.) Just to be clear: Rajan and Subramanian’s own tables show that a well specified regression specification would reveal aid’s positive impact on growth. They would get results similar to those reported here which find a strong, positive, and statistically robust correlation.

There are other, even more technical problems about their econometric strategy. In particualr, they do not make a clear case that their instruments capture a large fraction of the variance in aid, i.e. that their instruments have "power", which is absolutely essential if one is going to claim that the coefficient on an instrumented variable is indeed zero.

Rajan and Subramanian second paper seeks to explain their findings by reference to the macroeconomic effects of aid flows, and in particular the "crowding out" effect of Dutch Disease. The point is that an increase in aid flows can cause an appreciation of the real exchange rate which then crowds out the very export industries on which a country may depend for growth. When I was at the Department for International Development (the UK aid agency), we did quite a careful study of this. The results were published here. The paper found that, sustained aid inflows raise incomes and welfare in the recipient country, but in doing so they generally cause some appreciation of the nominal and real exchange rates. This is a necessary aspect of the adjustment process. If aid is used effectively, this appreciation will not undermine growth, either in aggregate or in the export sector. We also found that temporary increases in aid require greater flexibility on the part of the recipient government, and that this effect was an additional reason why donors should ensure ensure that aid flows and their disbursement are sustained and predictable.

Rajan was quoted in the FT saying "We need to be careful given the chequered history of aid, that we do not place more hopes on aid as an instrument of development than it is capable of delivering." This is absolutely right: aid is not the whole answer to development. But the study which claims to contradict the substantial body of evidence that aid makes a significant contribution to economic growth in developing countries is seriously flawed at a technical level.

4 thoughts on “Technical shortcomings with IMF analysis”

  1. It’s a little too convenient to select the regressions which yielded significant estimates then to claim they’re clearly the correct ones. I find it hard to believe the authors would run a raft of regressions with varied functional forms, estimations methods, instrumental variables, independent variables, etc. if the true model was at all obvious from independent observation and past literature. For example, you could justify a linear model by claiming countries in question are so far behind in the development process there’s little reason to expect any approach towards productive capacity for all countries in question any time over the period studied. I’m sure that certain types of aid can yield economic growth, but it’s unlikely that the issue is as black and white as you suggest. Not being an econometrician, more concrete thoughts on your colleague’s criticisms aren’t possible.

    Owen replies: Iyobosa – I’m afraid I did not make my point clear. There are good theoretical reasons for a different model specification, and that different model is well supported by the data. It shows a strong, statistically significant correlation between aid and growth. The authors have chosen a functional form which does not demonstrate a statistically significant relationship between aid and growth; but they should not reach the more general conclusion that there is no relationship between aid and growth. There is plenty of evidence of such a relationship.

  2. Hi Owen,

    No doubt you (and others) must have seen the new posting on IMF website from the authors today (as well as the article in FT). They made it clear that their study is not the view of IMF.

    “”Andrew Balls’ article “Aid will not lift growth in Africa, IMF warns” (June 29), reporting on my research with Arvind Subramanian, is misleading; nor does it represent the International Monetary Fund’s position.

    First, the aim of research papers is to inform and generate internal and external debate, but every research paper, including the ones on which Mr. Balls reports, clearly states that the views are those of the authors and not necessarily those of the IMF, its management or its board. What is more, the pessimism in your headline mischaracterises the implications drawn by us, as well as the position of the IMF. Your editorial “Time to prove aid works” (July 1) helps clarify what our studies do, and do not, say and is most welcome.

    Consistent with the implications drawn out in our research, the IMF has long held that promoting growth in Africa, and developing countries generally, requires comprehensive efforts by all concerned. The approach suggested by the Commission on Africa of experimentation to find what works is absolutely spot on and should be supported by the needed resources. The achievement of the Millennium Development Goals demands a truly global partnership. The developing countries themselves must take the lead through good governance and sound economic policies. But these efforts will work only in harmony with a global trading regime that is conducive to their exports and more – especially more effective – financial assistance.

    With so much of the world’s attention focused on initiatives to reduce poverty, this is indeed a time of great possibilities. Our ambitions should be informed by a better understanding of the past, so that we can help forge a better future for those who most desperately need it.” source: http://www.imf.org/external/np/vc/2005/070505.htm
    cheers,
    Davina

  3. Pingback: Does aid promote economic growth?

  4. Pingback: Should Aid Focus on Economic Development? (part 2) | Development Policy blog

Leave a Reply

Your email address will not be published. Required fields are marked *