Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff

RR’s “stylised fact” of a 90% debt threshold, which had become the principal empirical support for post-crisis austerity. The authors quote RR’s own summary of the overarching result, and note that “To build the case that they had established a new set of stylised facts… RR stress the robustness of their overarching findings to a range of countries and time periods. They also stress the robustness of these findings to alternative measurement techniques and ways to categorise data.” That robustness claim is what the replication is testing: “A necessary condition for establishing a stylised fact is that the calculations on which such facts are based are accurate and that the results of such calculations are robust across alternative reasonable methods of calculation. Through our replication exercise, we conclude that their findings are neither accurate nor robust.” On influence, the paper documents rather than asserts: RR’s findings underpinned Reinhart’s March 2011 testimony to the House Budget Committee and a Financial Times opinion piece; a Google Scholar search excluding self-citations returned 538 results as of 7 April 2013; RR’s website listed 76 high-profile media features; RR (2010B) “was the only evidence cited on the consequences of high public debt on economic growth in the 2013 US Federal Budget plan proposed by Republican Paul Ryan,” whose “Path to Prosperity” described the research as having “found conclusive empirical evidence that gross debt… exceeding 90 percent of the economy has a significant negative effect on economic growth”; and George Osborne and Olli Rehn cited it frequently. Krugman’s assessment is quoted: “Reinhart–Rogoff may have had more immediate influence on public debate than any previous paper in the history of economics.” The authors are careful about attributing motive: “it is an indisputable fact that ‘Growth in a Time of Debt’ has provided a critical intellectual underpinning on behalf of austerity policies. This is the case regardless of whether it was RR’s intention to exert this type of influence. We do not attempt to discern RR’s intentions on this matter.”

A non-parametric binning of country-years by debt/GDP, with the direction of causation conceded for the sake of argument. “The methods employed by RR in ‘Growth in a Time of Debt’ are non-parametric and appealingly straightforward. They organise data for a range of countries and different time periods based on the ratios of public debt relative to GDP in each country. They present four data categories in terms of public debt/GDP ratios: ≤30%, 30-60%, 60-90% and >90%. They then compare average real GDP growth rates across each of the public debt/GDP groupings.” The scope restriction on causality is stated explicitly and is essential to reading the result correctly: “For the purposes of this replication, we follow RR in assuming that the direction of causation in the relationship between public debt levels is that high public debt levels produce declines in average GDP growth rates. In other work (see, e.g., RR, 2011; Reinhart et al., 2012), RR acknowledge the potential for reverse causation. This would occur primarily as a result of recessions, which would raise public indebtedness by both reducing tax revenue and increasing public expenditures through countercyclical interventions. However, in the papers that we are replicating (RR, 2010A, 2010B), RR make clear that their analysis is organised around the premise that the primary direction of causation runs from high public debt to slower GDP growth.” RR examine three datasets – 20 advanced economies 1946-2009, the same 20 over 1790-2009, and 20 emerging markets 1970-2009 – and the replication covers only the first two, focusing on the post-war sample “since these figures are clearly the most relevant to ongoing US and European policy debates” and because “the more recent data are also the most reliable, since they entailed much less splicing together of data by RR from multiple sources that frequently used different statistical methodologies.”

Because the publicly posted data did not permit replication. “On their web site, RR provide public access to country historical data for public debt and GDP growth in spreadsheets with complete source documentation. However, these publicly available spreadsheets do not include information on the exact data series, years and methods used in their paper. As such, we were unable to replicate the RR results from the data they posted on their web site. In response to our request of April 2013, RR did provide us with the working spreadsheet that they used in producing the RR papers. Through using their working spreadsheet, we were able to approximate closely the published RR results. This was how we were able to identify the selective exclusion of available data, coding errors and inappropriate methods for weighting summary statistics.” The authors thank RR both for sending the spreadsheet on 4 April 2013 “and for their constructive responses to our initial HAP working paper.”

Early post-war observations for three fast-growing, highly indebted countries, while the contemporaneous US observations – which cut the other way – were kept. The exclusions are Australia 1946-50, New Zealand 1946-49 and Canada 1946-50, and “At no point do RR either explicitly explain they why they chose to make these data exclusions or even indicate that they had done so.” The nearest thing to a rationale in RR’s text is their remark that “there is considerable variation across the countries, with some countries such as Australia and New Zealand experiencing no growth deterioration at very high debt levels. It is noteworthy, however, that those high-growth high-debt observations are clustered in the years following World War II.” The authors’ objection is the inconsistency: “in contrast with this reasoning as applied to the cases of Australia, Canada and New Zealand, RR include all four of the immediate post-World War II observations in which the USA was in the >90% public debt/GDP category. In three of these four years, the US economy was contracting at the same time as it was in the highest public debt/GDP category.” The US figures are given: debt/GDP of 121.3% in 1946 with growth of -10.9%, and growth over 1946-49 of -10.9%, -0.9%, 4.4% and -0.5% – declines “associated with post-World War II demobilisation.” What the exclusions cost: for Canada, “all five omitted years were in the >90% public debt/GDP category. Those years were also the only ones in which Canada was in this highest public debt/GDP category. Mean GDP growth in Canada for the excluded years was 3.0%, while median Canadian GDP growth for these years was 2.2%.” The same holds for Australia. New Zealand is the pivotal case: it “was in the highest public debt/GDP category in all four of these excluded years. New Zealand’s real GDP growth rates in those years was +7.7%, +11.9%, -9.9% and +10.8%. After RR chose to exclude the 1946-49 New Zealand data, New Zealand then contributes only one year, 1951, to the highest public debt/GDP category. RR report New Zealand’s real GDP growth in 1951 as being -7.6%.”

An averaging range in the spreadsheet that stopped five rows short, silently removing five countries from every calculation in both samples. “In addition to these deliberate data exclusions by RR, a coding error in the RR working spreadsheet also unintentionally excludes five countries entirely (Australia, Austria, Belgium, Canada and Denmark) from all parts of the analysis. The error appears in the calculations of both mean and median GDP growth with the 1946-2009 sample as well as with the mean and median GDP growth for the sample over the 220-year period 1790-2009. The omitted countries are selected alphabetically. It is clear from the spreadsheet itself that these are random exclusions. RR have since acknowledged this to be the case.” The mechanics are given in a footnote: “In their analysis with the 1946-2009 dataset, RR calculated both means and medians of cells in lines 30-44 instead of lines 30-49. In their analysis with the 1790-2009 dataset, RR calculated both means and medians for cells in lines 5-19 instead of lines 5-24.” The authors are careful not to overstate its effect on the headline number: “Austria and Denmark (i.e. two of the five countries RR excluded from their analysis due to their spreadsheet error) did not, in fact, experience any years over 1946-2009 in which their public debt exceeded 90% of their GDP. As such, RR’s inadvertent exclusion of these two countries from their analysis did not affect their estimates of average GDP growth when public debt exceeded 90% of GDP.”

More than a third of the country-years above 90%, and three of the ten countries. “As we see from Table 2, the correct total in RR’s 1946-2009 data sample in which a country is in the >90% public debt/GDP category is 110 country-years. With RR’s chosen data exclusions, the total falls to 96 country-years. With both their chosen exclusions and their spreadsheet errors, the total number of country-years in the >90% public debt/GDP category falls to 71.” The country count falls from 10 to 8 to 7. The countries appearing above 90% at any point, with their year counts, are Australia (5), Belgium (25), Canada (5), Greece (19), Ireland (7), Italy (10), Japan (11), New Zealand (5), the UK (19) and the USA (4); Australia, Belgium and Canada end up contributing nothing.

It treats a country as the unit of observation regardless of how long it stayed in the category, which magnifies brief episodes. “After assigning each country-year to one of the four public debt/GDP categories, RR calculate the mean real GDP growth for each country within the category, i.e. a single average value for the country for all the years it appeared in the category. In other words, RR compute overall averages as means of country means.” The consequence is stated concretely: the UK’s “2.4% per year during the 19 years that the UK appeared in the >90% public debt/GDP category… then counts as one country observation,” and “According to RR’s methodology, this one year experience for New Zealand counts equally with the 19 years in which the UK was in the highest public debt/GDP category.” The general point: “Clearly, the impact of RR’s approach is to greatly amplify the effects of short-term episodes with high public debt levels in calculating the overall impact of high public debt on GDP growth.” A second example, from the long sample: Norway appears in the 60-90% category for exactly one year (1946) with 10.2% growth “due to rapid recovery after occupation during World War II,” and “This one extraordinary growth experience contributes fully 5.3% (one of 19 countries) of the weight for the mean GDP growth in the 60-90% public debt category even though it constitutes only 0.2% (one of 455 country-years) of the country-years in this category. Indeed, Norway’s one year in the 60-90% public debt category receives a weight equal to, for example, 23 years for Canada, 35 years for Austria, 42 years for Italy and 47 years for Spain.” The same procedure applies to medians, so what RR call the median “is, more precisely, the median of each country’s median GDP growth figure.”

No – the objection is that it is undefended, and the authors concede a case could be made for it. “RR need to explain and justify in detail their weighting methodology for generating means and medians, yet at no point do they do so in either version of their 2010 paper. As such, their methodology appears arbitrary and unsupportable.” The concession follows immediately: “In fact, it is possible that within-country serially correlated relationships could support an argument that not every additional country-year observation contributes a proportional amount of additional useful information. Thus, the existence of serial correlation could suggest that, with the case of the UK, for example, 19 years of carrying a public debt/GDP load greater than 90% and averaging 2.4% GDP growth over those years does not warrant 19 times the weight of New Zealand’s single year at -7.6% GDP growth. But RR do not themselves offer any argument as to why the one-year experience in New Zealand should have 19 times the influence as each year in which the UK economy operated with high public debt levels.” A footnote records RR’s actual reply – “It is the accusation that our weighting procedure is unconventional that is itself unconventional” – and notes it “is not followed by any substantive discussion as to why their methodology should be preferred.”

From a flat 14.3% per included country to a range from 3.6% to 22.7%. Under RR’s accounting, Australia, Belgium and Canada carry zero weight and each of the remaining seven countries carries one-seventh. Under country-year weighting with the full sample, the weights become Belgium 22.7%, Greece 17.3%, the UK 17.3%, Japan 10.0%, Italy 9.1%, Ireland 6.4%, Australia 4.5%, Canada 4.5%, New Zealand 4.5% and the USA 3.6%. The effect is largest where it matters most: Belgium, entirely absent from RR’s calculation, “should properly account for fully 22.7% of the total weight of country-years in the >90% public debt/GDP category,” while the four contracting US years fall from 14.3% of the sample to 3.6%.

From three large factors and one small one, with strong interaction between them. The paper decomposes the gap in the >90% bin. The three main factors: “The full exclusions of Australia, Belgium and Canada from the highest public debt category. The GDP growth rates for these three countries while in the highest public debt category averaged 3.8%, 2.6% and 3.0%, respectively”; “The exclusions of 1946-49 data for New Zealand. This meant that RR included only the one year, 1951… Given their weighting methodology, this one year, with -7.6% growth, counted as 14.3% of the entire sample of observations”; and “the four years (1946-49) in which the USA is in the highest public debt category and averaged -2.0% GDP growth are weighted as 14.3% of all observations by RR, as opposed to 3.6% of all observations through proper accounting and country-year weighting.” The fourth is a transcription slip: RR “made a transcription error in transferring the country average figure from the country-specific spreadsheets to the summary spreadsheet. This transcription error reduced New Zealand’s average growth in the >90% public debt/GDP category from -7.6% to the figure they report, -7.9%,” which “reduces RR’s estimate of mean real GDP growth by 0.1 percentage point.” The decomposition table shows how far each factor gets on its own and how they compound: each of the three acting alone brings the corrected 2.2% down only to 1.9%; the spreadsheet error plus the selective exclusions gives 1.7%; the spreadsheet error plus country weights gives 1.4%; the selective exclusions plus country weights gives 0.3%; all three together give 0.0%; and adding the transcription error gives the published -0.1%. Crucially, the same factors barely move the other bins: “for the 0-30% public debt/GDP category average GDP growth remains consistently around 4% per year. For the 30-60% and 60-90% public debt/GDP categories, average GDP growth is consistently around 3% per year with or without adjusting for the RR errors and methodological choices.”

Yes, in the same direction, though by less. For the long 1790-2009 sample, RR’s means are 3.7%, 3.0%, 3.4% and 1.7% – “a steep drop-off of 1.7 percentage points” entering the top bin. Corrected and country-year weighted, they are 3.7%, 3.2%, 2.5% and 2.1%, giving “two important points… (i) the growth decline in the >90% public debt/GDP category relative to the 60-90% category is a modest 0.4 percentage points; and (ii) this growth drop-off is less than the decline that occurs between the 30-60% and 60-90% public debt/GDP categories, where the decline is 0.7 percentage points.” For 1946-2009 medians – which RR emphasised in their reply as the figures they “had always accorded greater significance,” having “never used anything but the conservative median estimate in our public discussions” – RR’s published values are 4.2%, 3.0%, 2.9% and 1.6%. The authors’ corrected country-year medians are 4.1%, 3.1%, 2.9% and 2.3%, “a 0.6 percentage point drop-off.” And the sharpest point is that RR’s own corrected numbers agree: in their Errata, “properly includ[ing] all the relevant figures from their spreadsheet while still calculating medians through their one-country/one-observation methodology… RR themselves find that GDP growth falls from 2.9% for the 60-90% category to only 2.5% for the >90% category. That is, using medians as their preferred measure of central tendency as well as their preferred one-observation-per-country weighting methodology, the GDP growth drop-off for the >90% public debt/GDP category is now only 0.4 percentage points.”

No – and three separate exercises are run to show it. First, splitting the top bin: adding a 90-120% category so that >120% becomes the top group, “mean GDP growth in the 90-120% category is 2.4%, which is reasonably close to the 3.2% GDP growth figure for the 60-90% category. Mean GDP growth in the new category (>120% public debt/GDP) is lower at 1.6% but does not fall off a non-linear cliff.” With the long sample the point is stronger still: “mean real GDP growth in the 90-120% public debt/GDP category is 2.5%. This GDP growth rate is identical to the figure for the 60-90% category. Even with the new highest category of >120%, mean GDP growth is lower at 1.6%, but GDP growth does not decline in a sharp non-linear way even at this highest public debt/GDP level.” Second, a scatter of all country-years with a locally fitted regression – a generalised additive model with cross-validated smoothing, with LOESS and alternative smoothing parameters reported to give similar results – shows that “no particular boundary or non-linearity is evident in either dimension around the 90% figure for the public debt/GDP ratio. The data thin out gradually between 70% and 120% public debt/GDP ratios, as is clear from both the points in the scatter plot and the widening of the 95% confidence interval for mean GDP growth.” Third, the flatness is quantified: “between public debt/GDP ratios of 38-117%, we cannot reject a null hypothesis that average real GDP growth is 3%.” The paper also emphasises dispersion as a finding in its own right – “the plot also shows large variation in real GDP growth in each public debt/GDP category… But RR do not examine this issue at all in their 2010 studies.”

Yes, at the bottom of the debt range, which is not where the policy debate is. “What we see clearly in Figure 5 is that at 0% public debt/GDP ratio, average GDP growth is almost 5%. But when the public debt/GDP ratio reaches 30%, average GDP growth is only slightly greater than 3%, i.e. average GDP growth drops off by nearly 2 percentage points as countries’ public debt/GDP ratios rise from 0% to 30%.” The authors note this “contradicts RR’s claim that ‘it is evident that there is no obvious link between debt and growth until public debt reaches a threshold of 90 percent’,” and observe in the introduction that the genuine non-linearity lies “between the lowest two public debt/GDP categories, 0-30% and 30-60%, a range that is not relevant to current policy debate.” They also engage RR’s analogy to “debt intolerance,” noting that the 2003 Reinhart-Rogoff-Savastano concept “refers to the propensity with developing countries for debt crises and default to result when a country’s external debt approaches a context-specific threshold,” where such countries undergo “extreme duress” – so “By suggesting a similarity to this ‘debt intolerance’ scenario… RR make it clear that they envision a public debt/GDP threshold that leads to sharp reductions in GDP growth once a country crosses the historical threshold.”

No: growth in the lower bins deteriorates over time while the top bin stays flat, so the gap narrows and by 2000-09 inverts. Across sub-samples starting in 1950, 1960, 1970, 1980, 1990 and 2000, “the higher GDP growth rates for the 0-30% public debt/GDP category erodes substantially in the shorter and more recent time periods. Thus, GDP growth for the 0-30% category was 4.1% per year in the 1950-2009 sample, but declines to only 2.5% in the 1980-2009 sample,” while “average GDP growth in the highest public debt/GDP category remains stable across all samples of years, remaining within 0.3 percentage points of 2% per year throughout.” For 2000-09 the ordering breaks entirely: the means are 2.7% (≤30%), 1.9% (30-60%), 1.3% (60-90%) and 1.7% (>90%), so “the growth trajectory for the >90% category actually outperformed growth in the 60-90% category and is only 0.2 percentage points below the 1.9% growth rate for the 30-60% category.” The authors flag the sample size themselves: only four countries appear above 90% in those years, contributing 31 country-years (Belgium 8, Greece 10, Italy 3, Japan 10), and “as is conveyed by the larger size of the standard errors as our sample size narrows, our estimates for GDP growth over 2000-09 are less reliable than the figures for the longer time periods.” The two conclusions drawn are correspondingly measured: “even the apparent non-linearity between the 0-30% public debt/GDP category and the higher categories is a historically specific pattern, not a robust result across the full 1946-2009 time period; and… the relationship between public debt and GDP growth is weaker in more recent years relative to the earlier years of the sample.”

They conceded the spreadsheet error and the absence of a sharp threshold; four items were left unaddressed. “In the aftermath of the public debate generated by the posting of our April 2013 working paper, RR did acknowledge their spreadsheet errors. They also recognised that, in fact, there is no clear public debt threshold beyond which GDP growth will fall off sharply. At the same time, RR have not addressed other crucial problems that we identified.” The four: (i) the asymmetric treatment of the USA versus Australia, Canada and New Zealand in the early post-war years, where “The US figures for these years support their hypothesis while those from Australia, Canada and New Zealand weaken their hypothesis”; (ii) the finding “that the relationship between public debt levels and GDP growth varies substantially by country and over time,” especially for 2000-09, since “Relative to experiences from 60 or 200 years ago, such recent patterns for GDP growth under high public debt levels are likely to be more informative for assessing present-day policy concerns”; (iii) that on RR’s own preferred medians and own preferred weighting, their Errata figures show a drop-off of only 0.4 points for 1946-2009 and 0.3 points for 1790-2009 – “RR themselves find no substantial GDP growth decline through their own recalculations, but they have not acknowledged this result”; and (iv) robustness as such – “their main finding on mean GDP growth for the >90% public debt/GDP category can swing by almost 2 percentage points of GDP growth based on the treatment of New Zealand’s early postwar years alone. We strongly support what we take to be a consensus view of research standards: that any major empirical conclusions need to hold up consistently when one moves from using one method of calculation to another. RR’s findings do not meet this standard test for robustness.”

That the 90% austerity argument cannot be defended on this evidence – not that debt is harmless, and not that causality runs the other way. The overall verdict on the original: “RR made significant mistakes in reaching the conclusion that countries facing public debt levels in excess of 90% of GDP will experience a major decline in their GDP growth rate… The full extent of their mistakes transforms the reality of modestly diminished average GDP growth rates for countries carrying high public debt levels into a false image that high public debt ratios inevitably entail sharp declines in GDP growth.” Note the phrasing: modestly diminished, not absent. The paper adds that “there is a wide range of GDP growth performances at every level of public debt among the 20 advanced economies that RR survey,” and that “The relationship between public debt and GDP growth varies significantly by period and country.” The policy conclusion is stated narrowly: “it has established that policy makers cannot defend austerity measures on the grounds that public debt levels greater than 90% of GDP will consistently produce sharp declines in economic growth.” What the paper does not do it says up front: it does not survey the literature on public indebtedness and growth, does not attempt “an extended discussion on austerity policies or any other related policy issue,” and does not challenge the causal direction it has adopted from RR for the purposes of the exercise.