Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

“This difficulty arises even though agents do not directly ‘care about’ the distribution, i.e. it does not enter their objective functions; instead, as is standard in competitive equilibrium models, they only care about prices. Intuitively, low-dimensional equilibrium prices do not follow a Markov process but the extremely high-dimensional distribution does. Therefore, agents with rational expectations forecast prices by forecasting distributions” (Introduction, p. 1). This indirect channel – needing the distribution only to forecast prices, which are themselves Markov only jointly with the distribution – is what the paper’s method is designed to break.

“Our approach differs from standard RL in that we assume that agents have structural knowledge about the dynamics of their own individual states (e.g. their budget constraint and idiosyncratic income process). We term this hybrid approach structural reinforcement learning (SRL)… In contrast to standard policy gradient methods which estimate approximate policy gradients, this assumption allows us to compute exact policy gradients by differentiating through these individual dynamics” (Introduction, pp. 1, 4). “The only part of the environment that is treated as unknown is the process for general-equilibrium prices and aggregate shocks, which is learned from simulated data” – the analogy to structural VARs is explicit: agents’ own laws of motion are structurally known, while the aggregate price process is estimated (Introduction, p. 2, fn. 4).

“By the Wold representation theorem, this assumption does not, by itself, imply any departure from full information rational expectations” when agents observe the full history of prices; the further restriction that policy functions depend “only on current prices or, perhaps, a short price history” means the model solves “a restricted perceptions equilibrium (RPE) in the sense of Sargent (1991), Evans and Honkapohja (2001), or Branch (2006): while agents’ expectations are restricted, they are nevertheless statistically consistent with actual equilibrium outcomes, thereby fulfilling one of the desiderata of rational expectations”** (Introduction, p. 2).

“According to Maliar and Maliar (2020), in the influential JEDC special issue on numerical solution methods for HA models with aggregate risk (Den Haan et al., 2010), all participants were initially asked to solve two benchmark models globally – the Krusell and Smith (1998) model and the Huggett (1993) model with aggregate shocks… However, as they report, no single team was able to successfully solve the latter model, and it was ultimately dropped from the JEDC project” (Introduction, p. 2, fn. 6). The paper’s ability to solve this exact model in about a minute is presented as direct evidence of the SRL method’s power relative to prior global solution approaches.

“Our treatment of market clearing mirrors the RL literature’s distinction between agents and environment: agents can interact with their environment under any given policy including suboptimal ones; finding optimal policies is conceptually separate. In line with this dichotomy, we treat market clearing as part of the environment and find equilibrium prices also for suboptimal policies. This approach differs from standard practice in macroeconomics which first finds optimal policies given prices in an inner loop” (Introduction, pp. 3-4). Because policy functions depend only on current prices, they “double as individual supply schedules,” so integrating over the distribution to get the aggregate supply schedule makes solving for the period-by-period market-clearing price straightforward along any simulated path – “efficiently handl[ing] non-trivial market clearing conditions like in Huggett (1993)” that other methods have found intractable.

“Almost all existing global solution methods for heterogeneous agent models use dynamic programming… [either directly on] the high-dimensional Master equation… or, as in Krusell and Smith (1998), they use model-generated data to estimate a low-dimensional Markovian ‘perceived law of motion’ (PLM) for moments of the distribution and then apply dynamic programming to this lower-dimensional approximate problem. Our SPG algorithm instead works directly with the sequential formulation of the problem and never attempts to force it into the standard Markovian structure required for applying dynamic programming… we never estimate a PLM” (Introduction, p. 2). Instead, the method exploits the basic RL insight that value functions are expected discounted lifetime utilities that “can… be approximated by averaging across simulated trajectories,” then finds the low-dimensional policy maximizing expected lifetime utility via stochastic gradient ascent.

Plotting the bond market’s aggregate demand against its (fixed, zero-net) supply over a simulated path, “asset demand fluctuates tightly around the supply level, and deviations from the horizontal supply line measure residuals in the bond market clearing condition. On average, the relative absolute deviation of aggregate demand from supply is 0.22%,” which the authors attribute primarily to the use of linear interpolation when solving for market-clearing prices (section 4, pre-Conclusion). In the Krusell-Smith model, the paper separately reports that “the resulting policies are close to those obtained from alternative global solutions of the rational expectations equilibrium,” and that letting agents condition on a longer lagged-price history “hardly moves the solution,” indicating most forecast-relevant information is already in current prices (Conclusion).

“The core idea of our approach – that agents form price expectations by sampling – could, in principle, serve as a building block for an empirically realistic theory of expectations formation in macroeconomics (which the algorithm in this paper is not),” noting that “the plausibility of RL-based approaches is supported by evidence that reinforcement learning underpins a substantial share of human and animal learning” (Conclusion). The authors are explicit, however, that developing such a theory would require converting the algorithm to a fully online, incremental RL procedure in which agents update policies continuously while interacting with their environment, rather than only after observing many pre-simulated price trajectories as the current SPG algorithm does.

The paper explicitly frames itself as sidestepping the Master equation that underlies Den Haan (1996), Krusell and Smith (1998), Schaab (2020), and Bilal (2023) – three of which appear elsewhere in this reading list – by never representing the distribution as a state variable at all, in contrast to Schaab’s (2020) global finite-dimensional distribution representation and Bilal’s (2023) analytic Master-Equation perturbation (Introduction, p. 1, fn. 1). Relative to the other machine-learning-based global methods on this list, DeepHAM (Han, Yang and E) and Kase, Melosi and Rottner (2022), the key methodological distinction is that this paper’s agents differentiate exactly through their own known individual dynamics rather than learning a full policy or value-function approximation purely from data, and its market-clearing-as-environment design is specifically built to handle the nontrivial market-clearing conditions (like Huggett’s) that those other neural-network approaches do not target as their central test case.