Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

Standard recursive formulations of heterogeneous-agent models with aggregate risk force the entire cross-sectional distribution of agents into the Bellman equation characterizing individual decisions – the “Master equation” – purely because low-dimensional equilibrium prices, unlike the distribution itself, do not follow a Markov process, so rational agents forecasting prices end up needing to forecast the whole distribution. This extreme curse of dimensionality remains the central computational bottleneck for global solutions of heterogeneous-agent models, so severe that even a Huggett (1993) model with aggregate risk – despite looking simple – proved impossible for any team to solve in an influential benchmarking exercise and was dropped from the project altogether. This paper sidesteps the Master equation entirely using ideas from reinforcement learning (RL): agents learn equilibrium price dynamics directly from simulated paths, as standard RL would, but the paper’s “structural reinforcement learning” (SRL) approach departs from standard RL by assuming agents have structural knowledge of their own individual-state dynamics (their budget constraint and idiosyncratic income process), letting the authors compute exact policy gradients by differentiating through these known dynamics rather than relying on the noisy, approximate policy gradients standard RL methods estimate; only the equilibrium price process itself is treated as unknown and learned from simulation. By further restricting agents to condition their policies only on current (or briefly lagged) prices rather than the full price history or the distribution, the paper solves for a low-dimensional “restricted perceptions equilibrium” in the sense of Sargent (1991) rather than the full rational-expectations equilibrium – expectations are restricted in functional form but remain statistically consistent with actual outcomes. Because policy functions depend only on prices, they double as individual supply/demand schedules that can be integrated across the distribution and market-cleared period-by-period along a simulation, treating market clearing as part of the “environment” (in RL parlance) rather than something solved inside an optimization loop – which is what lets the method efficiently handle nontrivial market-clearing conditions that have historically been very hard. Implemented in JAX on a single GPU, the resulting structural policy gradient (SPG) algorithm solves the Krusell and Smith (1998) model in about 55 seconds, the previously-unsolved Huggett (1993) model with aggregate risk in around one minute, and a one-asset HANK model with a forward-looking New Keynesian Phillips curve in around three minutes – with the Krusell-Smith solution closely matching alternative global solutions of the rational-expectations equilibrium, and allowing agents a longer history of lagged prices barely moving the solution, indicating most of the information relevant for forecasting prices is already contained in current prices. The paper is explicit that its algorithm, as presented, is not itself intended as an empirically realistic theory of how real economic agents form expectations, though it suggests the “sampling”-based logic behind SRL could in principle be developed into one.

Chiyuan Wang | Macro Paper Warehouse

Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics