DeepHAM: A global solution method for heterogeneous agent models with aggregate shocks

Solving heterogeneous-agent (HA) models with aggregate shocks efficiently, reliably, interpretably, and generally has proven difficult: the Krusell-Smith (KS) method approximates the distribution with a small number of moments and is efficient for simple models but suffers a curse of dimensionality with multiple shocks or endogenous states, while the local-perturbation method of Reiter (2009) handles complex models but is unreliable wherever aggregate shocks generate nonlinear or nonlocal effects (a zero lower bound, large shocks, or a risky steady state that departs from the deterministic one). This paper’s method, DeepHAM, is designed to satisfy all four requirements – efficiency, reliability, interpretability, and generality – at once. It represents each agent’s value and policy functions with deep neural networks, and, rather than feeding these networks the entire cross-sectional distribution, first extracts a small number of “generalized moments”: neural-network-determined, permutation-invariant summary statistics of the distribution that play a role analogous to classical moments (such as the first moment of wealth) but are automatically optimized rather than fixed a priori. The networks are trained by directly optimizing the model’s objective over simulated economic paths, sidestepping the fixed-point iterations that both KS-style and Reiter-style methods require. In a calibrated Krusell-Smith benchmark, DeepHAM using only the first moment already reduces the Bellman equation error by 27.2% relative to the classic KS solution, and DeepHAM with one algorithmically-optimized generalized moment reduces the error by 40.3%; the resulting generalized moment reveals that the mapping from individual wealth to the moment is concave, implying that a purely redistributive, unanticipated policy shock changes the welfare of “middle” households who are not part of the redistribution program – a channel invisible to the standard KS solution, in which welfare depends only on the first moment. DeepHAM also efficiently solves more complex environments, including a model with a Brunnermeier-Sannikov-style financial sector, without suffering the curse of dimensionality that afflicts moment-matching or discretized state-space methods as the number of shocks or endogenous states grows. Finally, because DeepHAM’s neural networks are trained directly against a stated objective rather than derived from a decentralized equilibrium concept, the same framework solves the planner’s constrained-efficiency problem as easily as the competitive equilibrium, which the authors note “opens up new possibilities for studying optimal monetary and fiscal policies in heterogeneous agent models with aggregate shocks.” The paper’s stated scope excludes models in which aggregate variables are determined recursively as a function of expected future aggregate variables (such as inflation in a forward-looking New Keynesian Phillips curve), which the authors flag as requiring an additional price function and leave for companion work.

Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

Standard recursive formulations of heterogeneous-agent models with aggregate risk force the entire cross-sectional distribution of agents into the Bellman equation characterizing individual decisions – the “Master equation” – purely because low-dimensional equilibrium prices, unlike the distribution itself, do not follow a Markov process, so rational agents forecasting prices end up needing to forecast the whole distribution. This extreme curse of dimensionality remains the central computational bottleneck for global solutions of heterogeneous-agent models, so severe that even a Huggett (1993) model with aggregate risk – despite looking simple – proved impossible for any team to solve in an influential benchmarking exercise and was dropped from the project altogether. This paper sidesteps the Master equation entirely using ideas from reinforcement learning (RL): agents learn equilibrium price dynamics directly from simulated paths, as standard RL would, but the paper’s “structural reinforcement learning” (SRL) approach departs from standard RL by assuming agents have structural knowledge of their own individual-state dynamics (their budget constraint and idiosyncratic income process), letting the authors compute exact policy gradients by differentiating through these known dynamics rather than relying on the noisy, approximate policy gradients standard RL methods estimate; only the equilibrium price process itself is treated as unknown and learned from simulation. By further restricting agents to condition their policies only on current (or briefly lagged) prices rather than the full price history or the distribution, the paper solves for a low-dimensional “restricted perceptions equilibrium” in the sense of Sargent (1991) rather than the full rational-expectations equilibrium – expectations are restricted in functional form but remain statistically consistent with actual outcomes. Because policy functions depend only on prices, they double as individual supply/demand schedules that can be integrated across the distribution and market-cleared period-by-period along a simulation, treating market clearing as part of the “environment” (in RL parlance) rather than something solved inside an optimization loop – which is what lets the method efficiently handle nontrivial market-clearing conditions that have historically been very hard. Implemented in JAX on a single GPU, the resulting structural policy gradient (SPG) algorithm solves the Krusell and Smith (1998) model in about 55 seconds, the previously-unsolved Huggett (1993) model with aggregate risk in around one minute, and a one-asset HANK model with a forward-looking New Keynesian Phillips curve in around three minutes – with the Krusell-Smith solution closely matching alternative global solutions of the rational-expectations equilibrium, and allowing agents a longer history of lagged prices barely moving the solution, indicating most of the information relevant for forecasting prices is already contained in current prices. The paper is explicit that its algorithm, as presented, is not itself intended as an empirically realistic theory of how real economic agents form expectations, though it suggests the “sampling”-based logic behind SRL could in principle be developed into one.

Yucheng Yang | Macro Paper Warehouse

DeepHAM: A global solution method for heterogeneous agent models with aggregate shocks

Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics