C45 | Macro Paper Warehouse

DeepHAM: A global solution method for heterogeneous agent models with aggregate shocks

Solving heterogeneous-agent (HA) models with aggregate shocks efficiently, reliably, interpretably, and generally has proven difficult: the Krusell-Smith (KS) method approximates the distribution with a small number of moments and is efficient for simple models but suffers a curse of dimensionality with multiple shocks or endogenous states, while the local-perturbation method of Reiter (2009) handles complex models but is unreliable wherever aggregate shocks generate nonlinear or nonlocal effects (a zero lower bound, large shocks, or a risky steady state that departs from the deterministic one). This paper’s method, DeepHAM, is designed to satisfy all four requirements – efficiency, reliability, interpretability, and generality – at once. It represents each agent’s value and policy functions with deep neural networks, and, rather than feeding these networks the entire cross-sectional distribution, first extracts a small number of “generalized moments”: neural-network-determined, permutation-invariant summary statistics of the distribution that play a role analogous to classical moments (such as the first moment of wealth) but are automatically optimized rather than fixed a priori. The networks are trained by directly optimizing the model’s objective over simulated economic paths, sidestepping the fixed-point iterations that both KS-style and Reiter-style methods require. In a calibrated Krusell-Smith benchmark, DeepHAM using only the first moment already reduces the Bellman equation error by 27.2% relative to the classic KS solution, and DeepHAM with one algorithmically-optimized generalized moment reduces the error by 40.3%; the resulting generalized moment reveals that the mapping from individual wealth to the moment is concave, implying that a purely redistributive, unanticipated policy shock changes the welfare of “middle” households who are not part of the redistribution program – a channel invisible to the standard KS solution, in which welfare depends only on the first moment. DeepHAM also efficiently solves more complex environments, including a model with a Brunnermeier-Sannikov-style financial sector, without suffering the curse of dimensionality that afflicts moment-matching or discretized state-space methods as the number of shocks or endogenous states grows. Finally, because DeepHAM’s neural networks are trained directly against a stated objective rather than derived from a decentralized equilibrium concept, the same framework solves the planner’s constrained-efficiency problem as easily as the competitive equilibrium, which the authors note “opens up new possibilities for studying optimal monetary and fiscal policies in heterogeneous agent models with aggregate shocks.” The paper’s stated scope excludes models in which aggregate variables are determined recursively as a function of expected future aggregate variables (such as inflation in a forward-looking New Keynesian Phillips curve), which the authors flag as requiring an additional price function and leave for companion work.

Estimating Nonlinear Heterogeneous Agents Models with Neural Networks

Economists routinely approximate away features of their models – nonlinear dynamics, aggregate uncertainty, agent heterogeneity – to make estimation feasible, but it is often unclear how much these simplifications distort a model’s predictions. This paper develops a neural-network-based solution and estimation method designed to avoid such approximations entirely. The key device is to treat a model’s structural parameters as additional “pseudo state variables” fed into the neural networks that approximate its policy functions, so that a single (more expensive) training run yields the model’s entire solution mapping across the whole parameter space, rather than the solution at one parameter point – exploiting the fact that neural networks scale cheaply to extra inputs. Because likelihood-based estimation of nonlinear models also requires a computationally costly Monte Carlo (particle) filter at every parameter draw, the paper trains a second “surrogate” neural network – the neural network particle filter – on a modest sample of particle-filter-evaluated likelihoods, giving a near-instant approximate mapping from parameters to likelihood that can be plugged into a standard Metropolis-Hastings sampler. After validating the approach on a linearized New Keynesian model (where the true solution is known analytically) and on a tractable representative-agent model with an aggregate zero-lower-bound nonlinearity (where results closely match a conventional particle-filter estimation), the paper applies its method to a fully nonlinear Heterogeneous Agent New Keynesian (HANK) model with 100 households, idiosyncratic labor-productivity risk, an individual borrowing limit, and a zero lower bound on the nominal interest rate – a model with hundreds of state and pseudo-state variables and 12 estimated structural parameters, including parameters that directly govern the degree of household heterogeneity. Using the calibrated model as the true data-generating process and 500 simulated periods of output growth, inflation, and the interest rate, a 1-million-draw Bayesian estimation recovers all 12 parameters, with the true value falling inside the 90% credible interval in every case, completed in under two days on a modern desktop computer – what the authors describe as the first estimation of a HANK model in its fully nonlinear specification. The exercise also reveals a strong asymmetry in identification: parameters governing aggregate dynamics (habit formation, price-adjustment costs, monetary-policy responses, shock persistences) are estimated precisely, while the posteriors for parameters governing idiosyncratic risk and the borrowing limit are “rather flat,” suggesting standard macro aggregate data contains little information about the underlying degree of heterogeneity and that richer, distributional data would likely be needed to pin these down.

Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

Standard recursive formulations of heterogeneous-agent models with aggregate risk force the entire cross-sectional distribution of agents into the Bellman equation characterizing individual decisions – the “Master equation” – purely because low-dimensional equilibrium prices, unlike the distribution itself, do not follow a Markov process, so rational agents forecasting prices end up needing to forecast the whole distribution. This extreme curse of dimensionality remains the central computational bottleneck for global solutions of heterogeneous-agent models, so severe that even a Huggett (1993) model with aggregate risk – despite looking simple – proved impossible for any team to solve in an influential benchmarking exercise and was dropped from the project altogether. This paper sidesteps the Master equation entirely using ideas from reinforcement learning (RL): agents learn equilibrium price dynamics directly from simulated paths, as standard RL would, but the paper’s “structural reinforcement learning” (SRL) approach departs from standard RL by assuming agents have structural knowledge of their own individual-state dynamics (their budget constraint and idiosyncratic income process), letting the authors compute exact policy gradients by differentiating through these known dynamics rather than relying on the noisy, approximate policy gradients standard RL methods estimate; only the equilibrium price process itself is treated as unknown and learned from simulation. By further restricting agents to condition their policies only on current (or briefly lagged) prices rather than the full price history or the distribution, the paper solves for a low-dimensional “restricted perceptions equilibrium” in the sense of Sargent (1991) rather than the full rational-expectations equilibrium – expectations are restricted in functional form but remain statistically consistent with actual outcomes. Because policy functions depend only on prices, they double as individual supply/demand schedules that can be integrated across the distribution and market-cleared period-by-period along a simulation, treating market clearing as part of the “environment” (in RL parlance) rather than something solved inside an optimization loop – which is what lets the method efficiently handle nontrivial market-clearing conditions that have historically been very hard. Implemented in JAX on a single GPU, the resulting structural policy gradient (SPG) algorithm solves the Krusell and Smith (1998) model in about 55 seconds, the previously-unsolved Huggett (1993) model with aggregate risk in around one minute, and a one-asset HANK model with a forward-looking New Keynesian Phillips curve in around three minutes – with the Krusell-Smith solution closely matching alternative global solutions of the rational-expectations equilibrium, and allowing agents a longer history of lagged prices barely moving the solution, indicating most of the information relevant for forecasting prices is already contained in current prices. The paper is explicit that its algorithm, as presented, is not itself intended as an empirically realistic theory of how real economic agents form expectations, though it suggests the “sampling”-based logic behind SRL could in principle be developed into one.