Quality Adjustment at Scale: Hedonic versus Exact Demand-Based Price Indices

Q1. What problem in price index construction does the paper address?

The paper addresses the long-standing challenge of simultaneously accounting for consumer substitution and quality change due to product entry and exit in official price statistics, and shows these two corrections are now feasible to implement in real time from item-level retail transactions data. Standard official statistics (CPI, PCE) use an arithmetic Laspeyres index that holds spending weights fixed — the Boskin Commission documented this overstates the true cost of living due to substitution bias. Scanner data permit superlative indices (Tornqvist, Fisher) that correct for substitution at the item level, but further correcting for quality change requires dealing with the millions of products that enter and exit the market each quarter. The paper’s contribution is to show that both corrections can be combined at scale.

Q2. What data infrastructure does the paper use, and what does it reveal about product turnover?

The NPD Group data cover five product groups with quarterly turnover rates of 4.5–13.5 percent per quarter (both entry and exit), and exhibit a characteristic life-cycle pattern: prices peak at entry and decline steadily thereafter while market shares follow a hump shape — rising as products gain distribution and then declining as newer products displace them. Memory cards, for example, show approximately a 50 percent price decline over their life cycle and a 200 percent increase in market share in the first year after entry. These interrelated price-quantity dynamics mean that any matched-model index that ignores entering and exiting goods misses substantial quality improvement. The NielsenIQ data cover over 2.6 million UPCs across 40,000 stores, with nominal food sales closely tracking BEA PCE food expenditures, validating its representativeness.

Q3. What is the EP-TV hedonic method and why does the paper prefer it?

The Erickson-Pakes time-varying unobservables (EP-TV) method estimates hedonic price indices from item-level transactions data using a two-step procedure: first predict log price levels from observable characteristics to recover item-level residuals, then predict log price changes from characteristics plus the lagged residual, which allows the model to track changing valuations of unobservable attributes over time. This approach outperforms the simpler log-level hedonic and the EP-F (fixed unobservables) approach in model fit for price changes: EP-TV achieves R² of 0.13–0.50 versus 0.05–0.24 for log-level models across product groups. The paper extends EP-TV to the NielsenIQ data using deep neural networks to decode sparse abbreviated product descriptions (e.g., “ZR DT LN/LM CF NBP CT” for a diet soft drink), achieving out-of-sample R² of roughly 75% for price-level predictions and above 50% in-sample for price changes.

Q4. What are the main quantitative findings for the hedonic indices?

The EP-TV hedonic Tornqvist index indicates price declines approximately 2.9 percentage points per year faster than the matched-model Tornqvist for memory cards, 2.5 pp/year for headphones, 1.3 pp/year for boys’ jeans, 0.7 pp/year for coffee makers, and 0.4 pp/year for occupational footwear; for NielsenIQ food categories, the hedonic Tornqvist is approximately 4 percentage points lower cumulatively over 2006–2015 than the matched-model Tornqvist. These gaps represent the quality improvement embedded in product turnover — the fact that new memory cards at a given price embody more storage than their predecessors, new headphones better sound quality, etc. The paper also shows that the hedonic Laspeyres, which only adjusts for exiting goods (following the standard Pakes 2003 bounding result), is substantially lower than the matched-model Laspeyres, confirming that the selection bias from ignoring exiting goods in official statistics is quantitatively important.

Q5. How do the demand-based exact price indices compare with the hedonic approach?

The Feenstra (1994) lambda-ratio index, which adjusts the Sato-Vartia index for product entry and exit via the ratio of entering to exiting expenditure shares scaled by (1/(σ−1)), shows cumulative disinflation approximately 2 percentage points beyond the matched-model Sato-Vartia across all five NPD product groups and approximately 5 percentage points for NielsenIQ food, broadly comparable in magnitude to the hedonic adjustment. The estimated substitution elasticities (σ) range from about 5.2 to 7.8 across NPD product groups and have a median of about 6 across food product groups, consistent with the literature. The Feenstra-hedonic agreement is a reassuring finding: two methodologically distinct approaches based on different identifying assumptions yield similar magnitudes of quality correction.

Q6. What is the CUPI and why is it problematic?

The Redding-Weinstein (2020) CES Unified Price Index (CUPI) generalizes the Feenstra index by adding a taste-shock correction (S ratio) and a Jevons index (P ratio), both of which are unweighted geometric means across common goods — making the CUPI extremely sensitive to products with tiny expenditure shares, because any product with a low share is inferred to have low appeal, which the model translates into a large quality-adjustment downward.** Without a common-goods rule (CGR), the CUPI shows 30–40 percent per year price declines for high-tech goods and boys’ jeans, 10–30 percentage points below the Feenstra; with a 25th-percentile market-share CGR, the CUPI is still more than 40 percentage points below the Feenstra for food in 2015 on a cumulative basis. The key concern is that very low expenditure shares for entering or exiting products can reflect search frictions, limited distribution, or clearance-rack effects rather than genuinely low consumer appeal, so the CUPI’s unweighted components may conflate these factors with quality change. The paper concludes that more research on the appropriate CGR is needed before the CUPI can be recommended for official statistics.

Q7. Is the hedonic approach robust to missing or omitted product attributes?

Yes — omitting key observable characteristics (memory size for memory cards, major brand dummies for apparel and footwear) from the EP-TV estimation has only minimal effect on the resulting hedonic price index, in contrast to log-level hedonic models where such omissions produce much larger distortions. For memory cards, the baseline EP-TV Tornqvist index produces an average annual cumulative chained price change of −20.12%; excluding entering products whose size or speed is outside the range of continuing products changes this to −20.09%, even though roughly 50% of entering products (accounting for 25% of entering-product sales) are excluded. This robustness reflects the EP-TV design: the first-stage residual absorbs time-varying unobservable characteristics that would otherwise confound the hedonic mapping.

Q8. What are the implications for official statistics?

The paper argues that adopting hedonic superlative price indices from real-time scanner data would produce official CPI and PCE price measures that simultaneously correct for substitution bias and quality change, resulting in meaningfully lower measured inflation in categories with high product turnover — likely understating quality-adjusted price declines in current official statistics by several percentage points per year in high-tech consumer goods and by roughly half a percentage point per year in food. The practical case for adoption is that the EP-TV approach can be implemented across heterogeneously encoded data (both structured NPD attributes and unstructured NielsenIQ text), is feasible in real time with transaction data, is relatively insensitive to chain drift (full-imputation hedonic indices avoid the transitory price volatility that contaminates matched-model chained indices), and satisfies bounding properties under general conditions established by Pakes (2003). The paper thus operationalizes long-standing recommendations of the Boskin Commission (1998) that called for exactly these corrections.

Key Concepts

EP-TV hedonic index : the Erickson-Pakes time-varying unobservables hedonic price index, which imputes quality-adjusted price changes for entering and exiting products using a two-step regression that includes lagged residuals from a log-level hedonic to control for products whose unobservable attributes change their market valuations over time; the paper’s preferred hedonic approach.

Feenstra (1994) lambda-ratio adjustment : a correction to the Sato-Vartia CES price index that accounts for product entry and exit by multiplying by (λ_{t,t-1}/λ_{t-1,t})^{1/(σ-1)}, where the lambda terms are the expenditure shares of continuing goods relative to all goods in each period; larger entry shares relative to exit shares produce a downward adjustment reflecting quality improvement from new products.

CES Unified Price Index (CUPI) : the Redding-Weinstein (2020) extension of the Feenstra index that additionally incorporates time-varying product appeal shocks via an unweighted Jevons index (P* ratio) and an unweighted expenditure-share ratio (S* ratio); the paper finds this index is highly sensitive to the common-goods rule and may overstate quality adjustment in practice due to sensitivity to low-share goods.

common-goods rule (CGR) : a threshold rule that restricts the set of goods entering the CUPI’s unweighted components to those with sufficiently large or long-duration market shares, introduced by Redding and Weinstein (2020) to limit the influence of fringe products; the paper finds CUPI results are sensitive to the CGR specification in a way that varies across product groups.

full-imputation hedonic index : a hedonic price index that uses the hedonic mapping to impute price changes for all goods (including continuing goods), rather than only entering and exiting goods; reduces chain drift relative to partial-imputation approaches because imputed prices are less volatile than observed prices.

product turnover : the quarterly entry and exit of products from the market, ranging from 4.5% to 13.5% per quarter in NPD data; the primary mechanism through which quality change is embedded in item-level scanner data and the main source of mismeasurement in matched-model price indices.