C81 | Macro Paper Warehouse

Enlightenment Ideals and Belief in Progress in the Run-up to the Industrial Revolution

Mon, 01 Jan 0001 00:00:00 +0000

This paper tests Joel Mokyr’s claim that Britain’s industrialization was preceded and enabled by a cultural shift — specifically, that Enlightenment ideals produced a “progress-oriented” view of science that diffused to artisans and craftsmen. The central research question is whether and when the language of science became more progress-oriented in the build-up to the Industrial Revolution, and whether this shift was concentrated in volumes directly linked to industrial production.

The authors assemble 173,031 unique volumes printed in England and written in English between 1500 and 1900, drawn from the Hathitrust Digital Library. Because copyright law prohibits downloading full text, they use HDL’s Extracted-Features “bag of words” dataset. After removing duplicates and Latin-language volumes from an initial set of 420,081, they apply Latent Dirichlet Allocation (LDA) with cross-validated perplexity minimization to identify an optimal T=60 topics. Topic-pair co-occurrence analysis identifies three categories — science, religion, and political economy — each anchored by three defining topics. Volume-level category weights are derived by multiplying each topic’s weight by its category coefficient. The resulting classification yields 50,090 science volumes, 102,565 political economy volumes, and 14,124 religion volumes.

Progressive sentiment is measured using a seven-word dictionary (progress, improvement, stride, betterment, advance, rise, amelioration) assembled from thesaurus synonyms for “progress,” manually vetted by all four authors, and restricted to words attested in the Oxford English Dictionary before 1643 (Newton’s birth year). Sentiment for each volume equals the count of progress-dictionary words divided by total word count. An analogous optimism-sentiment placebo dictionary is constructed separately.

Industrial relevance is scored using the digitized indexes of all five volumes of Appleby’s Illustrated Handbook of Machinery (1877–1903); the top industrial root words are crane (weight 51), electr (42), weight (37), rope (27), and cost (27). Each volume receives an industry score equal to the weighted occurrence of industrial root words normalized by volume length.

Three main findings emerge. First, the language of science and religion showed little overlap beginning in the 17th century — that is, the secularization of science predates the onset of industrialization. Science volumes shifted from approximately 40 percent religious content around 1700 to only about 10 percent by 1850, with scientific content rising correspondingly from roughly 40 percent to over 60 percent. This trend was stable from 1650 through 1900.

Second, while scientific volumes became more progress-oriented during the Enlightenment, this progressive shift was concentrated in volumes at the nexus of science and political economy. Volumes of “pure” science were largely neutral with respect to progress sentiment, and those at the science-religion nexus had on average negative progress sentiment. The marginal effect of scientific content on progress sentiment was greatest for volumes mixing science and political economy, and most of the increase in predicted sentiment at that nexus occurred during the 18th century, remaining stable thereafter. A placebo test using optimism sentiment finds the opposite pattern: volumes at the science-political economy nexus were among the least optimistic, while the most optimistic language appeared at the religion-political economy nexus. This rules out the interpretation that the measured shift reflects a general increase in positive affect rather than specifically progress-oriented language.

Third, volumes employing industrial terminology that also sat at the science-political economy nexus were distinctively progressive beginning in the mid-18th century. At the 90th percentile of industry score, predicted progress sentiment at the science-political economy nexus was positive throughout the sample; at zero industry score, it was negative until the mid-18th century. Volumes at the religion-political economy nexus showed modestly positive and time-stable progress sentiment regardless of industry score.

The paper concludes that it was the pragmatic, applied volumes — those bridging science and political economy, written for artisans and a broader literate public rather than for the human-capital elite alone — that embodied the cultural values Mokyr identifies as central to Britain’s industrialization.

Q: What gap in the existing literature does this paper address?

A: Prior work on the cultural deep roots of economic growth rarely tracks how culture changes over time, relying instead on cross-sectional variation or qualitative case studies. Quantitative evidence that the language of science itself became more progress-oriented — and that this change reached beyond elite thinkers to artisans and craftsmen — had not been marshaled before. The paper provides inaugural quantitative support by analyzing 173,031 volumes spanning four centuries.

Q: Why does the paper restrict the progress-sentiment dictionary to words attested before 1643?

A: Words that entered English only after 1643 (Newton’s birth year) could not have appeared in volumes from the early Enlightenment, so including them would bias sentiment scores toward the later part of the sample. The restriction ensures the dictionary is applicable and unbiased across the full 1500–1900 period. The final retained words are: progress, improvement, stride, betterment, advance, rise, amelioration.

Q: How does LDA classify volumes, and how is T=60 selected?

A: LDA treats each volume as a bag of words and derives a Dirichlet distribution such that observed documents are generated by repeated topic sampling. The number of topics T is selected by minimizing perplexity on held-out data via 4-fold cross-validation, rotating training and test sets across folds; this procedure yields T=60 as optimal. Each volume is then represented as a mixture over those 60 topics.

Q: What are the three categories and their anchor topics?

A: Political Economy is anchored by topics on law/public opinion, governance/parliament, and trade/price/labour. Religion is anchored by topics on church/Christian doctrine, God/faith/sin, and virtue/fame/religion. Science is anchored by topics on engineering/steam/electricity, chemistry/acid/heat, and geometry/equations/trigonometry. These three sets of topics were selected for high corpus-wide importance and mutual independence.

Q: What does the finding on science-religion separation imply for timing?

A: The separation of scientific and religious language was already visible by 1600 and firmly established by the mid-17th century, well before the Industrial Revolution conventionally dated to the mid-18th century. This supports Mokyr’s argument that the secularization of science was an Enlightenment-era precursor to industrialization rather than a product of it. The trend remained stable from 1650 through 1900.

Q: How does the progressive sentiment differ between pure science and the science-political economy nexus?

A: Volumes of pure science were largely neutral with respect to progress-oriented language and in some periods showed slightly negative predicted progress sentiment. The science-religion nexus showed consistently negative progress sentiment. By contrast, volumes at the science-political economy nexus showed the highest level of progressive sentiment beginning in the mid-18th century, and most of this growth in predicted sentiment occurred during the 18th century, after which it remained stable.

Q: What does the placebo optimism test show?

A: The optimism sentiment scores are nearly the mirror opposite of the progress scores: the most optimistic language appears at the religion-political economy nexus, while volumes at the science-political economy nexus are among the least optimistic. This dissociation rules out the interpretation that the measured progress-sentiment rise reflects a general shift toward positive language rather than a specific cultural embrace of science as a tool for improving human welfare.

Q: How is the industrial score constructed and what are the most heavily weighted terms?

A: The authors digitized the detailed indexes of all five volumes of Appleby’s Illustrated Handbook of Machinery (1877–1903), restricted to words attested before 1643, and weighted each industrial root word by its index frequency. Each corpus volume’s industry score equals the sum of (word count × index weight) across all industrial words, normalized by volume length, yielding a score between 0 and 1. The top-weighted terms are crane (51), electr (42), weight (37), rope (27), and cost (27).

Q: What is the key result linking industrial scores to progressive sentiment?

A: At the science-political economy nexus, volumes with industry scores at the 90th percentile had persistently positive predicted progress sentiment throughout the sample, while volumes at that nexus with zero industry score had negative predicted sentiment until the mid-18th century. The shift to positive sentiment for high-industry volumes at this nexus occurred in the mid-18th century — roughly coinciding with the onset of Britain’s industrialization — and those volumes remained the most progress-oriented in the corpus thereafter.

Q: What is the paper’s interpretation of the science-political economy nexus finding in relation to Mokyr?

A: The authors interpret volumes at the science-political economy nexus as pragmatic, applied works aimed at a broader literate audience including artisans and craftsmen, not exclusively the human-capital elite. These are precisely the volumes Mokyr’s “Industrial Enlightenment” thesis predicts would carry progress-oriented cultural values into the mechanical and artisanal pursuits that drove industrialization. The finding that pure-science volumes were not especially progressive, while applied volumes bridging science and political economy were, is consistent with Mokyr’s argument that it was the diffusion of Enlightenment ideals to skilled practitioners — not just to elite scientists — that mattered.

Q: What qualitative examples support the quantitative findings?

A: Martin Clare’s The Motion of Fluids (1735) explicitly addresses “the Unlearned” and states in its preface that the work is meant to be “of singular Use and Benefit to Mankind” — a direct expression of the progress-oriented language the algorithm detects. George Stephenson’s 1831 railway report argues that rail infrastructure would allow Ireland to “reciprocate with England and with other nations, the products of industry,” exemplifying how progress-oriented language pervaded industrial writing by the early 19th century. These examples confirm that the high progress-sentiment scores for industrial volumes at the science-political economy nexus reflect genuine rhetorical content, not measurement artifacts.

Q: What are the paper’s limitations regarding early sample periods?

A: The corpus is thin in earlier eras, particularly around 1550, so results from the earliest decades must be interpreted with caution. The HDL data derive from digitized scans with OCR output of very old books, introducing errors such as the “long-S” misread (e.g., “juftice” for “justice”) that require manual correction. Additionally, the bag-of-words model discards word order, which may obscure some semantic distinctions.

Q: What future research directions do the authors identify?

A: The authors propose applying the same textual analysis techniques to test whether English-language volumes began reflecting greater freedom of expression in the run-up to Britain’s economic takeoff, connecting to the literature on European political fragmentation and the marketplace of ideas. They also suggest applying the approach to corpora in other languages — Dutch (following McCloskey’s argument about bourgeois values) and Spanish (to examine whether the Counter-Reformation and Spain’s economic lag are reflected in cultural attitudes toward progress and science).

LDA (Latent Dirichlet Allocation): An unsupervised generative statistical model that treats each document as a bag of words and extracts latent topics as multinomial distributions over vocabulary; used here to reduce 173,031 volumes to mixtures of 60 topics without imposing prior scholarly interpretations.

Progressive Sentiment Score: The fraction of words in a volume belonging to a seven-word dictionary of progress synonyms (progress, improvement, stride, betterment, advance, rise, amelioration), normalized by total word count; measures the cultural orientation toward the betterment of humankind as embedded in text.

Industrial Score: A volume-level measure equal to the weighted count of industrial root words — derived from the indexes of Appleby’s Illustrated Handbook of Machinery (1877–1903) — normalized by volume length; captures the degree to which a volume’s vocabulary overlaps with industrial production terminology.

Science-Political Economy Nexus: The region of the topic simplex where volumes carry substantial weight in both the science and political economy categories but low weight in religion; the paper finds this is where progress-oriented language was most concentrated from the mid-18th century onward, interpreted as applied science aimed at artisans and a broader literate public.

Industrial Enlightenment: Joel Mokyr’s (2009) concept describing the diffusion of Enlightenment ideals about the practical utility of science into the mechanical and artisanal pursuits that drove Britain’s industrialization; the paper provides quantitative support for this thesis by showing that industrial volumes at the science-political economy nexus were distinctively progress-oriented.

Culture of Growth: Mokyr’s (2016) broader argument that a pan-European network of elite intellectuals fostered a progress-oriented view of science — the idea that scientific understanding could improve the human condition — and that this cultural norm, in combination with Britain’s stock of skilled craftsmen, made industrialization possible.

Bag of Words: A representation of text that records only word frequencies within a document, discarding word order; used here both because HDL copyright restrictions prevent full-text download and because it is the input format required by LDA.

Quality Adjustment at Scale: Hedonic versus Exact Demand-Based Price Indices

Mon, 01 Jan 0001 00:00:00 +0000

Layer 1: Overview

This paper implements and evaluates methods for constructing quality-adjusted price indices from item-level retail scanner data at scale — across hundreds of product categories, heterogeneously encoded product attributes, and rapid product turnover. Using proprietary NPD Group data covering five general merchandise categories (2014–2018) and NielsenIQ scanner data covering 50+ food product groups (2006–2015), the paper compares hedonic superlative indices (using the Erickson-Pakes EP-TV methodology and a novel machine-learning extension) against exact demand-based indices (the Feenstra 1994 lambda-ratio adjustment and the Redding-Weinstein 2020 CES Unified Price Index, CUPI). The central finding is that quality adjustment is quantitatively large: the hedonic Tornqvist index shows roughly 2.5–2.9 percentage points per year faster price decline than the matched-model Tornqvist in high-tech categories (headphones, memory cards) and 4–5 percentage points of cumulative additional disinflation relative to matched-model indices for food. The Feenstra index agrees closely in magnitude with the hedonic approach, but the CUPI is highly sensitive to the choice of common-goods rule (CGR) and in some specifications shows 20–40 percentage points more cumulative disinflation than the Feenstra, a gap the paper attributes to the CUPI’s extreme sensitivity to low-share goods. The paper establishes that hedonic superlative price indices are feasible to implement at scale, including with machine learning on sparse text descriptions, and recommends them as the practical benchmark for re-engineering official price statistics.

Summary of a published paper, AI-assisted and human-reviewed. See the linked original for the authoritative claims and full conditions.

In depth

Q1. What problem in price index construction does the paper address?

The paper addresses the long-standing challenge of simultaneously accounting for consumer substitution and quality change due to product entry and exit in official price statistics, and shows these two corrections are now feasible to implement in real time from item-level retail transactions data. Standard official statistics (CPI, PCE) use an arithmetic Laspeyres index that holds spending weights fixed — the Boskin Commission documented this overstates the true cost of living due to substitution bias. Scanner data permit superlative indices (Tornqvist, Fisher) that correct for substitution at the item level, but further correcting for quality change requires dealing with the millions of products that enter and exit the market each quarter. The paper’s contribution is to show that both corrections can be combined at scale.

Q2. What data infrastructure does the paper use, and what does it reveal about product turnover?

The NPD Group data cover five product groups with quarterly turnover rates of 4.5–13.5 percent per quarter (both entry and exit), and exhibit a characteristic life-cycle pattern: prices peak at entry and decline steadily thereafter while market shares follow a hump shape — rising as products gain distribution and then declining as newer products displace them. Memory cards, for example, show approximately a 50 percent price decline over their life cycle and a 200 percent increase in market share in the first year after entry. These interrelated price-quantity dynamics mean that any matched-model index that ignores entering and exiting goods misses substantial quality improvement. The NielsenIQ data cover over 2.6 million UPCs across 40,000 stores, with nominal food sales closely tracking BEA PCE food expenditures, validating its representativeness.

Q3. What is the EP-TV hedonic method and why does the paper prefer it?

The Erickson-Pakes time-varying unobservables (EP-TV) method estimates hedonic price indices from item-level transactions data using a two-step procedure: first predict log price levels from observable characteristics to recover item-level residuals, then predict log price changes from characteristics plus the lagged residual, which allows the model to track changing valuations of unobservable attributes over time. This approach outperforms the simpler log-level hedonic and the EP-F (fixed unobservables) approach in model fit for price changes: EP-TV achieves R² of 0.13–0.50 versus 0.05–0.24 for log-level models across product groups. The paper extends EP-TV to the NielsenIQ data using deep neural networks to decode sparse abbreviated product descriptions (e.g., “ZR DT LN/LM CF NBP CT” for a diet soft drink), achieving out-of-sample R² of roughly 75% for price-level predictions and above 50% in-sample for price changes.

Q4. What are the main quantitative findings for the hedonic indices?

The EP-TV hedonic Tornqvist index indicates price declines approximately 2.9 percentage points per year faster than the matched-model Tornqvist for memory cards, 2.5 pp/year for headphones, 1.3 pp/year for boys’ jeans, 0.7 pp/year for coffee makers, and 0.4 pp/year for occupational footwear; for NielsenIQ food categories, the hedonic Tornqvist is approximately 4 percentage points lower cumulatively over 2006–2015 than the matched-model Tornqvist. These gaps represent the quality improvement embedded in product turnover — the fact that new memory cards at a given price embody more storage than their predecessors, new headphones better sound quality, etc. The paper also shows that the hedonic Laspeyres, which only adjusts for exiting goods (following the standard Pakes 2003 bounding result), is substantially lower than the matched-model Laspeyres, confirming that the selection bias from ignoring exiting goods in official statistics is quantitatively important.

Q5. How do the demand-based exact price indices compare with the hedonic approach?

The Feenstra (1994) lambda-ratio index, which adjusts the Sato-Vartia index for product entry and exit via the ratio of entering to exiting expenditure shares scaled by (1/(σ−1)), shows cumulative disinflation approximately 2 percentage points beyond the matched-model Sato-Vartia across all five NPD product groups and approximately 5 percentage points for NielsenIQ food, broadly comparable in magnitude to the hedonic adjustment. The estimated substitution elasticities (σ) range from about 5.2 to 7.8 across NPD product groups and have a median of about 6 across food product groups, consistent with the literature. The Feenstra-hedonic agreement is a reassuring finding: two methodologically distinct approaches based on different identifying assumptions yield similar magnitudes of quality correction.

Q6. What is the CUPI and why is it problematic?

The Redding-Weinstein (2020) CES Unified Price Index (CUPI) generalizes the Feenstra index by adding a taste-shock correction (S ratio) and a Jevons index (P ratio), both of which are unweighted geometric means across common goods — making the CUPI extremely sensitive to products with tiny expenditure shares, because any product with a low share is inferred to have low appeal, which the model translates into a large quality-adjustment downward.** Without a common-goods rule (CGR), the CUPI shows 30–40 percent per year price declines for high-tech goods and boys’ jeans, 10–30 percentage points below the Feenstra; with a 25th-percentile market-share CGR, the CUPI is still more than 40 percentage points below the Feenstra for food in 2015 on a cumulative basis. The key concern is that very low expenditure shares for entering or exiting products can reflect search frictions, limited distribution, or clearance-rack effects rather than genuinely low consumer appeal, so the CUPI’s unweighted components may conflate these factors with quality change. The paper concludes that more research on the appropriate CGR is needed before the CUPI can be recommended for official statistics.

Q7. Is the hedonic approach robust to missing or omitted product attributes?

Yes — omitting key observable characteristics (memory size for memory cards, major brand dummies for apparel and footwear) from the EP-TV estimation has only minimal effect on the resulting hedonic price index, in contrast to log-level hedonic models where such omissions produce much larger distortions. For memory cards, the baseline EP-TV Tornqvist index produces an average annual cumulative chained price change of −20.12%; excluding entering products whose size or speed is outside the range of continuing products changes this to −20.09%, even though roughly 50% of entering products (accounting for 25% of entering-product sales) are excluded. This robustness reflects the EP-TV design: the first-stage residual absorbs time-varying unobservable characteristics that would otherwise confound the hedonic mapping.

Q8. What are the implications for official statistics?

The paper argues that adopting hedonic superlative price indices from real-time scanner data would produce official CPI and PCE price measures that simultaneously correct for substitution bias and quality change, resulting in meaningfully lower measured inflation in categories with high product turnover — likely understating quality-adjusted price declines in current official statistics by several percentage points per year in high-tech consumer goods and by roughly half a percentage point per year in food. The practical case for adoption is that the EP-TV approach can be implemented across heterogeneously encoded data (both structured NPD attributes and unstructured NielsenIQ text), is feasible in real time with transaction data, is relatively insensitive to chain drift (full-imputation hedonic indices avoid the transitory price volatility that contaminates matched-model chained indices), and satisfies bounding properties under general conditions established by Pakes (2003). The paper thus operationalizes long-standing recommendations of the Boskin Commission (1998) that called for exactly these corrections.

Key Concepts

EP-TV hedonic index : the Erickson-Pakes time-varying unobservables hedonic price index, which imputes quality-adjusted price changes for entering and exiting products using a two-step regression that includes lagged residuals from a log-level hedonic to control for products whose unobservable attributes change their market valuations over time; the paper’s preferred hedonic approach.

Feenstra (1994) lambda-ratio adjustment : a correction to the Sato-Vartia CES price index that accounts for product entry and exit by multiplying by (λ_{t,t-1}/λ_{t-1,t})^{1/(σ-1)}, where the lambda terms are the expenditure shares of continuing goods relative to all goods in each period; larger entry shares relative to exit shares produce a downward adjustment reflecting quality improvement from new products.

CES Unified Price Index (CUPI) : the Redding-Weinstein (2020) extension of the Feenstra index that additionally incorporates time-varying product appeal shocks via an unweighted Jevons index (P* ratio) and an unweighted expenditure-share ratio (S* ratio); the paper finds this index is highly sensitive to the common-goods rule and may overstate quality adjustment in practice due to sensitivity to low-share goods.

common-goods rule (CGR) : a threshold rule that restricts the set of goods entering the CUPI’s unweighted components to those with sufficiently large or long-duration market shares, introduced by Redding and Weinstein (2020) to limit the influence of fringe products; the paper finds CUPI results are sensitive to the CGR specification in a way that varies across product groups.

full-imputation hedonic index : a hedonic price index that uses the hedonic mapping to impute price changes for all goods (including continuing goods), rather than only entering and exiting goods; reduces chain drift relative to partial-imputation approaches because imputed prices are less volatile than observed prices.

product turnover : the quarterly entry and exit of products from the market, ranging from 4.5% to 13.5% per quarter in NPD data; the primary mechanism through which quality change is embedded in item-level scanner data and the main source of mismeasurement in matched-model price indices.