Both Datasets are True
But “the data shows” is a way to stop thinking
In a rabbit hole, I read two studies this week about AI and jobs this week. First, Goldman Sachs estimated1 that AI reduced US payroll growth by roughly 16,000 jobs a month over the past year, with younger workers taking much of the hit.
Second, Ramp Economics Lab alongside Revelio looked at 21,559 US companies2 and found that the firms spending most aggressively on AI grew headcount 10.2% in the two years after adoption. Entry-level roles at those firms grew 12%.
Wait what? This is the same economy and technology, right?
The online reaction has been predictable. On one hand you have those posting Goldman number as proof that the job collapse has started. And of course others posting the Ramp numbers as proof that panic is nonsense. Both sides said, in one form or another, “the data shows.” Which is a dumb phrase that often does damage.
Not because data is bad. Data is how you escape vibes. The problem is that people use “the data shows” as a way to stop thinking. Both datasets can be true while neither argument is.
The Ramp study does not say “AI creates jobs.” It’s much more narrow than that: Firms in the top third of AI spend per employee grew faster after adoption. These firms were spending about $34 per employee per month on AI in the first three months, versus under $3 for low adopters. And low adopters showed no statistically significant headcount change.
The high adopters were already larger, more technical, more likely to be venture-backed, and faster-growing before they spent money on AI. The sample is not “the economy.” It is a set of tech-forward, knowledge-work firms. The study pushes back on a simple job-loss story.
Goldman is measuring something else. Its estimate is about net payroll growth across the whole US economy.
Then there is Stanford and ADP. Their work found that early-career workers, ages 22 to 25, in the most AI-exposed occupations saw a 16% relative employment decline. More experienced workers in those same occupations stayed stable or continued to grow.
The labor market is not moving as one thing. It is moving by firm type, age, occupation, exposure, and career stage. Strong firms that adopt AI hard can be growing. The economy-wide effect can still be modest and negative. Young workers in exposed jobs can still be taking the first hit.
Disturbance comes only from within—from our own perceptions. — IV. 3
There is a name for this…
In 1973, Berkeley looked at graduate admissions and saw a damning table. Men were admitted at a rate of 44%. Women were admitted at 35%. At the aggregate level, the data seemed to show obvious bias.
Then Peter Bickel, Eugene Hammel, and J. W. O’Connell looked department by department and found most showed no meaningful bias against women. Several tilted the other way. Women had applied more often to departments that rejected almost everyone. Men had applied more often to departments with higher admit rates.
The aggregate table was true. The department tables were true. The argument built from the aggregate table was wrong.
That is Simpson’s paradox. The same data can reverse meaning when you change the level of analysis.
This is also what is happening with AI and jobs.
Ramp answers: what happens to headcount at well-funded, tech-forward firms that adopt AI aggressively?
Goldman answers: what is happening to net payroll growth across the whole economy right now?
Stanford and ADP answer: who inside exposed occupations absorbs the adjustment first?
None of them answers the question people keep asking: will AI create or destroy jobs?
Which is really a pile of questions. By sector. By age. By firm quality. By adoption intensity. By time horizon. By substitution. By augmentation. By demand creation. By what happens to the training path for people who used to start at the bottom..
I really think a lot about that last question though (as articulated here in June) because the pain is showing up first among young workers in exposed jobs: the entry-level job was never just a bundle of tasks. It was an apprenticeship. The company got some work. The junior got reps. The senior learned to delegate. The organization manufactured judgment.
If AI removes the junior work without replacing the apprenticeship, the damage will not show up in this month’s payroll report but rather later as a missing generation of people who never got the reps.
There is one more number floating around: close to 90,000 job cuts announced this year as AI-related. Treat that state carefully because announcements measure what companies choose to say.
Some cuts are real AI substitution. Some are over-hiring, margin repair, interest-rate math, and executive fashion dressed up as AI efficiency. “We are becoming more efficient with AI” sounds better than “we misjudged 2021.”
Self-report is data but it is also the weakest data here….
“The data shows” has become a spell. People recite it to end an argument instead of using it to conduct a real one. A spell does not need to be understood. It only needs to be said with confidence in front of the right audience.
Measured what? At what level? Over what window? Against what comparison group? Who is missing from the sample? What would make the sign flip?
An incantation treats those questions as bad faith.
This is why “data-driven” has become a suspicious phrase. At its best, it means reality gets a vote. At its worst, it means a conclusion found a statistic to sit on.
Before you repeat a number, please ask one question. What question was this dataset built to answer?