Barton Friedland is the founder of Luminous Group, where he builds architecture that sustains human judgment as AI scales. For his doctoral research at Warwick Business School, he studied how computational technologies participate in leadership and organizational decision-making.
Editor’s note: Noema is transparent about any AI use in its pieces. We publish original, human-generated ideas but allow authorized, disclosed use of AI in certain cases. Please see details and our policy at the end of this piece.
When a human works with an AI system, something emerges that exists in neither participant. It is not consciousness; the machine does not feel anything. But it is not mere computation either. It is enacted intelligence: situated, distributed, directional and irreducible to either party.
If AI is not conscious, then every act of human-AI collaboration places a specific demand on the human participant. The human must continuously project, test and stabilize meaning within the collaborative field. Context drifts. Coherence flattens. Outputs that appeared aligned with the human’s intention subtly diverge. The human must notice, re-anchor and redirect. This is not a bug of current AI systems that better engineering will resolve. It is a permanent structural feature of working with entities that process without understanding.
Anyone who works seriously with AI systems knows this well. The experience is cognitively demanding. It requires sustained attention, interpretive judgment and a particular kind of presence — the willingness to hold meaning steady across interactions that do not hold it for you. It is, in the precise sense of the word, work: the cognitive labor of maintaining coherence in a field that offers no guarantee of it.
This work is economically valuable. In a randomized clinical trial conducted remotely and in-person at Stanford, Beth Israel Deaconess Medical Center and the University of Virginia, physicians who were given access to GPT-4 alongside conventional diagnostic resources were no more accurate in their diagnoses than physicians without it — even as GPT-4 alone outperformed both groups by more than 15%. The same AI. The same clinical task. No significantly measurable benefit — because the arrangement was naive: The technology was bolted onto existing workflows with no designed interaction, no structured dialogue, no preservation of the clinician’s independent reasoning.
When the collaboration was redesigned — requiring clinician and AI to generate independent assessments, then structuring a dialogue that surfaced disagreements and held the clinician’s reasoning in the loop — diagnostic accuracy rose from 75% without AI to 82 to 85% with the collaborative AI. The difference was in the quality of the human-AI arrangement: whether the human’s judgment was preserved, amplified and compounded by the collaboration, or bypassed, flattened and ultimately eroded.
Researchers at the Stockholm School of Economics and the University of Geneva deployed the same AI system to pharmaceutical sales professionals. The results were stark. When the system was tailored to the expert’s cognitive style — structuring authority, workflows and incentives to preserve expert judgment — on average, client meetings rose more than 40% and sales rose 16%. When the same system was imposed without regard for human thinking, sales fell about 20% below the no-AI baseline. Worse than no AI at all.
In strategic consulting, the evidence acquires a sharper edge. Researchers at Harvard, MIT, Penn and the University of Warwick studied 758 Boston Consulting Group consultants working with and without AI. For tasks the AI could handle well, those working with AI had, on average, a 12% higher completion rate, finished 25% faster and produced 40% higher quality results. But for tasks requiring the kind of judgment that AI processes but cannot itself exercise, AI-assisted consultants performed significantly worse than those working alone. The technology did not fail. The arrangement did. When humans deferred to computational fluency on tasks where human judgment was needed most, the collaboration became a liability.
“The mythology of automation is the belief that removing the human from the loop is always an efficiency gain.”
There are early signs that these assumptions are already reshaping labor markets. A 2025 study by the Stanford Digital Economy Lab found a 16% relative decline in employment among workers aged 22 to 25 in AI-exposed occupations, even as employment among more experienced workers remained comparatively stable. The concern is not simply job displacement. It is that organizations may be automating the very tasks through which people acquire experience, creating a future shortage of the judgment they still depend upon. The strategic challenge is deciding which forms of human capability organizations cannot afford to lose.
Some firms are already finding out the hard way. In 2024, for example, the Swedish fintech company Klarna laid off hundreds of customer service employees and outsourced their work to AI chatbots. By 2025, it had reversed that decision and rehired human agents, acknowledging that aggressive automation had reduced service quality and customer satisfaction. The technology had not failed; the value had been misidentified. Rather than enhancing human capability with AI, companies like Klarna have replaced human contributions in pursuit of efficiency gains — eliminating the judgment, contextual interpretation, relationship management and forms of situated intelligence that had quietly been generating value all along.
The lesson extends beyond any single company. The question is not whether humans or AI are nominally “in charge.” It is whether the arrangement preserves the conditions under which human judgment remains active and capable of shaping outcomes. Defense strategists distinguish “centaur” systems, where the human directs and the machine executes, from “minotaur” systems, where the AI directs and the human carries out its recommendations. AI researcher Ethan Mollick has observed that the worst outcomes occur when people “fall asleep at the wheel,” ceding judgment to the system precisely when judgment is most needed.
Lucy Suchman, who studies human-computer interactions, has argued that autonomy is not a property of either humans or machines, but of the configurations in which they operate. The significance of this insight is easy to miss. It suggests that success or failure does not reside in the intelligence of the model, nor in the capability of the human, but in the quality of the relationship between them. Human-AI systems create value when they preserve the human capacity to interpret context, exercise judgment and intervene when conditions change. They destroy value when those capacities are designed out of the arrangement.
The absence of consciousness in AI is not merely a philosophical finding. It is a design condition with measurable economic consequences. It means that every human-AI system must be architected to preserve the human’s capacity for meaning-making, judgment and coherence-holding, because no one else in the arrangement will do it.
The mythology of automation is the belief that removing the human from the loop is always an efficiency gain. That judgment is a cost to be eliminated. That capability is a fixed input rather than a compounding asset. That the purpose of AI is to perform tasks currently performed by people, only faster and cheaper.
This mythology arrives dressed in the respectable garments of return-on-investment calculations, headcount reduction targets and productivity dashboards. Increasingly, it arrives dressed as augmentation itself. Major AI companies now use the word “augmentation” to describe their respective LLMs’ services while building infrastructure that moves in the opposite direction. Every significant platform release of the past two years — agent frameworks, coding agents, computer use, research agents — follows the same trajectory: Make the AI do more, make the human do less. None of them measures whether human capability is growing or atrophying. The vocabulary of augmentation has been captured. The practice it names has not. The mythology asks: What can we automate? It never asks: What are we destroying?
Last year, the inaugural Anthropic Economic Index analyzed more than 1 million conversations between people and the AI assistant Claude, categorizing each by whether the person was using AI to do the work for them — automation — or to think alongside them — augmentation. It found that 57% of the time, people were thinking with AI rather than delegating to it. And yet organizations consistently default to the other direction — not because automation delivers superior returns, but because they possess the institutional muscles for cost reduction and lack the muscles for capability cultivation. The platforms they purchase are engineered to reward exactly that default.
The economist Carl Benedikt Frey’s historical analysis in “The Technology Trap” reveals this to be a recurring pattern. The first Industrial Revolution produced what economists call “Engels’ Pause,” a period where output per worker grew by 46% while wages rose by a mere 12%. The Second Industrial Revolution, dominated by enabling technologies, produced broadly shared prosperity. The third, dominated by replacing technologies, has coincided with stagnant wages and rising inequality. The pattern is not technological but institutional: When societies build frameworks for augmentation, wealth distributes; when they default to automation, it concentrates.
We need to build institutions capable of recognizing where the value lies — not inside the machine, not inside the human skull, but in the arrangement between them. In the conditions under which judgment compounds. In the quality of attention that sustains coherence across discontinuous interactions. In the human capacity to project meaning, test it against reality and revise it under constraint. In the irreducibly relational intelligence that emerges when humans and machines coordinate with shared orientation.
“We need to build institutions capable of recognizing where the value lies — not inside the machine, not inside the human skull, but in the arrangement between them.”
The European Union has begun to recognize this. Article 14 of the EU AI Act requires that high-risk AI systems be designed so they “can be effectively overseen by natural persons.” The regulation mandates that humans must remain capable of understanding, interpreting and intervening. It does not say how. It does not specify the infrastructure that would make such oversight possible or the metrics that would demonstrate its presence. And the deadline for complying with the act’s high-risk requirements has already been pushed back — from August 2026 to December 2027 — before a single organization has been required to demonstrate that meaningful human oversight exists.
The phrase “human-centric AI” appears throughout European policy as though naming the aspiration were equivalent to building it. It is not. A foothold has been established in regulation. The architecture to stand on it has barely been imagined. It will likely not come from the companies whose revenue grows with every token consumed.
The assumption that human presence in the loop is a cost rather than the source of compounding value is a quiet, pervasive and economically devastating myth. It operates every Monday morning, in every organization that automates what it should augment, that measures what it eliminates rather than what it enables, that mistakes efficiency for intelligence and confuses the absence of friction with the presence of thought.
The question — the one that will determine whether AI becomes an engine of institutional intelligence or an accelerant of institutional decay — is whether we will learn to see, measure and preserve the conditions under which human capability compounds. And what we irrecoverably, invisibly lose when we fail to protect it.
Editor’s notes: Noema is transparent about any AI use in its pieces. We publish original, human-generated ideas but allow authorized, disclosed use of AI in certain cases. The initial submitted draft of this piece utilized Anthropic’s Claude as an editorial assistant.
Specifically, it was used for research analysis (comparing the source essay against the author’s existing body of work to identify convergences and gaps), structural development (refining the essay’s architecture and mapping which arguments carried primary weight), source verification (tracing and confirming empirical citations against published studies) and iterative drafting and revision under the author’s direction. It was not used to originate the argument, intellectual frameworks or interpretive claims. The argument was developed by the author — informed by his published research and professional practice — and directed throughout by his editorial judgment.
This draft has since received multiple human edits. Noema verified the piece’s conceptual originality using various scanners and review processes and conducted a detailed human fact-check. See our AI policy here.
This article was updated on Sept. 28, 2026, to the version edited for our print magazine. This new version was edited for greater directness and accessibility.
