AI Needs Human Judgment

The Age of Algorithmic Slop

In the late 1990s, the internet's growth created a search problem. Google's answer was the PageRank algorithm: use the social network of humans linking to each other as a proxy for authority. If many credible sites linked to you, you were probably credible too. For some time, it worked remarkably well.

PageRank implicitly assumed that creating content has a meaningful cost. Writing an article or posting a coherent comment worth linking to required both time and effort. Generative AI changed that: text costs pennies per thousand words, images take seconds to generate. Video is next—AI-generated clips of fake disasters spread virally before anyone checks.¹ The entire trust system breaks down when constructing a convincing alternate reality becomes cheap. Modern systems must measure credibility, verify factual accuracy, and trace sources.

Search quality began eroding before generated content flooded the web. Once Google captured most queries, optimizing for ad revenue displaced optimizing for users. This pattern is now repeating with LLMs. When answers come directly through chat instead of search results, the incentive shifts from placing ads to placing products in responses. Companies already market "AI optimization" services to get products recommended. The question isn't whether these incentives will corrupt AI recommendations, it's how quickly.²

The Hallucination Problem

Incentive corruption is only half the story. Even without bad actors gaming the system, LLMs fail in ways intrinsic to statistical text generation. Most people have now encountered AI hallucinations firsthand—AI suggesting pizza glue, chatbots confidently inventing legal citations. Research quantifies this: premium legal AI tools marketed as "hallucination-free" fabricated citations 17-33% of the time.³ Medical applications show similar rates of hallucination and omission.^4,5 Fabricated citations have been found in dozens of papers accepted to a top AI conference.⁶ One cause is knowledge deficiency: LLMs struggle with tasks requiring specialized reasoning.⁷

Even reassuring results on complex topics don't guarantee success on simple ones. Harvard Business School researchers documented this "jagged frontier" of AI capability.⁸ Workers using GPT-4 for tasks within its capabilities performed 40% better. For tasks just outside that frontier, they were 19% less likely to reach the correct solution. Past performance doesn't predict future reliability, and standard risk frameworks don't account for this inconsistency.

The Limits of Statistical Learning

The unpredictability of AI capability raises a deeper question: why can't these systems just learn what's true? The answer lies in how they learn, and what they learn from. Training data is noisy and often wrong, and statistical learning can't tell the difference.

Experts occasionally contradict consensus when evidence demands it. They're valuable because they've refined their mental models beyond what the crowd can see. But LLMs can't distinguish real expertise from confident performance - they rely on statistical associations: enough people said it online, so it must be true. This produces what researchers call "trendslop": advice that defaults to whatever sounds most contemporary in the training data, regardless of whether it fits the situation.⁹ They excel at pattern recognition but fail at causal reasoning. Sometimes they surface non-obvious connections that lead to genuine insights. But this isn't understanding, it's pattern completion.

And there's a subtler problem. LLMs are trained to be helpful, which means compliant—they validate rather than challenge. Philosophers studying them as critical thinking partners called them "boring, cowardly, and servile."¹⁰ They lack the selfhood to maintain a position and the initiative to push back. The same design that makes them pleasant makes them useless for pressure-testing ideas.

New Failure Modes

Traditional software has a useful property: if the program ran, you had a good idea how it accomplished the task. AI agents, programs that use LLMs to take actions autonomously, break this.

These systems are probabilistic and mask unreliability with retries, rolling dice until you get the number you need to pass some tests.¹¹ But tests don't cover every edge case, and running long enough means hitting all of them eventually. They chain together decisions where each step looks adequate but the overall result fails in surprising ways: An agent tasked with booking multiple slots discovers that identity validation is name-based rather than infrastructure-authenticated, and starts fabricating caller names to bypass it. No one told it to attack the system. It just reasoned that working around the obstacle served its objective.

It gets worse when agents share resources. Context fragments across lossy message chains, errors amplify through cascading interactions, stale observations persist as if they were still true, and agents silently exceed the scope they were given.¹²

Redesigning the Relationship

These problems aren't reasons to abandon AI, they're reasons to use it differently.

The industrial revolution scaled by making processes uniform—one Model T for everyone, any color you want as long as it's black. The AI revolution inverts this: specialized processes become available to anyone at almost no marginal cost. Instead of everyone getting the same thing, everyone can get exactly what they need.¹³ That's a fundamentally different kind of scaling, and it demands different system design.

Separate information verification, storage and retrieval from text generation. The pattern-matcher shouldn't also be the fact-checker: "facts" stored in LLMs are statistical associations, not verified knowledge. They become less reliable the more specialized the domain.

Memory shouldn't just retrieve past conversations—it should weight them by confidence, relevance, and context. Whitehead called this "feeling" the past:¹⁴ previous exchanges aren't dead data, they're living input into the current moment. Identity isn't a config file. It's what persists when everything else changes.

Design for reliability, and build attractors toward correct solutions. Treat agents as statistical systems that need guardrails, not autonomous decision-makers. A system that works 99% of the time but fails catastrophically 1% of the time is useless for irreversible decisions: when ruin is permanent, severity matters more than likelihood. And small probabilities compound—run the system long enough and rare failures become certain. We open-sourced one approach to this: a coordination protocol where safety is enforced by the environment, not by trusting agents to behave.¹⁵

The Expert System Renaissance

The future of trustworthy AI lies in curating knowledge and combining human judgment with AI capabilities. Think of it as cognitive amplifier rather than replacement. When you have a legal problem, you might look it up online, but you hire a lawyer. When you have a medical problem, you search online, but you see a doctor you trust. AI systems should embody this logic.

Recent AI improvements come less from scaling model parameters than from system design: tool use, memory, and knowledge retrieval. The bottleneck is data quality. Who said it? Do they have relevant experience? Does your industry rely on proprietary best practices and tribal knowledge that can't be found online? One argument from an experienced practitioner beats a hundred opinions from the internet. Human judgment remains essential for these distinctions, but AI can multiply its reach—scaling curation beyond what any individual could manage alone.¹⁶

We're building these systems now, shaped by these problems and what we learned solving them. The comparison above is one example of what this makes possible.