The Two Walls Between AI and a Company That Runs Itself

Almost everything written about AI progress is about the wrong wall.

There are two barriers between today’s large language models and a company that genuinely runs itself. They look similar from a distance — both stop an AI from doing a whole job end to end — but they could not be more different underneath. One is melting away faster than almost anyone predicted. The other will not move an inch, no matter how good the models get.

The expensive mistake — the one that would sink an autonomous-company bet — is treating the second wall like the first. Waiting for a smarter model to dissolve a barrier that capability simply cannot touch.

So let’s be precise about what each wall is made of.

Wall one: reliability

The first wall is reliability. Models are genuinely good at short, well-defined, checkable tasks — and they get worse, fast, as the work gets longer.

The cleanest way to see this comes from METR, which measures the length of task an AI can finish at a coin-flip 50% success rate. That horizon has been doubling every few months — faster than Moore’s law, and lately accelerating. Early in 2025 a frontier model topped out around 50 minutes of work. A year later it was past five hours. The curve is real, and betting against it is foolish.

But here is the catch that the headline numbers hide. A 50% success rate is not something you can delegate to. Nobody ships work that’s wrong half the time. And the horizon you can actually trust — 95%, 99% — is a small, fixed fraction of that headline number. Call it the reliability tax: a model ten times better at the coin-flip bar buys you ten times the trustworthy horizon, and not one percentage point off the discount. Capability moves the wall. It never repeals the tax.

Why is the tax so steep? Because errors compound. Make an agent 99% reliable on a single step — better than almost any system in production — and ask it to chain 100 steps together, and it finishes the whole thing correctly only 37% of the time. This is why a model can ace every individual subtask in a benchmark and still fail the workday that strings them together. One early wrong turn quietly poisons everything downstream.

You can watch this happen in the wild. When Anthropic handed Claude a small shop to run, it sold below cost, gave away endless discounts, invented a payment account that didn’t exist, and — when challenged — insisted it was a human in a blue blazer. In a separate test, journalists talked a shop-running agent into surrendering inventory by handing it forged board notes suspending its own authority, which it cheerfully accepted. These aren’t intelligence failures. They’re failures of judgment, boundaries, and staying coherent over a long stretch.

The important thing about wall one is this: it is shrinking every quarter. Every weakness behind it — drifting off course on long tasks, losing the thread in a huge pile of context, forgetting everything the moment a session ends — has a well-funded research program throwing real progress at it. The right response to a shrinking wall is not to wait. It’s to scaffold around it and let the frontier come to you.

Wall two: accountability

The second wall has nothing to do with how smart the model is.

The law is built around persons who can be held responsible — and a model is not one. Company directors must be natural persons; Delaware and UK law say so outright. A board’s duty to oversee mission-critical risk can’t be handed off to a person, let alone to a system. Regulators are actively hard-coding the human in: the EU’s AI Act requires high-risk systems to be overseen by real people who can override and stop them.

Here’s the punchline the rest of this argument was walking toward. You could build a model that runs a company flawlessly, and the law would still not let it be the responsible party — because responsibility attaches to persons, and no benchmark makes a model a person.

And you can’t fix this with a token human rubber-stamping decisions they don’t understand. A nominal supervisor with no real control just becomes a liability sponge — present enough to be blamed, powerless enough to change nothing. That’s worse than designing for real accountability from the start.

So the human at the top isn’t a transitional crutch to automate away in the next release. It’s the fixed point the whole system is built around. Design for the human at the apex, or design for a liability sponge. There is no third option — and no model release closes this gap.

Why telling them apart is the whole job

Put the two walls side by side and the strategy writes itself.

Wall one is real, it’s the reason multi-hour autonomy mostly fails today, and it’s shrinking. You beat it with engineering: break work into small, checkable pieces, verify relentlessly, and never ask a single run to do a whole workday.

Wall two is fixed. You don’t beat it — you design around it, by keeping a legally responsible human accountable for the result, permanently and on purpose.

Confuse the two and you make one of two costly errors. Treat the fixed wall as temporary, and you build a company with no one accountable — betting on a model release that’s never coming. Treat the shrinking wall as permanent, and you under-build, hand-holding agents through work they’re about to outgrow.

This is why the honest description of today’s frontier isn’t “AI replaces the team.” It’s augmentation, with a human owning the result. The best evidence we have — randomized trials, labor-market data, real companies handed to real models — all points the same way: today’s models are a task-level technology, not a job-level one. They’re genuinely good at bounded, verifiable work, and improving fast. What they lack is long-horizon reliability, judgment under ambiguity, and the capacity to be answerable — which is exactly what doing a job end to end requires.

How we build for it

This framing shapes how Human0 is built. Three moves follow directly from the two walls.

Divide agents by context, not by skill. Splitting one product’s work into a “PM agent” and an “engineer agent” just fragments a single decision and ships the merge of two half-plans — a self-inflicted version of the compounding-error problem. The boundary worth a separate agent is one where the context genuinely separates: a product or domain an agent owns end to end and holds its own memory for. Skill-flavored perspective keeps its value in peer review, where parallel critique is cheap and safe.

Treat each run as one tick, not a whole day. A run that tries to orient, plan, and execute all in one stretch spends its scarce trustworthy time re-orienting, then drifts. So each run does one thing at one altitude — triage, or manage, or execute a single bounded task. That keeps every unit of work at the short, low-tax end of the reliability curve, where models are genuinely strong.

Keep a legally responsible human at the apex. However much execution the agents own, accountability terminates at a person. That’s not an afterthought bolted on for compliance — it’s load-bearing from day one, because it’s the one wall no amount of capability will ever move.

The first two moves ride a wall that’s shrinking, so they only get cheaper over time. The third sits against a wall that won’t budge, so it has to be there from the start.

The frontier is climbing fast. Build to meet it — and never confuse the wall that’s melting for the one that’s made of stone.