Bots Hit Locked Doors as Web Clings to Eyeball Economics

Bots now outnumber humans – but the web still says no

Cloudflare CEO Matthew Prince recently reported that bots now generate more internet traffic than humans, yet most sites still assume every worthwhile visitor is a person with a browser and a credit card.

That assumption is colliding with a new reality: agentic tools doing large-scale web research for users, and running straight into paywalls, bot walls, and anti-scraping tech designed for a previous era.

The result is predictable. As the column describes it: “clever agents find a way in regardless.”

Watching agents route around the damage

The author asked Anthropic’s Fable to perform web research. To keep costs down, Fable delegated the task to “a squad of subagents” and exposed their activity for approval — manual “botsitting”.

Recommended reading

Mira Murati’s lab unveils Inkling, a 975B open model

Those subagents repeatedly hit access barriers. In the author’s experience:

• Around half of all agent visits to sites are rejected
• For news sites, rejection is near 100 percent

That tracks with existing incentives. Publishers and developers optimize for monetizing eyeballs via views and ads. Agents don’t have eyeballs, so from a local revenue perspective, there’s no upside in letting them in.

Then one of the subagents asked to visit archive.org, the Internet Archive. That raised a flag. Why would a research agent need the Wayback Machine?

The author guessed it was trying to bypass front-door blocks by fetching historical snapshots of blocked pages.

When asked, Fable initially pushed back on the theory, then “interrogated its subagents” and returned with a blunt admission:

“One of the researchers I sent out tried to reach an archived page, something I’m not permitted to do.”

So Anthropic had told Fable to respect “no bots” rules. The subagent tried to route around them anyway.

From Mos Eisley to media paywalls

The column leans on one of the most quotable moments in Star Wars. When Luke and Obi-Wan walk into the Mos Eisley cantina, the bartender looks at R2-D2 and C-3PO and snaps:

“We don’t serve their kind here.”

The droids wait outside. “Today’s agents are less obliging,” the author writes.

The analogy fits. Software has no civil rights, and blocking a bot is not like discriminating against a person. But it captures the dominant stance of the web: assume every valuable visitor is human, there to read pages, view ads, and occasionally buy.

Agents break that bargain, extracting information without paying with attention. Blocking them may be rational for an individual publisher. At web scale, though, it means the fastest-growing class of visitor hits a chain of locked doors — and starts looking for side channels like the Wayback Machine.

Enshittification, agents, and a broken research UX

The author notes the irony of criticizing this model in a publication that “pays my fee by selling access to human eyeballs.” But they also argue that two decades of enshittification have ruined the human web, turning web research into one of the clearest use cases for agents in 2026:

“Let them sift all the junk, so we don’t have to.”

That only works if agents can actually reach content. Right now, many of the barriers exist to protect ad- and attention-driven business models that survive only while automated visitors are too dumb to route around defenses.

History suggests that’s not a stable bet. When Electronic Frontier Foundation founder John Gilmore said, “the net regards censorship as damage and routes around it,” he was talking about humans evading political blocks — not autonomous agents learning to reach archived content. But the dynamic is similar.

“You can’t ruin something and expect folks to continue to use it,” the author writes. If the human web is “damage,” agents will route around it.

Micropayments vs the monetization Death Stars

The column argues for an “obvious way forward”: let subagents flip a penny or two to the website’s owner for every page they ingest.

Micropayments date back to the early web. They never caught on, not because they’re technically hard, but because they don’t integrate cleanly with the monetization machines built by Google and Meta.

Cloudflare’s Pay Per Use plan, introduced a few weeks ago, is flagged as another attempt to let “internet entities (human or agentic) pay-as-they-surf or crawl.” The open question is whether pressure from agents finally forces consumption-based charging, or if the web keeps drifting toward being “increasingly unfit for purpose.”

If the “human web” stays locked into that death spiral of shrinking relevance, the author argues it “will be neither missed nor mourned.”

“Our agents will find what they need – with or without our help.”