Bots were 40% of our traffic. We still let AI crawlers in.

In August we took a hard look at who actually visits introzy.com. At least 40% of the anonymous visitors to our marketing site were bots.

Our in-browser bot check had flagged none of them.

We left the door open anyway. Our robots.txt still opens with "All crawlers and AI agents welcome," and it points every crawler to a plain-language summary of what we do.

We think both decisions were right, and the reasoning applies to most small B2B companies with a website and a sales team.

How we missed almost half our traffic

There were two reasons, and one of them was our fault.

The code that spotted a likely bot was supposed to label the visit. Instead it dropped it. So our data warehouse never contained a flagged bot, because the flags never arrived. "Zero bots detected" was a bug.

The bigger reason is that modern bots don't look like bots. They run real browsers from residential internet connections, the same kind your customers use at home. They pass the classic checks: no automation flags, normal browser names, normal screen sizes. Imperva's 2025 Bad Bot Report found that automated traffic had passed human traffic for the first time, with bad bots alone making up 37% of all internet traffic.

What we changed

We stopped throwing data away and started labeling it.

• Every page view is recorded and labeled likely bot or likely human. We still keep likely bots out of signup and lead tracking.
• Declared crawlers are labeled by name. Many AI and search crawlers announce who they are. We take them at their word for labeling purposes.
• Inhuman speed gets flagged. A visitor that moves between pages in under a second, repeatedly, gets marked as a suspected bot.
• Our own visits are tagged, so internal testing doesn't count as traffic.

It isn't perfect. Our speed rule labeled at least one real person as a bot. That's the case for labeling over deleting: a label can be audited and corrected. A filter that silently deletes can't be, which is how ours hid a bug.

Why we didn't block them

Cloudflare sees a large share of the web, and its September report on the "second audience" is blunt. More than half the traffic it handles is now automated. Requests from AI agents grew more than 1,700% in a year. Training went from 22% of AI crawler requests in spring 2025 to 52% by June 2026. Human visits are down as much as 40% in some heavily crawled categories.

If you sell ads, that's a serious problem. Every page a bot reads is an ad a person didn't see. Publishers have good reasons to block, and Cloudflare reports that 17% of its sites now block training crawlers.

We don't sell ads. We sell software to businesses, and businesses increasingly start their research by asking an assistant:

• G2 surveyed 1,076 B2B software buyers in March 2026. 51% now start research with an AI chatbot more often than with Google, up from 29% a year earlier. 69% chose a different vendor than they'd planned because of a chatbot's recommendation, and one in three bought from a vendor they'd never heard of.
• Gartner's survey of 645 B2B buyers found 45% used generative AI, mostly to gather information on vendors and products. 69% prefer to check what the AI told them with a sales rep.
• Forrester reports that 94% of business buyers use AI somewhere in the buying process.

Those are surveys, and categories differ, but all three point the same way. A growing share of shortlists get written by an assistant before anyone visits your site.

Being in the answer works like a referral

We build referral tracking software, so we notice the parallel. When an assistant names you in a shortlist, it's doing what a trusted contact does when they say, "You should talk to these people." The buyer arrives already leaning your way. Gartner's 69% are the buyers who then check that recommendation with a person, the same way they'd check a referral.

The analogy isn't perfect. An assistant's mention brings no click to track, no introducer to thank, and no referral fee to pay. You can't attribute it the way you'd attribute a real introduction. But you can make sure you're in the room where the shortlist gets written, and blocking the crawlers that build those answers takes you out of it.

The fair objections

"They're training on our content for free." Some are. Training, search, and agents acting for a user are different things, and you can decide on each separately. Cloudflare now lets every site set Search, Agent, and Training policies on their own, including on the Free plan. Today we allow all three, and we'll revisit that.

"Bots cost money." For a marketing site our size, serving crawlers costs very little. For a large content site it can cost a lot. Measure before you decide.

"Bots ruin our analytics." This is the real cost, and better measurement fixes it. Blocking doesn't.

"robots.txt doesn't stop anyone." True. It's a request. Well-behaved crawlers honor it. Badly behaved ones ignore it, which is why the bot problem and the crawler policy are separate problems.

What we'd tell another small B2B marketer

1. Separate counting from access. Count first. Decide who gets in second. They're different questions with different owners.
2. Keep bots out of your conversion math. Report the human funnel on its own. If 40% of your "visitors" never existed, your traffic looks better than it is and your conversion rate looks worse.
3. Label, don't drop. You can't audit what you've thrown away.
4. Decide on search, agents, and training separately. Cloudflare's free bot settings cover the basics.
5. Don't over-invest in llms.txt. We publish one because it's cheap and honest. But Google says there are no additional technical requirements to appear in its AI features, beyond normal search eligibility. Common Crawl found 584,107 llms.txt files in its July crawl. Two-thirds were generated by a plugin, and a few contained prompt injections. Write good pages first.
6. Give agents a real front door. Crawlers read your site. Your customers' assistants should be able to do more than read. Ours can connect to Introzy over MCP and work with real referral data, with the same permissions a person would have.

What's next for us

The next step is moving classification out of the browser, where a bot can simply decline to run our script, and closer to the server, where it can't. We'll write up what changes.

If you're at Cloudflare Connect next week, the Wednesday session "Beyond 'Are you human?'" is about exactly this problem. I'll be there.

FAQ

What percentage of website traffic is bots?

It depends on the site, but often more than you'd expect. On our B2B marketing site, at least 40% of anonymous visitors were bots in August 2026. Cloudflare reports that more than half of the traffic it handles is now automated, and Imperva's 2025 report found automated traffic had passed human traffic.

Should a B2B company block AI crawlers?

Usually not all of them. B2B buyers increasingly ask AI assistants for vendor shortlists, and blocking the crawlers that build those answers can keep you off the list. Decide on search, agent, and training access separately, and measure bot traffic so it doesn't distort your reporting.

Why didn't our bot detection catch anything?

Two common reasons: the detection code drops flagged visits instead of labeling them, so nothing shows up in reports, or the bots run real browsers on residential connections and pass client-side checks. Label visits rather than dropping them, and add signals a bot can't easily fake, such as page-to-page speed.

Does llms.txt help with AI search?

There's little evidence it does much on its own. Google says AI Overviews and AI Mode have no additional technical requirements beyond normal search eligibility. A short, accurate llms.txt doesn't hurt, but clear, useful pages matter more.

Stop reading about referrals. Start tracking them.

Free up to 5 referrals a month. Every feature included. No card to start.