ChatGPT, Bytespider, PetalBot… more and more bots are visiting your WordPress not to find you on Google, but to copy your content and use it to train AI models. And many of them don’t even respect the rules you set for them.
Why this is a new and growing problem
AI training bots that crawl your content without permission
Unlike Googlebot (which crawls to index you and send you traffic), these bots collect your content to train language models — with no attribution, no link back, no benefit to your site whatsoever.
Rising AI bot traffic in 2025-2026
The volume of this kind of traffic has grown a lot in a short time, and on content-heavy sites it can end up being a notable share of total traffic — consuming bandwidth and server resources with zero return.
Why robots.txt isn’t enough
Bytespider and other bots that simply ignore it
The robots.txt file is just a polite request: it tells a bot “please don’t crawl this,” but nothing forces it to comply. Several well-known AI bots, including Bytespider (from ByteDance), have a reputation for frequently ignoring it.
How to actually block these bots, at the server level
List of known AI bot user-agents
Some of the most common: GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), Bytespider (ByteDance), PetalBot (Huawei), CCBot (Common Crawl, used by many models), Google-Extended (Google’s training crawler, distinct from Googlebot).
Real firewall-level blocking, not just a suggestion the bot can ignore
The only way to truly block a bot that doesn’t respect robots.txt is at the server level: identifying its user-agent (or its behavior, if it spoofs the user-agent) and returning a real block before it consumes any resources on your WordPress. SeenSecure’s UA Blocklist layer keeps that list of known AI bot user-agents up to date, so you don’t have to add them by hand every time a new one shows up.
Which AI bots you actually want to let through
Difference between training bots and bots that cite your content with attribution
Not all AI-related bots are the same. Some (like the ones powering answers with a direct link to your page) do send you real traffic. Before blocking everything that “sounds like AI” in bulk, it’s worth telling apart which ones you actually want to keep.
Frequently asked questions
Is robots.txt enough to block AI bots like Bytespider?
Not always — robots.txt is a voluntary request, and some AI bots ignore it outright. For a real block, you need to do it at the firewall/server level.
Does blocking AI bots affect my Google ranking?
Not if you tell the user-agents apart correctly: Googlebot is a search crawler, not an AI training bot, and they’re independent blocks.
What happens if I don’t block AI training bots?
Your content can be used to train AI models without your consent or attribution, on top of consuming your server’s resources with their crawling.