Not every robot is malicious. Search engines, uptime monitors and authorised integrations visit websites automatically for a legitimate purpose. Other bots test passwords, copy prices, abuse forms, harvest addresses or request thousands of useless URLs. The objective is therefore not to “block bots”, but to recognise traffic and respond proportionately.
Helpful, questionable and harmful automation
A legitimate crawler identifies itself, follows reasonable patterns and can be technically verified. A scraper may collect public content without directly attacking the site, yet still consume resources or copy intellectual property. Malicious bots target logins, vulnerable software, forms and payment flows. A user-agent alone proves little because it is easily forged.
What does a site owner notice?
- Large numbers of requests to missing pages, wp-login.php or xmlrpc.php.
- Unexpected CPU, PHP worker, database or bandwidth peaks.
- Form spam and fake registrations.
- Price, inventory or content scraping.
- Distorted analytics and slower responses for real customers.
robots.txt is not a security wall
robots.txt asks cooperative crawlers not to visit specified locations. A hostile bot can ignore it. Use it for crawl guidance, never for secrets or access control. Sensitive data requires authentication and server-side protection.
Combine multiple layers
Start with logs and behaviour patterns. Rate limiting slows clients making excessive requests. A web application firewall blocks known attacks. Login controls, MFA and CAPTCHA reduce automated abuse at high-risk endpoints. Reputation, behaviour, request frequency and technical verification together are more reliable than one IP block.
Cacheable pages can be delivered efficiently through LiteSpeed caching, while dynamic login, search, basket and API endpoints need tailored protection. Account isolation also prevents one attacked site from exhausting every other account.
What about AI crawlers?
Organisations can decide which AI crawlers they want to allow. Balance discoverability, attribution, licensing and server load. Remember that robots.txt is honoured only by cooperative operators. Measure actual traffic before applying broad restrictions.
Practical action plan
- Measure the URLs, agents and networks creating the load.
- Verify legitimate search crawlers through their official methods.
- Protect login, forms, XML-RPC and APIs specifically.
- Apply rate limits appropriate to each endpoint.
- Block demonstrably harmful behaviour and review false positives.
- Monitor performance, conversions and crawlability after changes.
Frequently asked questions
Should every unknown bot be blocked?
No. That may disrupt monitoring, search engines, payments or integrations. Restrict suspicious behaviour first.
Does CAPTCHA stop every bot?
No. It is useful on forms and authentication flows, but remains one layer.
Can bots damage SEO?
Indirectly, through outages, slow pages, copied content or uncontrolled URL discovery. Stable hosting and sensible crawl controls reduce that risk.
Suspect unwanted bot traffic? Open a ticket through the support portal and include the time and affected URLs.
