CDN and Firewall Blocking Bot Access

Cloudflare Bot Fight Mode

Issue: Cloudflare’s Bot Fight Mode blocks or challenges AI bots before they reach your site.

Diagnosis: Check Cloudflare dashboard → Security → Bots. Look for blocked requests from known bot IPs.

Option A: Allow Verified Bots (Recommended)

1. Cloudflare Dashboard → Security → Bots
2. Enable “Allow Verified Bots”
3. This permits Googlebot, Bingbot, and other verified crawlers

Option B: Custom Bot Management Rules

1. Security → WAF → Custom Rules
2. Create rule to allow specific bots (GPTBot, ClaudeBot, PerplexityBot, etc.)
3. Action: Skip (WAF, Bot Fight Mode)

Sucuri Firewall

Issue: Sucuri blocks bots at firewall level, preventing server access.

Solution:

1. Sucuri Dashboard → Firewall → Whitelist
2. Add IP ranges for legitimate bots
3. Or whitelist specific user-agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended

Wordfence Firewall

Issue: Wordfence Rate Limiting or Advanced Blocking stops bots.

Solution:

1. Wordfence → Firewall → Rate Limiting
2. Under “How should we treat Google’s crawlers?”: Change to “Verified bots are not subject to rate limiting”
3. Check Advanced Blocking for bot-related rules

ModSecurity WAF

Issue: ModSecurity WAF rules block bot requests as suspicious.

Solution – Whitelist User Agents:

SecRule REQUEST_HEADERS:User-Agent "@contains GPTBot" \
    "id:1000001,phase:1,nolog,pass,ctl:ruleEngine=Off"

SecRule REQUEST_HEADERS:User-Agent "@contains ClaudeBot" \
    "id:1000002,phase:1,nolog,pass,ctl:ruleEngine=Off"

SecRule REQUEST_HEADERS:User-Agent "@contains PerplexityBot" \
    "id:1000003,phase:1,nolog,pass,ctl:ruleEngine=Off"

Testing CDN/Firewall Changes

Manual Test with cURL:

# Test GPTBot access
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  https://yoursite.com/

# Test ClaudeBot access
curl -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com)" \
  https://yoursite.com/

Expected Result: HTTP 200 status code, no firewall block pages, no CAPTCHA challenges

Best Practices for Bot Access

DO:

• Allow verified bots (Googlebot, Bingbot)
• Whitelist major AI bots
• Use official IP ranges when possible
• Monitor firewall logs regularly

DON’T:

• Block all bots indiscriminately
• Use “Under Attack Mode” permanently
• Forget to exclude tracking scripts from CDN cache
• Set rate limits too low for crawlers