---
title: CDN and Firewall Blocking Bot Access
description: "##### Cloudflare Bot Fight Mode **Issue:** Cloudflare's Bot Fight Mode blocks or challenges AI bots before they reach your site. **Diagnosis:** Check Cloudflare dashboard → Security → Bots. Look for blocked requests from known bot IPs. **Option A: Allow Verified Bots (Recommended)**…"
url: "https://pulserank.ai/docs/cdn-and-firewall-blocking-bot-access/"
updated: "2026-01-09"
category: Troubleshooting
---

# CDN and Firewall Blocking Bot Access

##### Cloudflare Bot Fight Mode

**Issue:** Cloudflare's Bot Fight Mode blocks or challenges AI bots before they reach your site.

**Diagnosis:** Check Cloudflare dashboard → Security → Bots. Look for blocked requests from known bot IPs.

**Option A: Allow Verified Bots (Recommended)**

1. Cloudflare Dashboard → Security → Bots
2. Enable "Allow Verified Bots"
3. This permits Googlebot, Bingbot, and other verified crawlers

**Option B: Custom Bot Management Rules**

1. Security → WAF → Custom Rules
2. Create rule to allow specific bots (GPTBot, ClaudeBot, PerplexityBot, etc.)
3. Action: Skip (WAF, Bot Fight Mode)

##### Sucuri Firewall

**Issue:** Sucuri blocks bots at firewall level, preventing server access.

**Solution:**

1. Sucuri Dashboard → Firewall → Whitelist
2. Add IP ranges for legitimate bots
3. Or whitelist specific user-agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended

##### Wordfence Firewall

**Issue:** Wordfence Rate Limiting or Advanced Blocking stops bots.

**Solution:**

1. Wordfence → Firewall → Rate Limiting
2. Under "How should we treat Google's crawlers?": Change to "Verified bots are not subject to rate limiting"
3. Check Advanced Blocking for bot-related rules

##### ModSecurity WAF

**Issue:** ModSecurity WAF rules block bot requests as suspicious.

**Solution - Whitelist User Agents:**

```
SecRule REQUEST_HEADERS:User-Agent "@contains GPTBot" \
    "id:1000001,phase:1,nolog,pass,ctl:ruleEngine=Off"

SecRule REQUEST_HEADERS:User-Agent "@contains ClaudeBot" \
    "id:1000002,phase:1,nolog,pass,ctl:ruleEngine=Off"

SecRule REQUEST_HEADERS:User-Agent "@contains PerplexityBot" \
    "id:1000003,phase:1,nolog,pass,ctl:ruleEngine=Off"
```

##### Testing CDN/Firewall Changes

Manual Test with cURL:

```
# Test GPTBot access
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  https://yoursite.com/

# Test ClaudeBot access
curl -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com)" \
  https://yoursite.com/
```

**Expected Result:** HTTP 200 status code, no firewall block pages, no CAPTCHA challenges

##### Best Practices for Bot Access

**DO:**

- Allow verified bots (Googlebot, Bingbot)
- Whitelist major AI bots
- Use official IP ranges when possible
- Monitor firewall logs regularly

**DON'T:**

- Block all bots indiscriminately
- Use "Under Attack Mode" permanently
- Forget to exclude tracking scripts from CDN cache
- Set rate limits too low for crawlers