Blocking Bot Traffic with nginx + fail2ban, No Cloudflare Needed

Why I'm Not Using Cloudflare

The chatbot for aicoreutility.com sends streaming responses via Server-Sent Events (SSE). On Cloudflare's Free plan, this connection frequently dropped.

• The proxy would close long-lived connections midway.
• Even with buffering options disabled, chunks larger than 100KB would get stuck.
• Some response headers were rewritten, breaking the `Last-Event-ID` functionality.

Upgrading to a paid plan would solve these issues, but the cost of ARGO is prohibitive for a solo-operated service. Ultimately, I reverted to a structure with Cloudflare removed, relying on a single GCP instance with my own Nginx handling SSL termination.

This, however, introduced a new problem: scanner bot traffic was coming through unfiltered.

Actual Traffic Observed

Pulling a day's worth of data from the Nginx access logs revealed:

GET /.env HTTP/1.1                       (157 times)
GET /wp-admin/admin-ajax.php             (89 times)
GET /.git/config                         (76 times)
GET /actuator/health                     (54 times)
GET /phpmyadmin/                         (43 times)
POST /api/auth/login (brute force)       (211 times)

This is all automated scanning. While I could ignore it, it was consuming CPU and filling up the log disk.

Defense 1: Immediate Nginx 444 Blocking

For meaningless paths, I decided not to even send a response. `444` is an Nginx-specific code that closes the connection entirely.

location ~* (/.env|/wp-admin|/wp-login|/.git|/phpmyadmin|/actuator) {
    return 444;
}

This also has the effect of making bots waste resources waiting for a timeout.

Defense 2: 5 Types of `limit_req` Zones

I implemented different rate limits per path.

# /etc/nginx/conf.d/zz-security.conf
limit_req_zone $binary_remote_addr zone=rl_general:10m rate=30r/s;
limit_req_zone $binary_remote_addr zone=rl_api:10m     rate=10r/s;
limit_req_zone $binary_remote_addr zone=rl_auth:10m    rate=5r/m;
limit_req_zone $binary_remote_addr zone=rl_track:10m   rate=20r/s;
limit_req_zone $binary_remote_addr zone=rl_chat:10m    rate=2r/s;
limit_conn_zone $binary_remote_addr zone=conn_per_ip:10m;

The key here is 5 requests per minute for authentication. A normal user won't attempt to log in 5 times a second. This effectively targets bots.

Defense 3: Fail2ban with 4 Jails

Fail2ban learns from IPs blocked by Nginx and blocks them at the iptables level. Subsequent requests never even reach Nginx.

# /etc/fail2ban/jail.d/nginx-aicoreutility.conf
[nginx-rate-limited]
enabled = true
filter  = nginx-rate-limited
logpath = /var/log/nginx/error.log
maxretry = 10
findtime = 60
bantime = 3600

[nginx-scanner]
enabled = true
filter  = nginx-scanner
logpath = /var/log/nginx/access.log
maxretry = 3
findtime = 600
bantime = 86400

[nginx-bad-request]
enabled = true
filter  = nginx-bad-request
logpath = /var/log/nginx/access.log
maxretry = 20
findtime = 60
bantime = 3600

[sshd]
enabled = true
maxretry = 3
findtime = 600
bantime = 86400

I set a 1-day ban for scanners with just one attempt and a 1-hour ban for brute force attempts, differentiating the severity.

Defense 4: Kernel Tuning

SYN floods and spoofed IP responses are handled by `sysctl`.

# /etc/sysctl.d/99-network-security.conf
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.somaxconn = 4096
net.ipv4.tcp_synack_retries = 2

Simply enabling `tcp_syncookies` effectively neutralizes SYN flood attacks.

Defense 5: Slowloris Timeout

Nginx's default timeouts are quite generous. I've tightened them.

client_body_timeout 10;
client_header_timeout 10;
keepalive_timeout 5 5;
send_timeout 10;

First Week of Operations Results

Points to Note

• `limit_req` too strict can block you. After deployment, static resource requests can spike due to cache refreshing. Use the nodelay option and sufficient burst values.
• Fail2ban self-blocking: If you don't add your admin IP to ignoreip, you might lock yourself out via SSH.
• Exclude SSE endpoints from `limit_conn`. A single user opening multiple tabs will create multiple concurrent SSE connections.

📌 Comment from 2026

It's entirely possible to build your own defense layer on a single instance even without Cloudflare. However, this requires monitoring. You need a routine of checking Fail2ban's block logs daily and verifying that legitimate users aren't being blocked incorrectly.

My next steps include implementing GeoIP-based blocking (for traffic from suspicious ASNs) and adding cost limit alarms.