Since yesterday my server has again been getting absolutely obliterated by AI scrapers. This time, though, load is below 1, but I’m getting up to 10 requests a second and all of my Apache workers are in state “R”. “apachectl restart” fixes it… for a while. And fail2ban is banning IPs full-tilt.
What levers do I have to pull on this? E.g. maybe it would be sensible to drop connections if they stay in “R” for more than a couple seconds?
This 12 year old post suggests some sysctl.conf changes, but I have no idea whether those suggestions are sensible today. My current settings for those are the defaults:
net.ipv4.tcp_fin_timeout = 60 net.ipv4.ip_local_port_range = 32768 60999 net.core.somaxconn = 4096 net.core.netdev_max_backlog = 1000
In httpd.conf, some vhosts have “Timeout 240” because I really do have some CGIs that take that long to run, and you can’t make exceptions on a per-URL basis.
I have reqtimeout_module loaded, with the default settings, which I believe are:
handshake=0 header=20-40,MinRate=500 body=20,MinRate=500
From jwz via this RSS feed