High server loads

[Phil Harvey]

Already we have had a break-in attempt or bad bot hammering our site with a moderate load.  On the old server, things like this could have caused problems, but the new sever handled it like a champ.  I manually denied the offending IP after about 30 minutes of this activity.  It was good I was watching at the time so I could see how things went, and it allowed me to implement and test a script to automatically block IP's that do this in the future.

Attached is a plot of the web site access rate in hits/sec averaged over 10 minute periods, with the red line being the denied accesses.

- Phil

[Phil Harvey]

I've implemented some limits to help deal with future attacks and misbehaving bots.

With the current settings an automatic IP ban is applied to any non-member of the forum that exceeds 500 page loads or 50 MB of data in a 10 minute period.

If you think you have been erroneously banned, send me an email (philharvey66 at gmail.com), but of course you will have trouble reading this post if that happened to you.

- Phil

Edit: I've also added a limit of 200 MB per day.

[Phil Harvey]

It turns out that Chocolatey clients were getting banned frequently for downloading > 70MB in a short amount of time.  The Chocolatey package maintainer has changed the way ExifTool is supplied to help reduce this load.  Read here.

- Phil

[StarGeek]

Cloudflare would be helpful, though I don't know if the amount of traffic would be covered by a free account or not.  It looks like their Pro plan is $25/month or $240 for an annual billing.  Hopefully, @gep13 from that thread will be able to help.

I've played around with Cloudflare trying to use their Cloudflare Tunnel to be able to access my home computer without having to open ports, but it's just a bit above my ability.

[Phil Harvey]

Yes.  DreamHost has recommended CloudFlare as well, but resist going to something like that.  Perhaps I don't fully understand the benefits, but I don't think it should be necessary.

- Phil

[StarGeek]

I think of it as if it was a cache drive/firewall for your website.  It would probably cache the more static things like the archives and images, while providing some protection from IP addresses that are making too many requests.

[Phil Harvey]

Thanks.  That's about what I thought too.

Honestly, the traffic on exiftool.org is quite low.  It averages about 1 request per second.  Even when it is being "hammered", the most I have seen is 50 requests/sec, which I think is pretty mild.  I ran this web site from a Mac Mini and an old iMac for 10 years without any appreciable system load.  And now that we're on a new server at DreamHost this load is easily handled.  But I'm just playing with limiting the maximum traffic per IP as a bit of future protection.

- Phil