I had set the forum to temporarily disable itself if the load average on the DreamHost server got above 96 (it is a 32-core system, so this is 3 processes per core, which is very heavily loaded). I did this to prevent DreamHost support from (permanently) disabling the ExifTool web site in apparent attempts to lighten the load.
However, recently the load average has been above 96 too often, so I have raised the threshold to 128. This should help with the the "high stress on the server" errors we have been getting, and hopefully won't result in DreamHost pulling the plug on us (again), especially since I'm going away on vacation in a couple of days (I'll be gone from Nov 24 to Dec 2).
- Phil
Is this just the forum (and not the main web site)? There isn't usually very much actual participation on the forum - does this mean some sort of automated attack?
Have a good vacation.
The load average is across all web sites hosted by the DreamHost server. The ExifTool site is usually only a small part of that, although occasionally we do get hit hard by 'bots.
- Phil
Hmm. The site was disabled temporarily once that I saw today, so the load average does get above 128 at times. Right now it is 18, and typically it is (and should be) less than 32. But someone on this shared server is running occasional CPU-intensive processes, possibly even unrelated to web hosting.
- Phil
It's been temporarily down multiple times this week. At least twice today.
:( That sort of sucks, but I think we'll just have to keep an eye on this for now. If it doesn't get better I'll talk to DreamHost support when I get back after my vacation (leaving tomorrow and back on Dec. 2).
- Phil
exiftool.org was down for about 40 minutes earlier today. The site was disabled by DreamHost because it "has been identified as a cause of instability on your shared web server". Yeah, right. The scripts at DreamHost suck because the traffic was light before they pulled the plug, and after I re-enabled it the site was still unresponsive for another 30 minutes. Looks like they are blaming exiftool.org for other problems on their server. But it is up again now. Just lucky I was there when it happened or else it would still be down.
- Phil
I just heard back from DreamHost support:
"We sincerely apologize for the inconvenience caused by the recent network issue. At around 11:30 AM PST today, our services experienced disruptions due to a technical failure within our network infrastructure during a scheduled maintenance"
Nice. So they disabled exiftool.org due to their own technical failure. Great. :(
- Phil
Dont know if it is related, but..
I'm getting a 'Forbidden 403' on my fiber network.
I'm typing this on my cell phone, that has a different IP, that works,
Is my IP blocked?
Frank
Edit: Works again. Whatever it was, it was temporary
Hi Frank,
Your IP was automatically blocked about 10 hours ago when dreamhost pulled the plug on the web site (again). This is because your IP was hitting the server hardest at the time of the problem, but I've looked at the logs and you definitely shouldn't have been the cause of the problem.
I've unblocked it now.
- Phil
Thanks Phil.
Confirms my suspicion.
Frank
DreamHost has disabled our web server 3 times in the last 2 days. I've lowered the load limit back to 96 to temporarily shut down our site when their server gets loaded in hopes of stopping DreamHost from blaming us for their problems. Their 32-core server has been frequently loaded with 100+ running processes, which is far too many, but exiftool.org isn't causing these high loads. Regardless, DreamHost has been pushing for us to upgrade to a more expensive plan (which I don't think is necessary).
With the lowered load limit, you can expect more brief site outtages. :(
- Phil
I have complained to DreamHost support about the high server loads and they have offered to move our web site to a newer, faster server. We'll see how that goes...
- Phil
They did the move last night. We are now running on a 128-core machine that is very lightly loaded, so things should be much better now. We were previously on a heavily-loaded 32-core machine.
The migration seems to have gone seemlessly, so I'm very happy. DreamHost support has really come through this time.
- Phil
Here is a plot of the server load average at 10-minute intervals for the last day or so. You can see the sharp drop this morning at 03:40 UTC when we started running on the new server.
load_average.png
Already we have had a break-in attempt or bad bot hammering our site with a moderate load. On the old server, things like this could have caused problems, but the new sever handled it like a champ. I manually denied the offending IP after about 30 minutes of this activity. It was good I was watching at the time so I could see how things went, and it allowed me to implement and test a script to automatically block IP's that do this in the future.
Attached is a plot of the web site access rate in hits/sec averaged over 10 minute periods, with the red line being the denied accesses.
- Phil
Screen Shot 2024-01-30 at 1.37.48 PM.png
I've implemented some limits to help deal with future attacks and misbehaving bots.
With the current settings an automatic IP ban is applied to any non-member of the forum that exceeds 500 page loads or 50 MB of data in a 10 minute period.
If you think you have been erroneously banned, send me an email (philharvey66 at gmail.com), but of course you will have trouble reading this post if that happened to you. :P
- Phil
Edit: I've also added a limit of 200 MB per day.
It turns out that Chocolatey clients were getting banned frequently for downloading > 70MB in a short amount of time. The Chocolatey package maintainer has changed the way ExifTool is supplied to help reduce this load. Read here. (https://github.com/orgs/chocolatey/discussions/299)
- Phil
Cloudflare would be helpful, though I don't know if the amount of traffic would be covered by a free account or not. It looks like their Pro plan (https://www.cloudflare.com/plans/) is $25/month or $240 for an annual billing. Hopefully, @gep13 from that thread will be able to help.
I've played around with Cloudflare trying to use their Cloudflare Tunnel to be able to access my home computer without having to open ports, but it's just a bit above my ability.
Yes. DreamHost has recommended CloudFlare as well, but resist going to something like that. Perhaps I don't fully understand the benefits, but I don't think it should be necessary.
- Phil
I think of it as if it was a cache drive/firewall for your website. It would probably cache the more static things like the archives and images, while providing some protection from IP addresses that are making too many requests.
Thanks. That's about what I thought too.
Honestly, the traffic on exiftool.org is quite low. It averages about 1 request per second. Even when it is being "hammered", the most I have seen is 50 requests/sec, which I think is pretty mild. I ran this web site from a Mac Mini and an old iMac for 10 years without any appreciable system load. And now that we're on a new server at DreamHost this load is easily handled. But I'm just playing with limiting the maximum traffic per IP as a bit of future protection.
- Phil