High server loads

Started by Phil Harvey, November 22, 2023, 09:21:31 AM

Previous topic - Next topic
Print
Go Down Pages1 2 All
User actions

Phil Harvey

November 22, 2023, 09:21:31 AM
I had set the forum to temporarily disable itself if the load average on the DreamHost server got above 96 (it is a 32-core system, so this is 3 processes per core, which is very heavily loaded).  I did this to prevent DreamHost support from (permanently) disabling the ExifTool web site in apparent attempts to lighten the load.

However, recently the load average has been above 96 too often, so I have raised the threshold to 128.  This should help with the the "high stress on the server" errors we have been getting, and hopefully won't result in DreamHost pulling the plug on us (again), especially since I'm going away on vacation in a couple of days (I'll be gone from Nov 24 to Dec 2).

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

greybeard

#1
November 22, 2023, 02:00:35 PM
Is this just the forum (and not the main web site)? There isn't usually very much actual participation on the forum - does this mean some sort of automated attack?

Have a good vacation.

Phil Harvey

#2
November 22, 2023, 10:02:57 PM
The load average is across all web sites hosted by the DreamHost server.  The ExifTool site is usually only a small part of that, although occasionally we do get hit hard by 'bots.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#3
November 23, 2023, 08:16:43 PM
Hmm.  The site was disabled temporarily once that I saw today, so the load average does get above 128 at times.  Right now it is 18, and typically it is (and should be) less than 32.  But someone on this shared server is running occasional CPU-intensive processes, possibly even unrelated to web hosting.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

StarGeek

#4
November 23, 2023, 09:18:26 PM
It's been temporarily down multiple times this week.  At least twice today.
* Did you read FAQ #3 and use the command listed there?
* Please use the Code button for exiftool code/output.
 
* Please include your OS, Exiftool version, and type of file you're processing (MP4, JPG, etc).

Phil Harvey

#5
November 23, 2023, 09:23:55 PM
:(  That sort of sucks, but I think we'll just have to keep an eye on this for now.  If it doesn't get better I'll talk to DreamHost support when I get back after my vacation (leaving tomorrow and back on Dec. 2).

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#6
January 24, 2024, 12:50:29 PM
exiftool.org was down for about 40 minutes earlier today. The site was disabled by DreamHost because it "has been identified as a cause of instability on your shared web server". Yeah, right. The scripts at DreamHost suck because the traffic was light before they pulled the plug, and after I re-enabled it the site was still unresponsive for another 30 minutes. Looks like they are blaming exiftool.org for other problems on their server. But it is up again now. Just lucky I was there when it happened or else it would still be down.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#7
January 24, 2024, 09:10:11 PM
I just heard back from DreamHost support:

"We sincerely apologize for the inconvenience caused by the recent network issue. At around 11:30 AM PST today, our services experienced disruptions due to a technical failure within our network infrastructure during a scheduled maintenance"

Nice.  So they disabled exiftool.org due to their own technical failure.  Great.  :(

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

FrankB

#8
January 27, 2024, 04:58:34 AM Last Edit: January 27, 2024, 08:27:55 AM by FrankB
Dont know if it is related, but..

I'm getting a 'Forbidden 403' on my fiber network.
I'm typing this on my cell phone, that has a different IP, that works,

Is my IP blocked?

Frank

Edit: Works again. Whatever it was, it was temporary

Phil Harvey

#9
January 27, 2024, 08:59:20 AM
Hi Frank,

Your IP was automatically blocked about 10 hours ago when dreamhost pulled the plug on the web site (again).  This is because your IP was hitting the server hardest at the time of the problem, but I've looked at the logs and you definitely shouldn't have been the cause of the problem.

I've unblocked it now.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

FrankB

#10
January 27, 2024, 09:19:49 AM
Thanks Phil.

Confirms my suspicion.

Frank

Phil Harvey

#11
January 27, 2024, 10:54:18 AM
DreamHost has disabled our web server 3 times in the last 2 days.  I've lowered the load limit back to 96 to temporarily shut down our site when their server gets loaded in hopes of stopping DreamHost from blaming us for their problems.  Their 32-core server has been frequently loaded with 100+ running processes, which is far too many, but exiftool.org isn't causing these high loads.  Regardless, DreamHost has been pushing for us to upgrade to a more expensive plan (which I don't think is necessary).

With the lowered load limit, you can expect more brief site outtages.  :(

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#12
January 28, 2024, 03:59:21 PM
I have complained to DreamHost support about the high server loads and they have offered to move our web site to a newer, faster server.  We'll see how that goes...

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#13
January 29, 2024, 07:52:19 AM
They did the move last night.  We are now running on a 128-core machine that is very lightly loaded, so things should be much better now.  We were previously on a heavily-loaded 32-core machine.

The migration seems to have gone seemlessly, so I'm very happy.  DreamHost support has really come through this time.

- Phil
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).

Phil Harvey

#14
January 29, 2024, 09:24:23 AM
Here is a plot of the server load average at 10-minute intervals for the last day or so.  You can see the sharp drop this morning at 03:40 UTC when we started running on the new server.

load_average.png
...where DIR is the name of a directory/folder containing the images.  On Mac/Linux/PowerShell, use single quotes (') instead of double quotes (") around arguments containing a dollar sign ($).
Print
Go Up Pages1 2 All
User actions