Server Stability Issues

[Pauven]

Hi everyone, just a quick note to let you know that the ChamConSoft.com server has been experiencing a lot of downtime lately. I am aware, and I'm troubleshooting trying to determine the root issue. I'm sorry for the inconvenience.

If you notice the server down, feel free to email me to make sure I know. I try to check it daily, but I've found that it will often go down without warning, so I might not catch it immediately if it goes down shortly after I check.

[Pauven]

I was able to spend a few hours today troubleshooting the stability issue, which seems to be related to a lack of server optimizations, combined with increased web-crawler (indexing) bots scanning the site. Too much load, combined with bad settings, and the server would just hang itself.

Hopefully the server is now optimized. It definitely feels a bit faster to me, which is a good sign. Now if it will just stay up for more than 2 days, that'll be awesome.

[Pauven]

Server speed issues are still a problem. The executive summary is that web crawlers hammer the site, pulling data, driving up server utilization. The web hosting company provides spare "burst" CPU capability which really makes browsing the site feel lightning fast, but after a while of getting hammered by crawlers this burst capacity is used up, and the server defaults to very slow speeds.

Right now, the current culprit is Facebook. They must be getting me back for never setting up a Facebook account. What a horrid company. Since I got the server back up and running an hour ago, they've already downloaded 1.6GB+++ of data from the server. I didn't even know I had that much data to download!!! Insanity.

I'll be making more changes to limit how much data these bots can access. Please bear with me, I expect it will take a while to regain control of the server.

[Pauven]

Okay, in addition to other countermeasures, I've completely blocked facebook from crawling the server!

The server was offline from yesterday until about 1:30p today, and in this chart you can see that instantly network traffic ramped up - this was 99% facebook:

image.png (117.26 KiB) Viewed 1243 times

At the end there, you can see traffic dropped back to nearly 0 once the facebook ban took effect.

Also, here's the CPU utilization during that time. You can see that it barely climbed into the burst capacity region (and in the past, this trend continues until burst is depleted, and the server becomes super slow). Crisis averted?

image.png (130.17 KiB) Viewed 1243 times

I'm really thinking this new traffic is AI training related. While I guess it would be nice if AI had all the intelligence shared here on this website, I honestly can't afford it. I pay for barebones web serving, to support CMC users. Not aggressive web crawlers or our future AI overlords.