Website stats checking

For the last few weeks I have been trying to buy Pikmin 3 and Super Luigi U on eBay. I haven't had much luck so far as the prices always go up too high. One thing I noticed though, is that there seems to be a variety of age ratings printed on the covers.

In the screenshot from eBay above for Pikmin 3 listings, you can see the bottom copy is rated E (Everyone), above that the copy is rated 3, the copy above that is rated 7, and the top copy is rated E, but has a different cover design to the other ones. Seems a bit strange that they would produce so many different covers. (And actually there was another copy further down the page rated RP).

I was going through my website stats again today, and one of my sites had a lot of 406 Document not acceptable to client errors. I checked the logs and they appeared to be mostly to /wp-login.php and /wp-comments-post.php. I checked that I could comment okay, and I could. So I suspect these error codes are actually coming from bots. There shouldn't be any hits on /wp-login.php coming from humans (other than me, and I've never had a 406 error when logging in).

Another error I found was HEAD requests from PaperLiBot that were generating 500 internal server errors. PaperLi is a content curation service, not a spam / hacking bot. I tried the same request myself, and got 1 200 response (with the headers as expected).

After looking into this problem for quite a while, I found that actually the problem must have been the server going down for a bit. There were lots of these errors, but then further down in the log HEAD requests from PaperLiBot were being responded to with a 200 response code.

Looking at further errors, I found another block of 500 errors, this time for normal GET requests, but again for a bot. It looks like when lots of requests are received around the same time, then the server starts responding with 500 errors. As the site in question is on a shared HostGator account, I am not sure if there is anything I can do about it.

I think that probably HG forks a new apache process for each request, but after a certain number / memory usage, they stop spawning any new processes. You'd hope that any extra requests would be queued, but it could be that they just get a 500 error instead. I think I will have to check with HostGator to see what the issue is, and if there is anything I can do about it.