-
Improvement
-
Resolution: Fixed
-
Minor
-
None
I did some access log analysis on our server and found three bots with significant activity on our forums.
Most notably the Amazonbot which does around 30x (yes thirty!) times more requests than Googlebot or Bing bot, despite we're registered at Google an Bing search console/webmaster tools to send in sitemaps and assure they both keep their index current. Amazonbot also tries over and over to send postings, see user profiles, access the user control panel and the login page, without being able to do anything there without a registered account. It's around 100.000 requests a day, which is more than 10 times the actual (canonical) pages we have, caused by repeatedly accessing the same pages without different non-canonical query strings, session IDs etc. Let's say it is a very dump bot that unnecessarily does a massive amount of requests and does not even try to remember meaningful server replies, where it has access to and where not, a little like a mayfly. Limiting its abilities in the first place by having it within the bots group hence makes sense.
AhrefsBot does at least 2.5x more requests, so also a significant amount.
I also found the SemrushBot being quite active, in the area of Google and Bing.