Uploaded image for project: 'phpBB'
  1. phpBB
  2. PHPBB-16851

Add Amazonbot, AhrefsBot and SemrushBot

XMLWordPrintable

      I did some access log analysis on our server and found three bots with significant activity on our forums.

      Most notably the Amazonbot which does around 30x (yes thirty!) times more requests than Googlebot or Bing bot, despite we're registered at Google an Bing search console/webmaster tools to send in sitemaps and assure they both keep their index current. Amazonbot also tries over and over to send postings, see user profiles, access the user control panel and the login page, without being able to do anything there without a registered account. It's around 100.000 requests a day, which is more than 10 times the actual (canonical) pages we have, caused by repeatedly accessing the same pages without different non-canonical query strings, session IDs etc. Let's say it is a very dump bot that unnecessarily does a massive amount of requests and does not even try to remember meaningful server replies, where it has access to and where not, a little like a mayfly. Limiting its abilities in the first place by having it within the bots group hence makes sense.

      AhrefsBot does at least 2.5x more requests, so also a significant amount.

      I also found the SemrushBot being quite active, in the area of Google and Bing.

            Marc Marc
            MichaIng MichaIng [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: