Uploaded image for project: 'phpBB3'
  1. phpBB3
  2. PHPBB3-16862

Remove obsolete bots

    XMLWordPrintable

Details

    • Improvement
    • Status: Open (View Workflow)
    • Trivial
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      After adding some new bots, I'm going through the ones which never crawled our forum and I think some of them are not active anymore at all since a long time and could be removed from the hardcoded phpBB bots list.

      The "Alta Vista [Bot]" btw shouldn't be active anymore, as Alta Vista was bought by Yahoo in 2003 and uses Yahoo search results in the meantime. When accessing the search engine website, one is redirected to Yahoo as well: https://en.wikipedia.org/wiki/AltaVista

      "FAST Enterprise Crawler" seems to be from SharePoint prior to 2010 and was deprecated in 2010, replaced by the "SharePoint 2010 Web Crawler" that time: https://social.technet.microsoft.com/wiki/contents/articles/18576.best-practices-using-the-fast-enterprise-crawler.aspx

      Is there a strategy about how to detect and remove obsolete bots by times? Data from multiple larger phpBB forums would be great, to be sure a certain user agent is not active anymore, else it is a nasty research. At least I couldn't find a good database with last active times of "all" web crawlers/bots.

      Attachments

        Activity

          People

            Unassigned Unassigned
            MichaIng MichaIng [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: