-
Improvement
-
Resolution: Unresolved
-
Trivial
-
None
-
None
-
None
After adding some new bots, I'm going through the ones which never crawled our forum and I think some of them are not active anymore at all since a long time and could be removed from the hardcoded phpBB bots list.
The "Alta Vista [Bot]" btw shouldn't be active anymore, as Alta Vista was bought by Yahoo in 2003 and uses Yahoo search results in the meantime. When accessing the search engine website, one is redirected to Yahoo as well: https://en.wikipedia.org/wiki/AltaVista
"FAST Enterprise Crawler" seems to be from SharePoint prior to 2010 and was deprecated in 2010, replaced by the "SharePoint 2010 Web Crawler" that time: https://social.technet.microsoft.com/wiki/contents/articles/18576.best-practices-using-the-fast-enterprise-crawler.aspx
Is there a strategy about how to detect and remove obsolete bots by times? Data from multiple larger phpBB forums would be great, to be sure a certain user agent is not active anymore, else it is a nasty research. At least I couldn't find a good database with last active times of "all" web crawlers/bots.