Google has announced the launch of two new web crawlers, bringing the total number of Google crawlers to 17.
Overview of Google Crawlers
Google's crawlers, also known as "search spiders" or "bots," play a crucial role in its search system. They visit websites to collect information, discover, and index new and updated web pages. This ensures that users get accurate and up-to-date search results when using Google Search.
Among the current Google crawlers is "Googlebot," which works for both mobile and desktop devices. Its job is to crawl existing pages and discover new ones to crawl in the future.
Another crawler is the "Google Inspection Tool" (Google-InspectionTool), used by Google's testing tools like the Rich Results Test and URL Inspection in Google Search Console.
The two new crawlers added are part of the "GoogleOther" program. This general-purpose crawler can be used by various product teams to fetch publicly accessible content from websites. For example, it can be used for one-time crawling for development and internal research.
The new crawlers are:
1. GoogleOther-Image
An enhanced version of GoogleOther designed to fetch publicly accessible image URLs.
2. GoogleOther-Video
An enhanced version of GoogleOther designed to fetch publicly accessible video URLs.
Controlling These New Crawlers
Website owners can control these crawlers' access to their sites through the usual robot.txt commands.
