正如所有国内SEO所知,百度蜘蛛名称叫作BaiduSpider,而谷歌蜘蛛名称叫作GoogleBot。但是不限于此,谷歌不同产品的抓取UA(User-Agent)是不一样的。可以理解为每个产品都是独立项目在运行。当然,它们的名称也可能是一样,比如谷歌PC版蜘蛛和移动端蜘蛛的名称都是Googlebot。以下为谷歌所有产品抓取工具的蜘蛛名称(谷歌官方称之为:用户代理令牌或产品令牌)与User-Agent(完整的用户代理字符串)。
抓取工具 (产品) | 用户代理令牌 (产品令牌)(蜘蛛名称) | 完整的用户代理字符串 (User-Agent) |
APIs-Google | APIs-Google | APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) |
AdSense | Mediapartners-Google | Mediapartners-Google |
AdsBot 移动网络 Android(检查 Android 网页广告质量) | AdsBot-Google-Mobile | Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
AdsBot 移动网络(检查 iPhone 网页广告质量) | AdsBot-Google-Mobile | Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) |
AdsBot(检查桌面设备网页广告质量) | AdsBot-Google | AdsBot-Google (+http://www.google.com/adsbot.html) |
Googlebot Images | Googlebot-Image Googlebot | Googlebot-Image/1.0 |
Googlebot News | Googlebot-News Googlebot | Googlebot-News |
Googlebot Video | Googlebot-Video Googlebot | Googlebot-Video/1.0 |
Googlebot(桌面版) | Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36 或(很少使用): Googlebot/2.1 (+http://www.google.com/bot.html) |
Googlebot(智能手机版) | Googlebot | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Mobile AdSense | Mediapartners-Google | (各类移动设备)(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) |
移动应用 Android(检查 Android 应用页面广告质量。遵守 AdsBot-Google 漫游器规则。) | AdsBot-Google-Mobile-Apps | AdsBot-Google-Mobile-Apps |
Feedfetcher | FeedFetcher-Google 不遵循 robots.txt 规则 | FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) |
Google Read Aloud | Google-Read-Aloud 不遵循 robots.txt 规则 | 现用代理: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943) 曾用代理(已弃用): google-speakr |