Getting more traffic to your website in the age of A.I search engine \u00a0is a bit tricky. You have to get yourself familiar with the technicality of website ranking in order to increase your website’s visibility online. In December 2019 Google updated Googlebot user agent strings to reflect the new browser version, and periodically updated the version numbers to match Chrome updates in Googlebot. Below depicts user agents in robots.txt where several user-agents are recognized in the robots.txt.<\/p>\n
If you want to block or allow Google’s crawlers from accessing some of your content, you can do this by specifying Googlebot as the user-agent. For example, if you want all your pages to appear in Google search, and if you want AdSense ads<\/a> to appear on your pages, you don’t need a robots.txt file. Similarly, if you want to block some pages from Google altogether, blocking the user-agent Googlebot will also block all Google’s other user-agents.<\/p>\n But if you want more fine-grained control, you can get more specific. For example, you might want all your pages to appear in Google Search, but you don’t want images in your personal directory to be crawled. In this case, use robots.txt to disallow the user-agent Googlebot-image from crawling the files in your \/personal directory (while allowing Googlebot to crawl all files), like this:<\/p>\n To take another example, say that you want ads on all your pages, but you don’t want those pages to appear in Google Search. Here, you’d block Googlebot, but allow Mediapartners-Google, like this:<\/p>\nUser-agent: Googlebot\r\nDisallow:\r\n\r\nUser-agent: Googlebot-Image\r\nDisallow: \/personal\r\n<\/pre>\n
User-agent: Googlebot\r\nDisallow: \/\r\n\r\nUser-agent: Mediapartners-Google\r\nDisallow:\r\n<\/pre>\n