Googlebot is the generic name of the web crawler used by Google Search. Discover what Googlebot is, how it accesses your site, and how to block Googlebot.| Google for Developers
Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.| Google for Developers
