Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers
Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.| Google for Developers
