Robots.txt instructs search engines on which parts of the website can crawl and which parts should be left untouched.| techseoaudits.com