Login
From:
adamj.eu
(Uncensored)
subscribe
How to add a robots.txt to your Django site - Adam Johnson
https://adamj.eu/tech/2020/02/10/robots-txt/
links
backlinks
robots.txt is a standard file to communicate to “robot” crawlers, such as Google’s Googlebot, which pages they should not crawl. You serve it on your site at the root URL /robots.txt, for example https://example.com/robots.txt.
Roast topics
Find topics
Find it!
