[RFC Home] [TEXT|PDF|HTML] [Tracker] [IPR] [Info page] | www.rfc-editor.org
Should you allow AI bots to scrape your website? Find answers, along with a guide on how to block ChatGPT, Gemini other AI bots here.| 20i
| ec.europa.eu
THE EUROPEAN PARLIAMENT AND THE COUNCIL OF THE EUROPEAN UNION,| eur-lex.europa.eu
On Wednesday, Parliament approved the Artificial Intelligence Act that ensures safety and compliance with fundamental rights, while boosting innovation.| www.europarl.europa.eu
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.| www.rfc-editor.org
