Last year, Robb Knight figured out how Perplexity, an artificial intelligence search engine, was evading instructions not to crawl particular sites. Knight learned that Perplexity’s engine would use an unlisted user agent to scrape summaries of pages on websites where Perplexity was blocked. In my testing, I found the summaries were outdated by hours-to-days, indicating […]
