Reddit plans to update robots.txt to block data scraping, protecting content from unauthorized use.
Reddit plans to update the Robots Exclusion Protocol (robots.txt), a web standard that determines which parts of a site can be crawled, to block automated data scraping from its website. This move follows reports that AI startups have been using the protocol to gather content without crediting or asking for permission, leading to accusations of plagiarism. Reddit's update aims to protect its content from unauthorized use, while ensuring that researchers and organizations like the Internet Archive can still access it for non-commercial purposes.
June 25, 2024
14 Articles