Meta launches Meta External Agent, a web crawler bypassing Robots.txt for data collection.

Meta has released a new web crawler called Meta External Agent to gather data from the internet, in particular from publicly displayed content such as news articles and online discussions. The crawler collects data for Meta's AI model, bypassing standard website protection measures relying on Robots.txt files. This move demonstrates Meta's commitment to training and improving its AI models.

August 21, 2024
6 Articles