Reddit is ramping up its crackdown on web crawlers. Over the past few weeks, Reddit has started blocking search engines from surfacing recent posts and comments unless the search engine pays up, according to a report from 404 Media.
Right now, Google is the only mainstream search engine that shows recent results when you search for posts on Reddit using the “site:reddit.com” trick, 404 Media reports. This leaves out Bing, DuckDuckGo, and other alternatives — and that’s likely because Google has struck a $60 million deal that lets the company train its AI models on content from Reddit.
It’s a bold move for a massive website like Reddit to block some of the most popular search engines, but it’s not all that surprising. Over the past year, Reddit has become more protective of its data as it looks to open up another source of revenue and appease new investors. After making its API more expensive for some third-party developers, Reddit reportedly threatened to cut off Google if it didn’t stop using the platform’s data to train AI for free.
The Verge reached out to Reddit with a request for comment but didn’t immediately hear back.
Last month, to enforce its policy against scraping, Reddit updated the site’s robots.txt file, which tells web crawlers whether they can access a site. “It’s a signal to those who don’t have an agreement with us that they shouldn’t be accessing Reddit data,” Ben Lee, Reddit’s chief legal officer, told my colleague Alex Heath in Command Line.
With AI chatbots filling the internet with questionable content, finding things written by a fellow human has never been more important. I, like many others, have started appending “Reddit” to many of my searches just to get human answers, and it’s pretty frustrating to know that I’ll now only be able to do that on Google (or search engines that rely on it) — especially when I do many of my searches on Bing.
(Originally posted by Emma Roth)