Reddit has up to date its robots.txt file, stopping Bing and plenty of different engines like google from crawling the location. “Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their web site,” a Microsoft consultant informed Search Engine Land.
What occurred. On July 1, 2024, Reddit up to date its robots.txt file stopping many engines like google and AI instruments from crawling the location. Reddit didn’t stop Google from crawling the location, regardless of what some individuals might have thought earlier this month. However Reddit did stop most different crawlers from crawling the location.
Earlier this morning, Mark Williams-Prepare dinner notified me that Reddit outcomes have been dropping out of the Bing Search index. Then a number of media retailers started protecting the information. I needed to verify that Bing’s crawlers have been certainly blocked, as a result of Reddit was utilizing IP detection to indicate engines like google one model of its robots.txt file and people one other model of the robots.txt file – as I defined earlier this month.
Bing has thus stopped crawling new content material on Reddit, which is why while you filter Reddit ends in Bing Seek for the final week, you see nothing:
Microsoft confirmed. A Microsoft spokesperson informed Search Engine Land:
“We respect the robots.txt commonplace. Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their web site.”
Reddit assertion. Reddit spokesperson Tim Rathschmidt says in an announcement to The Verge:
“This isn’t in any respect associated to our current partnership with Google. We now have been in discussions with a number of engines like google. We now have been unable to achieve agreements with all of them, since some are unable or unwilling to make enforceable guarantees concerning their use of Reddit content material, together with their use for AI.”
Why we care. With Reddit securing a licensing take care of Google, Reddit is ready to play hardball with different engines like google and AI instruments. So Reddit has blocked most different engines like google from crawling its content material. In the meantime, Google is driving insane site visitors to Reddit lately, together with testing particular remedy in its search outcomes for them.
It makes you surprise if different massive web sites can attempt to go down this route and the place which may go away smaller publishers and content material producers.
In the meantime, don’t anticipate to see a lot new Reddit content material being surfaced on Bing within the close to future.
