this post was submitted on 29 Jul 2024
36 points (100.0% liked)

TechTakes

1430 readers
145 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 1 year ago
MODERATORS
top 1 comments
sorted by: hot top controversial new old
[–] [email protected] 23 points 3 months ago

I've kinda said this before, but I expect shit like this to feed a growing trend of treating all crawlers as malicious until proven otherwise, and blocking them accordingly - even if it causes serious problems, its arguably better than the risk of getting fucked by an AI crawler.

I also expect robots.txt to fall out of use as a consequence of this plague of AI crawlers. Between OpenAI openly ignoring it, Perplexity openly deceiving it, and Anthropic spamming crawlers to get past it, the trust necessary for robots' "voluntary compliance" model to work has been thoroughly fucking shredded.