this post was submitted on 01 Aug 2024
552 points (97.8% liked)

Technology

59039 readers
3763 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago
MODERATORS
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
552
Blocking AI bots from Microsoft, others has been “pain in the a**”: Reddit CEO | Huffman says companies must pay to scrape Reddit data even though Reddit itself relies on free, user-generated content (arstechnica.com)
submitted 3 months ago* (last edited 3 months ago) by [email protected] to c/[email protected]
85 comments fedilink hide all child comments
 

Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren't Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: "Reddit believes in an open Internet, but not the misuse of public content." Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It's unclear how much Reddit's OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 425 points 3 months ago* (last edited 3 months ago) (3 children)

Honestly, any platforms hosting user-generated content who use the legal argument that they only provide hosting and aren't responsible for what their user post shouldn't also be able to sell the same data and claim owning any of it.

Otherwise, take away their legal immunity. Nazis or pedophiles post something awful? You get in front of the judge.

edit: typo

[–] [email protected] 180 points 3 months ago (1 children)

Exactly this. You can claim that their scraping is abusing your servers, but the moment you claim copyright for the content of the site, then you give up your Section 230 rights.

[–] [email protected] 35 points 3 months ago (2 children)

You'd also probably lose a whole lot more processing power trying to stop the crawlers vs just letting them have API access with some sort of limit to queries.

[–] [email protected] 29 points 3 months ago* (last edited 3 months ago)

Eh, not really.

I block bot user agents to my Lemmy instance, and the overhead is pretty negligible for that (it's all handled in my web firewall/load balancer).

Granted, those are bots that correctly identify themselves via user agent and don't spoof a browser's.

It's also cheaper and easier to add another load balancer than size up or scale out my DB server to handle the bot traffic.

[–] [email protected] 6 points 3 months ago (1 children)

I don't think they actually block malicious bots, the change they've made is just to the robots.txt, they don't have to do anything.

[–] [email protected] 10 points 3 months ago (1 children)

Robots.txt does literally nothing. It's a piece of courtesy that's easily ignored if you don't care.

[–] [email protected] 4 points 3 months ago* (last edited 3 months ago) (1 children)

Yeah but it stops bing and a bunch of AI scrapers that want to act like they're following the rules

[–] [email protected] 0 points 3 months ago (1 children)

How do we know it stops bing? As far as anyone knows they could have instructed their programmers to alter the crawlers so that it ignores robots.txt when on Reddit - that should have taken them a whole 2 minutes.

Reddit blocking any search crawler via robots.txt is such a non thing that it shouldn't even be reported.

[–] [email protected] 38 points 3 months ago (1 children)

Can't sell something you don't own.

So if they're selling the parts people want, they need to own the parts no one wants.

[–] [email protected] 3 points 3 months ago* (last edited 3 months ago) (1 children)

Well, you can give money to Reddit for a piece of paper, but unless Reddit is claiming copyright to the content posted there, then they can't sue anyone for not paying. It would be very interesting to see the text of these "licensing agreements".

[–] [email protected] 4 points 3 months ago

They're not claiming copyright. They have a perpetual, non-revokable license to the content, granted by the people who use their site when they post the content.

[–] [email protected] 4 points 3 months ago

Good point!