I loved scraping until my ip was blocked for botting lol. I know there's ways around it it's just work though

Or in the case of wikipedia, every table on successive pages for sequential data is formatted differently.

Relevant SO post. https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags#1732454

But, but, norobots.txt!

Chad doesn't care!

Much less beholden to arbitrary rules also. Way too many times companies will just up and lift their API access or push through restrictions. No ty, I'll just access it myself then

you scrape WITH BASH?

there's literally dozens of us!

or maybe just 2 idk

I created a shitty script (with ChatGPT's help) that uses Selenium and can dump a Confluence page from work, all its subpages and all linked Google Drive documents.

Why on earth would they have changed that. WEBooB is a way better name.

Scrapers have been a thing since the web exists.

One of the first search engines is even called WebCrawler

Websites and services create APIs for programmers to use them. So Spotify has code that let's you build a program that can use its features. But you need a token they give you after you sign up. The token can be revoked and used to monitor how much of their service you're using. That way they can restrict if its too much.

Scraping is raw dogging the web slut you met at the cougar ranch who went home with you because you reminded her of her dog

'Scraping' is the process of anonymously and programmatically collecting data from a webpage(s), often without the website's permission and only limited to the content made publicly available. This is in contrast to using an API provided by the database owner which is limited by tokens, access volume, available end points etc.

By getting someone to hire you to do it.

Imagine an investment firm looking at a property market. They need data like price trends in the surrounding area.

Real estate API is expensive, scraping is free. By hiring an employee the can save money.