What is web scraping — and is it even allowed?

Here's a fact that changes the picture: everything you see on a website has already been downloaded to your device. Prices, schedules, job listings — your browser received them as text and only then painted them nicely. Web scraping is when a program reads that same text instead of a human. No browser window, no eyes, no fatigue.

That's how "track your competitors' prices" services work, how ticket aggregators work, and how half of all notification bots work. Let's break down how it's built, where the limits are — and why AI recently changed the rules.

What web scraping means, in plain words

Web scraping is automated collection of data from web pages. The program does three steps:

• Download the page. The same request a browser makes — just without drawing anything on screen.
• Find what you need. In the downloaded text, locate the price, the title, or the date.
• Save it. Put the findings into a table or JSON — and repeat tomorrow.

Doing one pass by hand would take you an evening. A program does it in seconds — and not for one page, but for a thousand.

How it works under the hood

The page a scraper downloads is HTML: text with markup, where every piece has its place. A product's price doesn't live "somewhere on the page" — it lives in a specific element: roughly, "the third cell inside the product block."

A classic scraper is written exactly like that: "take the block with class price, pull out the number." That's called a selector — an element's address inside the page.

And right there is the big weakness. The site ships a redesign, the block gets renamed — and the scraper silently starts bringing back nothing. Not an error — nothing: the address exists, the apartment doesn't. That's why scrapers used to be a never-ending construction site: sites change, selectors break.

Where AI comes in

A language model reads a page by meaning, not by addresses — like a human. The prompt "here's the page text, return the product name and price" survives any redesign: the model finds the price wherever it stands.

The cost is money and speed. Running a thousand pages through a model is slower and pricier than running selectors. So mature projects often go hybrid: bulk collection with the classic approach, while AI repairs the spots where the layout changed — or reads pages that have no clean structure at all.

Is it allowed?

The honest answer: it depends on what you collect and how. No legal advice here, but three guideposts are worth knowing.

• The site's rules. Many services state in their terms of use that automated collection is forbidden. Break that, and you're looking at a banned account and blocked requests at minimum.
• Personal data is a separate, strict story. Collecting people's names, phone numbers, and profiles is nothing like collecting kettle prices. Most countries regulate it by law, and "the data was public" is not a free pass.
• Politeness is measurable. A site's robots.txt file tells robots where they may go. Request frequency is on you: a hundred requests per second from one address looks like an attack, and the site turns on its defenses — meet the rate limit.

A practical beginner's rule: collect public numbers and text for yourself, put pauses between requests, and don't touch personal data.

When you shouldn't scrape at all

The most common beginner mistake is writing a scraper for a site that already hands out its data. Many services have an API — an official door for programs: data arrives as clean JSON, nothing breaks on a redesign, and the usage rules are written down explicitly.

So the first step of any data project isn't code — it's a search: "service-name API." If there's a door, use the door. Scraping is the window for when there isn't one.

Are parsing and scraping the same thing?

In conversation — yes: "parsing a site" usually means what's formally called web scraping. Strictly speaking, parsing is analyzing text you've already downloaded, while scraping is the whole collection process.

Can a site tell I'm a bot?

It can: by request frequency, missing browser fingerprints, and repetitive paths. That's why large sites show bots a captcha or serve empty pages. Pauses between requests and modest volumes are the best way to stay off that radar.