A note before you read
This page is a plain-English summary for developers, not legal advice. The law differs by country and changes with new cases. For a specific project, speak to a lawyer.
The key US cases
| Case | Year | What it decided |
|---|---|---|
| Van Buren v. United States | 2021 | The Supreme Court read the Computer Fraud and Abuse Act narrowly: "exceeding authorised access" means entering parts of a system that are off limits to you, not misusing data you are allowed to see. |
| hiQ Labs v. LinkedIn | 2019, 2022 | The Ninth Circuit held that scraping public profiles is likely not "without authorisation" under that Act. The case still ended with hiQ liable for breaching LinkedIn’s user agreement through fake accounts, and a settlement in December 2022. |
| Meta v. Bright Data | 2024 | A federal court found that scraping public data while logged out did not breach Meta’s terms. Meta dropped its remaining claims. |
| X Corp v. Bright Data | 2024 | A federal court dismissed X’s claims, finding its contract claims against scraping public posts were largely pre-empted by copyright law. |
The common thread: reading public pages is generally not hacking. Creating fake accounts, logging in to reach data, or ignoring terms you agreed to is where cases have been lost.
Privacy law still applies
In the UK and EU, public does not mean free of GDPR. If scraped data identifies people, you need a lawful basis, must be transparent, and must honour people’s rights.
- In August 2023, twelve data protection authorities, including the UK ICO, said publicly accessible personal data is still protected and that sites should guard against unlawful scraping.
- Clearview AI, which scraped billions of photos to build a face search tool, was fined by several regulators, including €30.5 million by the Dutch authority in 2024.
- The ICO has said legitimate interests is the only realistic lawful basis for scraping to train generative AI, and only if it passes its balancing test.
Terms of use and copyright
Terms of use are a contract, and they bind people who agree to them, for example by creating an account. In the EU, the Court of Justice held in Ryanair v PR Aviation (2015) that a site can restrict use of an unprotected database by contract.
Copyright protects creative content such as posts, photos and videos. Facts such as follower counts are not protected in the same way. Collecting data for analysis is different from republishing someone’s work.
Lines to stay behind
- Collect only data visible without logging in.
- Never create fake accounts or share logins to reach data.
- Don’t bypass passwords, paywalls or other access controls.
- Take the minimum personal data you need, and have a lawful basis for it.
- Keep request rates reasonable, and don’t republish copyrighted content wholesale.
In Monocrawl
How Monocrawl approaches public data
Monocrawl returns publicly available data only. It does not log in to platforms on your behalf or reach private accounts, and a publisher’s refusal comes back as TARGET_ACCESS_DENIED at zero cost rather than being worked around.
Our public data policy explains what we collect, how long it is kept and how people can ask for data about them to be removed.
Common questions
Is it legal to scrape LinkedIn?
Courts have said reading public profiles is likely not hacking, but LinkedIn actively sues scrapers that use fake accounts or breach its terms, and privacy law applies to profile data.
Does robots.txt make scraping illegal?
robots.txt is a convention, not a law. Ignoring it can still weigh against you in a dispute, so treat it as a signal of the site’s wishes.
Can I scrape data to train an AI model?
It depends heavily on the country, the content and whether personal data is involved. In the UK and EU, privacy regulators expect a lawful basis and a balancing test.
Sources
- Van Buren v. United States, 593 U.S. 374 (2021)
- hiQ Labs v. LinkedIn, 9th Cir. (18 Apr 2022)
- Proskauer: hiQ and LinkedIn settlement (Dec 2022)
- Farella Braun + Martel: Meta v. Bright Data decision (2024)
- Skadden: X Corp v. Bright Data dismissal (May 2024)
- Joint statement on data scraping (24 Aug 2023)
- EDPB: Dutch authority fines Clearview AI (2024)
- ICO: Lawful basis for web scraping to train generative AI
- CJEU, Ryanair v PR Aviation, C-30/14 (2015)