About Common Crawl
Common Crawl is a non-profit that publishes a free, open collection of web crawl data. Anyone can use the archive for research or analysis.
What it does
The project crawls the public web and stores the pages in large archive files. Researchers and developers can then process that data to study language, web content and how the internet changes over time. Each crawl is published so the same material can be reused by other people, which makes results easier to check and compare.
Key features
- Large archives of crawled web pages
- Crawl history that covers many years
- Search and a curated list of papers that use the data
- Blog tools such as warc2zip for working with archives
Who it's for
Academic researchers, data scientists and students who need large amounts of real web text. It also suits teams that build language or search systems and want an open corpus to test ideas on.
Pricing and good to know
The archive is free to use and open to anyone. The files are large, so processing them usually needs considerable storage and computing power. New pages are added on a regular basis, which means a dataset built from one crawl will differ from one built from another.
Highlights
- Large archives of crawled web pages
- Crawl history that covers many years
- Search and a list of research papers that use the data
- Tools such as warc2zip for working with archives
