Common Crawl

The Common Crawl (CC) project browses and indexes all content available online. It generates 200-300 TiB of data per month (around 5% of which is in French), and constitutes the bulk of most NLP datasets nowadays.

Data and Resources

Cite this as

Julien Launay, Elena Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, Djamé Seddah (2024). Dataset: Common Crawl. https://doi.org/10.57702/lau3rl6z

DOI retrieved: December 2, 2024

Additional Info

Field Value
Created December 2, 2024
Last update December 2, 2024
Defined In https://doi.org/10.48550/arXiv.2201.06642
Citation
  • https://doi.org/10.48550/arXiv.2311.04711
Author Julien Launay
More Authors
Elena Tommasone
Baptiste Pannier
François Boniface
Amélie Chatelain
Alessandro Cappelli
Iacopo Poli
Djamé Seddah
Homepage https://commoncrawl.org/