RFC to introduce Datasets + Common Crawl as a Dataset

[desmondcheongzx]

Motivation

• Common Crawl is one of the most popular and important open web datasets, used for large-scale language model pretraining, search engines, and web-scale analytics. It is stored as WARC (Web ARChive format)/WAT (Web ARChive Timestamp)/WET (WARC Encapsulated Text) files across many segments.
• Today, Daft users can already read Common Crawl files using daft.read_warc, but they must manually construct complex S3 glob patterns to target specific segments and file types. By making Common Crawl a first-class citizen in Daft, we can dramatically simplify this workflow and make the dataset more accessible to researchers and practitioners.
• Daft is highly performant for streaming workloads, with efficient pipelining for downloading, decompressing, and parsing WARC files. It is also well-suited for large-scale, distributed data processing scenarios common in web-scale analytics.
• Reading from Common Crawl at scale should be done responsibly. Daft already has robust retry/backoff behavior and can respect rate limits (e.g., HTTP 429s), which makes it a good fit for sustainable access.
• Many public web-scale datasets (Common Pile, OSCAR, etc.) follow similar distribution patterns. By designing a clean integration for Common Crawl, we create a template for supporting these other datasets.

Current user experience

• Today, even a simple experiment (e.g. fetching 100 WET files from the latest crawl) requires digging through Common Crawl’s indexes and manually crafting S3 paths.
• Even when using Daft, Daft only provides the read_warc function and does not simplify any of the user experience of how to obtain the WARC file to read.

Goal

Provide a simple, ergonomic way to access Common Crawl from Daft. For example:

# Create a dataframe over a crawl.
daft.datasets.common_crawl("CC-MAIN-2025-33")

# Create a dataframe over a specific segment.
daft.datasets.common_crawl("CC-MAIN-2025-33", segment="1754151279521.11")

# Specify full crawl data vs text vs metadata-only
daft.datasets.common_crawl("CC-MAIN-2025-33", content="metadata")

# Specify number of files to sample.
daft.datasets.common_crawl("CC-MAIN-2025-33", num_files=1)

# Sharded across a cluster. Shard takes (shard_id, world_size).
daft.datasets.common_crawl("CC-MAIN-2025-33", shards=(2, 8))

# Find top MIME types.
(
    daft.datasets.common_crawl("CC-MAIN-2025-33", num_files=10)
    .select(daft.col("WARC-Identified-Payload-Type"))
    .groupby("WARC-Identified-Payload-Type")
    .agg(daft.col("WARC-Identified-Payload-Type").count().alias("count"))
    .sort("count", desc=True)
    .show()
)

This should “just work” without requiring users to know S3 bucket structure or individual segment URLs.

Additional Common Crawl background

• Segment: Each crawl release is split into 100 segments. Each segment contains 1000 files.
• WARC, WAT, WET:
  • WARC files store the raw crawl data.
  • WAT files store computed metadata for the data stored in the WARC.
  • WET files store extracted plaintext from the data stored in the WARC.
  • All WARC/WAT/WET files are in the WARC format.

Proposal

We can introduce the concept of Datasets to Daft and create abstractions for things like Hugging Face and Common Crawl etc without users needing to call read_warc, read_parquet etc.

Common Crawl would be a fantastic first Dataset to test drive this idea. We can do something like:

daft.datasets.common_crawl

Load Common Crawl data as a DataFrame. This function automatically resolves the specified crawl and segment into the appropriate Common Crawl files and loads them as a DataFrame, handling the WARC reading process internally.

Arguments

• crawl: The crawl identifier, e.g. "CC-MAIN-2025-33".
• segment: Specific segment to fetch within the crawl. If not provided, defaults to all segments in the crawl.
• content: Specifies the type of content to load. Options are:
  • "raw" or "warc": Raw WARC files containing full HTTP responses
  • "text" or "wet": Extracted plain text content
  • "metadata" or "wat": Metadata about crawled pages
• num_files: Limit the number of files to process. If not provided, processes all matching files.
• io_config: IO configuration for accessing S3.

Returns

A DataFrame containing the requested Common Crawl data.

---

Action items

• [P0] Implement common_crawl() to resolve crawl → dataframe
• [P0] Add num_files, segment, content, io_config support
• [P1] Further optimizations for retrieving manifest data (pushdown num_files limit into manifest retrieval stage)
• [P1] Integration with CC-index to support point lookups over URLs
• [P1] Accept multiple crawl dates and segments
• [P1] Validate 429 behaviour
• [P2] Performance improvements for common_crawl(...).show() (In particular, limit pushdowns into the WARC reader)
• [P2] shards support

Open questions:

• Do we want shards support?
• Should we also add metadata discovery? E.g. list_crawls or list_segments?
• What other features are missing?

[desmondcheongzx]

A concrete implementation of this proposal might look like #5244

[desmondcheongzx]

This dataset has been released :) https://docs.daft.ai/en/stable/datasets/common-crawl/