Making a Pretraining Dataset: CommonCrawl Processing at Scale

This is a recipe for step-by-step development of a pretraining dataset. We will walkthrough a small dataset that can be fully processed locally, or on a single remote instance, with callouts to pieces of the process that require special care for extreme scalability. We will focus on just the CommonCrawl aspects of this process.

Recipe Shorthand​

Before diving into the low-level details of each step, we first summarize each step applied in turning web archive files from commoncrawl into a tokenized dataset ready to be used in training:

• Linearization: Commoncrawl provides data in terms of raw web archive data, riddled with HTML tags and other metadata not directly helpful for training. Linearization is the process of text extraction from these WARC/WET files
• Heuristic Filtering: To convert the large, linearized dataset into something much more manageable, we apply a round of highly-efficient, but coarse, filtration techniques to remove data that is of low-quality or non-English.
• Deduplication: Duplicate data has been shown to be an inefficient use of training tokens. We apply three rounds of deduplication: first an exact-deduplication to remove identical documents; then a MinHash-based fuzzy deduplication to remove documents that are "nearly identical"; then a suffix-array based deduplication to remove repeated substrings within the surviving documents.
• Categorization and Quality Partitioning: We apply two separate classifiers to each remaining document to first classify the data into WebOrganizer categories and into quality buckets within each category. This coarse-grained partitioning of the data makes it amenable to mixing and upsampling.
• Mixing: The natural data distribution of CommonCrawl isn't optimized for language model pretraining. To rectify this, we filter and upsample documents according to their category and quality bucket from the previous step.
• Tokenization: Finally the data needs to be converted from human-readable text to token files before it can be passed off to training.

Tooling​

There are four main toolkits used for data preprocessing:

• Dolma Toolkit: Linearization tools to bring WARC/WET files into Resiliparse .jsonl files.
• Datamap-rs: Tooling for heuristic filtering and a large swath of miscellaneous
• Duplodocus: A toolkit for highly-efficient exact and minhash deduplication.
• Bsade: Suffix Array deduplication tooling

As a quick-hit, we recommend the following tools for general-purpose data-processing:

• uv: Python package management, done correctly
• s5cmd: For interactions with AWS S3, this go-based tool is much much faster than the AWS CLI
• i4i/i7i setup script: We use this script everywhere to glue all AWS Nitro drives on a given i4i/i7i together into a single Raid0 drive

Resource Requirements​

At Ai2 we generally operate with a data processing stack built on AWS. We heavily leverage S3 for storage of datasets and any intermediate artifacts. All CPU-heavy data-processing work is done using EC2 instances. We find that i4i and i7i instances, with their attached AWS Nitro drives, are a cost-effective choice for large-scale data processing workloads. To get a sense for the demands of a data processing workload, to process the 104 CommonCrawl shards used in building the CC section of Dolma 3, the total workload demand was ~

For the purposes of the rest of this walkthrough, we'll be working with a data pool that starts as 5GB of compressed jsonl files which admits the following runtimes on a single i4i.32xlarge instance on EC2.

Linearization:​

For Dolma 3, we used a total of 104 Commoncrawl dumps, with dates ranging from CC-MAIN-2013-20 until CC-MAIN-2024-51. For most of these crawls, we used the publicly available Resiliparse-extracted texts from the DCLM Pool for Commoncrawl dumps until the DCLM 2023 cutoff. For more recent dumps, we used the Dolma toolkit to extract the plain text. The command should look something like this:

uv init dolma-toolkit
cd dolma-toolkit
uv add dolma

export CRAWL_NAME=CC-MAIN-2024-51
export DESTINATION=/path/to/linearized/data
uv run dolma warc \ 
  --documents "s3://commoncrawl/crawl-data/$CRAWL_NAME/**/warc/*warc.gz" \
  --processes $(($(nproc --all) - 1)) \
  --source_name $CRAWL_NAME \
  --linearizer resiliparse \
  --destination $DESTINATION 

Note: For the purposes of this demo-walkthrough, we recommend not calling the above command, and instead using our pre-extracted demo-pool which we introduce in the next section.

Heuristic Filtering​

Demo Pool​

For the remainder of this document, we'll be working with a demo pool. This is only 12GB in size, spread over 256 files. This was created by taking the crawl=CC-MAIN-2022-40 shard of DCLM-Pool, accessible at s3://commoncrawl/contrib/datacomp/DCLM-pool/jsonl/crawl=CC-MAIN-2022-40/* and randomly subsampling 0.25% of the documents. To download the demo pool, set the path to the demo pool variable and run this script:

export DEMO=path/to/demo/pool
seq -f "%03g" 0 255 | xargs -P 8 -I {} wget -P "$DEMO" "https://olmo-data.org/pretraining-data/dolma3-doc-example-pool/2022_40_0p25sub/shard_00000{}.jsonl.zst"

Even after HTML artifacts are removed, datasets are often too large and contain mostly low-quality data that is easily identifiable. Following suit of several other common processing pipelines (see RefinedWeb, DCLM, FineWeb), we apply a set of heuristic filters to catch and remove much of this obviously bad data, in addition to performing language filtering to select only . We use a bespoke native-Rust toolkit to handle these steps. The specific steps to process this collection of documents is:

export SRC=path/to/demo/pool        # change with where the demo pool lives
export DST=path/to/processed/data   # change with where the output should live
git clone https://github.com/allenai/datamap-rs.git 
cd datamap-rs
git checkout release
bash configs/all_dressed/download_lid.sh # downloads FastText language classification
cargo run --release -- map --input-dir $SRC --output-dir $DST --config configs/all_dressed/all_dressed_config.yaml

This puts the output data in the $DST folder. For every step that removed some documents, the collation of those removed documents is in $DST/step_XX (for documents removed during the XX'th step), where the data that survives all filters is located in $DST/step_final.

Deduplication: Exact Deduplication​

We perform deduplication in three rounds of varying coarseness. The first round identifies and removes any documents that are exact copies of each other, down to byte-wise equality over their content strings only. The heuristic filtering step annotates each jsonl with hashes according to the text content, and these hashes are used to perform exact deduplication. We note that exact deduplication can be subsumed entirely by a MinHash fuzzy deduplication step. However, we found that the quantity of exact-duplicates in CommonCrawl is large enough that it is worth it, in terms of compute-efficiency, to perform exact-deduplication to prune the pool before running the more expensive MinHash fuzzy deduplication.

To perform exact deduplication for documents that have text hashes located in the field metadata.text_hash, the steps are:

export SRC=path/to/processed/data/step_final # Location of heuristic-filtered processed data
export DST=path/to/exact_dedup_data/

git clone https://github.com/allenai/duplodocus.git
cd duplodocus
git checkout main 
cargo run --release -- exact-dedup-memory \
  --input-dir $SRC \
  --output-dir $DST \
  --hash-key metadata.text_hash

Which should yield output printouts like

Finished exact deduplication in 2 seconds
Saw 908802 documents, and kept 883876 of them
Removal rate was 2.74%

Noting that, while this deduplication rate seems really low, remember that this pool we're working with has been aggressively subsampled to be tractable as an example pool. Subsampling is itself deduplication, and larger pools should have a larger deduplication rate!

Deduplication: MinHash Fuzzy Deduplication​

The second round of deduplication we perform is a MinHash deduplication to identify documents that are very similar but may not be exact duplicates. The canonical example of a use case such as this is when the main text of a document is copied across multiple web pages, but there are minor differences in the header and footer text -- these are different documents, but all the primary semantic content is directly duplicated. Our process for deduplication involves several steps: we first start by running MinHash to identify documents that are near-duplicates of each other; then we reorganize our dataset such that these clusters of near-duplicates can be examined in-memory, and then we selectively keep the most recent document from each cluster.

There is a significant amount of engineering to run the MinHash algorithm, but for small-to-medium sized datasets, annotation with MinHash cluster ID's can be performed with a single line. This imbues all documents that have fuzzy duplicates with an annotation in the field metadata.minhash that keeps track of the duplicate cluster (connected component/cc) ID, size, and index within the cluster:

export SRC=/path/to/exact_dedup_data/
export TEMP=/path/to/temporary/storage # <-- some auxiliary data needs to be stored, this gets cleaned up later
export DST=/path/to/minhash_anno_data/
cd duplodocus # clone this repo from https://github.com/allenai/duplodocus.git if does not exist
git checkout main
cargo run --release -- minhash-memory \
  --input-dir $SRC \
  --storage-dir $TEMP \
  --output-dir $DST \
  --num-buckets 14 \
  --bucket-size 8 \
  --ngram-size 5 \
  --num-sig-chunks 64 \
  --tokenizer cl100k \
  --annotate true \
  --annotate-key metadata.minhash \
  --remove-duplicates false \
  --cleanup-storage 

This should print out many things to mark the progress, but the salient feature should be a few lines like:

Saw 883876 docs
Removed 32868 docs
Removal rate would be 3.7186213%
Finished full minhash deduplication in 23 seconds

Again, noting that this deduplication rate is quite small since the data pool we're working with is an aggressive subsample of a CommonCrawl dump.

The next step is to group the data according to the MinHash cc-id, and keep only the most recent document from each cluster. For this we turn back to the datamap-rs tool.

export SRC=/path/to/minhash_anno_data/
export GROUP_DIR=/path/to/temporary/storage/ # <-- some auxiliary data needs to be stored
export DST=/path/to/minhashed_data/
cd datamap-rs 
git checkout release 
cargo run --release -- group \
  --input-dir $SRC \
  --group-dir $GROUP_DIR \
  --config configs/all_dressed/minhash_groupfilter_config.yaml 

cargo run --release -- group-filter \
  --input-dir $GROUP_DIR \
  --output-dir $DST \
  --config configs/all_dressed/minhash_groupfilter_config.yaml

Which should ultimately yield an output printout like:

Finished filtering in 2 secs
Saw 883876 docs
Kept 880549 docs

We note that running MinHash deduplication at a mega-scale requires much more sophisticated engineering and deserves a documentation page of its own (coming soon!).

Deduplication: Substring Deduplication​

The third and final round of deduplication we apply is a fuzzy substring deduplication, where we remove repeated sequences of tokens within documents. This case occurs when many web pages have been scraped from the same parent website such that the main semantic content differs in each of these documents, but there are repeated, identical headers and footers -- this round of deduplication specifically removes just these headers and footers. The primary machinery behind this step of deduplication is the suffix array. For small-scale experiments, a suffix array can be built over the entire corpus; but for large scale experiments, the corpus must be partitioned into smaller shards that are indepedently deduplicated with suffix arrays. The first step is to construct the suffix arrays and annotate each document with which byte ranges occur more than once:

UPDATE FOR DOLMA3.5: We have folded substring deduplication capabilities into duplodocus. This now can be run with the following steps:

cd duplodocus # clone this repo from https://github.com/allenai/duplodocus.git if does not exist
git checkout main

export MATCH_LEN=500
export REP_COUNT=2

export SRC=/path/to/minhashed/data/
export INDEX_DIR=/path/to/index/storage # Temporary storage, can be deleted later
export DST=/path/to/sa/annotated/data


cargo run --release -- sa-make-tables --input-dir $SRC --output-dir $INDEX_DIR --batch-size 0.0  # Makes suffix tables
cargo run --release -- sa-get-matches-parallel --storage-dir $INDEX_DIR --match-length $MATCH_LEN --rep-count $REP_COUNT
cargo run --release -- sa-merge-matches --storage-dir $INDEX_DIR 
cargo run --release -- sa-annotate --input-dir $SRC --storage-dir $INDEX_DIR --output-dir $DST --annotate-key metadata.sa

rm -rf $INDEX_DIR 

Once the data has been annotated with the ranges to be removed, we need to make a pass through the data and actually remove the repeated substrings. We apply a "fuzzy substring deduplication" procedure, where we remove the substrings that are marked by the suffix array, but also make sure that we don't remove intermediate strings, choosing to remove semantically coherent blocks. We have found that removing entire paragraphs (snapping the "removal range" to the nearest paragraph boundary), and removing documents entirely if substring deduplication decimates them, yields the best downstream results. To run the removal step, call:

export SRC=/path/to/sa/annotated/data
export DST=/path/to/sa/filtered/data

cd datamap-rs
git checkout main
cargo run --release -- map --input-dir $SRC --output-dir $DST --config SA_CONFIG.yaml

where the contents of SA_CONFIG.yaml look like:

name: sa_bytes
text_field: text
pipeline:
  - name: sa_byte_modifier
    kwargs:
      suffix_array_key: metadata.sa 
      text_key: text
      old_text_key: pre_sa_key
      check_paragraph: true
      check_newline: false
      check_sentences: false
      check_width: 50
      metadata_key: metadata.sa_stats
      gap_width: 20
      proportion_removal: 0.1
      doc_min_length: 200

Mixing: Quality Classification and Categorization​

With a sufficiently deduplicated pool, we can now prepare our pool for mixing. Mixing is the procedure in which we alter the distribution of our data pool by either filtering or selective upsampling in order to yield a higher quality data pool for pretraining. We apply mixing in a groupwise fashion, where we partition data along two axes: a categorization based on their WebOrganizer categories and a general purpose quality classifier (similar to that used in DCLM). Within each category, we partition documents into 20 buckets based on their percentile of quality within the bucket. This section of the documentation will cover how to apply the quality and category labels to the data and then to partition the data into groups based on these categories and quality buckets.

First we need to download these models:

export SRC=/path/to/sa/filtered/data
export DST=/path/to/classified/data

cd datamap-rs
git checkout release
bash configs/all_dressed/download_ultrafineweb.sh 
bash configs/all_dressed/download_weborganizer_topic.sh

cargo run --release -- map --input-dir $SRC --output-dir $DST --config QC_CONFIG.yaml

where the contents of QC_CONFIG.yaml look like:

name: quality_classifiers
text_field: text
pipeline: 
  - name: fasttext_annotator
    kwargs:
      fast_text_file: ft_classifiers/ultrafineweb.bin
      output_field: metadata.ultrafineweb
  - name: fasttext_annotator
    kwargs:
      fast_text_file: ft_classifiers/wo_topic.bin
      output_field: metadata.weborganizer
  - name: max_extractor
    kwargs:
      main_attribute: metadata.weborganizer
      output_attribute: metadata.wo_topic

Then we partition the data into groups based on the most-likely topic according to the WebOrganizer topic:

export SRC=/path/to/classified/data
export DST=/path/to/partitioned/topic/data
cd datamap-rs 
git checkout main 
cargo run --release -- discrete-partition --input-dir $SRC --output-dir $DST --partition-key metadata.wo_topic --no-ft-label

And finally we need to split the data into groups based on percentiles of quality. For Olmo3, we broke the data into "vigintiles", or 20 groups of 5-percentile each. We do this for each topic separately, as some topics may have different ranges of quality. With the expectation of doing this at scale, where it is impossible to hold an entire corpus in main memory, we must approximate the vigintile boundaries to chunk the data based on quality scores. To remedy this, we utilize reservoir sampling to sketch the approximate boundaries, where we must do this for every WebOrganizer topic. The plan, then, is to create a reservoir sample of the quality-score for every WebOrganizer topic, calculate the vigintile boundaries from this sample, and then partition the data according to these boundaries. The script to do this is as follows:

export TOPIC_SRC=/path/to/partitioned/topic/data
export RES_DIR=/path/to/reservoirs/ # Temporary storage for reservoir statistics
export DST=/path/to/final/data

cd datamap-rs
git checkout main

for TOPIC_SUBDIR in $TOPIC_SRC*; do

  export TOPIC=$(basename $TOPIC_SUBDIR)
  export RESNAME="${RES_DIR%/}/${TOPIC}_res.json"
  export TOPIC_DST="${DST%/}/${TOPIC}"
  echo "Processing topic: ${TOPIC}"
  cargo run --release -- percentile-finder --input-dir $TOPIC_SUBDIR --output-file vigintiles.json --score-key metadata.ultrafineweb.__label__pos --num-buckets 20
  cargo run --release -- range-partition --input-dir /mnt/raid0/input_data --output-dir /mnt/raid0/partition --range-groups-json vigintiles.json --bucket-name vigintile
done

The ultimate output of this format should be a set of nested directories of the form $DST/$TOPIC/vigintile_XXX where there are at most 20 vigintiles per topic. Due to the nature of the quality score, it may be the case that some of the lower vigintiles may not exist, due to too many documents having quality at the minimum value.

Tokenization​

The final step to craft the pretraining mix is to tokenize each (topic, quality) group and prepare this group for a final mixing/upsampling phase, and ultimately pass it off to the trainer. Tokenization is performed using the dolma toolkit

export TOKEN_SRC=/path/to/data/subdir/
export TOKEN_DST=/path/to/token/subdir/
uv init tokenizer
cd tokenizer 
uv add dolma
uv run --python 3.12 --with "smart_open==7.1.0" dolma tokens \
    --documents $TOKEN_SRC \
    --destination $TOKEN_DST \
    --tokenizer.name_or_path 'allenai/dolma2-tokenizer' \
    --tokenizer.eos_token_id 100257 \
    --tokenizer.pad_token_id 100277 \
    --no-tokenizer.segment_before_tokenization \
    --tokenizer.encode_special_tokens \
    --ring_size 8 \
    --processes 120 \
    --max_size 4_000_000_000 \
    --sample_ring_prop \
    --dtype 'uint32'

Noting that the dolma tool will not respect subdirectory boundaries when tokenizing, so if following along with the previous steps, where you have over 400 (topic, quality) buckets, each of these will need to be tokenized individually. At a larger scale, one would want to source these tokenization jobs to a separate machine, but perform tokenization serially, the following script does the trick:

export SRC=/path/to/final/data
export DST=/path/to/tokens/
uv init tokenizer
cd tokenizer 
uv add dolma
for topic_dir in $SRC/*; do
	export TOPIC=$(basename $topic_dir)
	for quality in $topic_dir/*; do
		export QUALITY=$(basename $quality)
		export SUBDST="${DST%/}/${TOPIC}/${QUALITY}"
		uv run --python 3.12 --with "smart_open==7.1.0" dolma tokens \
		    --documents $quality \
		    --destination $SUBDST \
		    --tokenizer.name_or_path 'allenai/dolma2-tokenizer' \
		    --tokenizer.eos_token_id 100257 \
		    --tokenizer.pad_token_id 100277 \
		    --no-tokenizer.segment_before_tokenization \
		    --tokenizer.encode_special_tokens \
		    --ring_size 8 \
		    --processes 120 \
		    --max_size 4_000_000_000 \
		    --sample_ring_prop \
		    --dtype 'uint32'	
	done
done