Pretraining data for a 125.8M-parameter Llama-style small language model: streamed from three public sources, deterministically cleaned, deduplicated, decontaminated against CaseHOLD, tokenized with a purpose-built 16,384-token BPE, and packed into 1,024-token windows.
Sampled 2,000 documents per source to project capacity. This is what justifies a legal-first budget rather than a 70/20/10 target: the legal sources simply cannot supply 70% of a 10B-token corpus.
| Source | Hugging Face ID | Keep rate | Projected | Budget |
|---|---|---|---|---|
| case-law | HFforLegal/case-law | 0.744 | 0.81B | 1.00B |
| sec | PleIAs/SEC | 0.982 | 1.15B | 1.30B |
| fineweb-edu | HuggingFaceFW/fineweb-edu | 0.964 | 11.67B | 0.50B |
Twenty parallel workers streamed parquet shards directly into Modal containers — no dataset was downloaded in full. Each document passed a fixed deterministic chain: whitespace normalization, line-length and symbol-density filters, boilerplate removal, a 600-character minimum, a repetition test, ASCII-first English detection, and — for case law only — a dictionary-based OCR-garble gate.
| Source | Streamed | Kept | Keep % | too_short | non_english | ocr |
|---|---|---|---|---|---|---|
| case-law | 238,213 | 232,299 | 97.5% | 5,228 | 0 | 686 |
| sec | 47,782 | 47,229 | 98.8% | 553 | 0 | 0 |
| fineweb-edu | 432,790 | 418,341 | 96.7% | 14,336 | 113 | 0 |
| Total | 718,785 | 697,869 | 97.1% | 20,117 | 113 | 686 |
Scope check: OCR drops occur only on case law, and non-English drops only on web text — exactly as scoped.
MinHash signatures (32 permutations, 5-word shingles) over every case-law document feed one cross-shard LSH at similarity 0.8. Exact duplicates are caught by a 16-byte BLAKE2b digest. Contamination is removed by 13-word n-gram overlap against the CaseHOLD evaluation set, so benchmark text cannot leak into training.
| Source | Read | Kept | Near-dup | Exact dup | Contaminated |
|---|---|---|---|---|---|
| case-law | 232,299 | 183,259 | 1,606 | 0 | 47,434 |
| sec | 47,229 | 43,849 | 0 | 2,000 | 1,380 |
| fineweb-edu | 418,341 | 418,280 | 0 | 61 | 0 |
| Total | 697,869 | 645,388 | 1,606 | 2,061 | 48,814 |
casehold/casehold publishes eleven configs —
all plus fold_1…fold_10 — which are cross-validation
re-partitions of the same examples. Matching every config unioned all folds and tiled the
entire dataset into the contamination set: 7,809,123 13-grams instead of 1,824,747, removing
81,983 documents instead of 48,814. Pinning the holdout to config all recovered
32,644 legitimate case-law documents. Note also that LexGLUE case_hold contributes
zero unique n-grams — it is derived from CaseHOLD.
Honest caveat carried in the report itself: exact-hash state is per output shard, not global.
A fresh byte-level BPE trained on every corpus line, verified from the saved artifact rather than the training log.
| Property | Value | Status |
|---|---|---|
| Vocabulary size | 16,384 | ✓ exact |
| Merges | 16,121 | 7 specials + 256 bytes + 16,121 = 16,384 |
| Pre-tokenizer | ByteLevel, add_prefix_space=False | ✓ |
| Decoder | ByteLevel | ✓ |
| Legal round trip | 23 tokens | ✓ byte-exact |
| Financial round trip | 36 tokens | ✓ byte-exact |
| Max token id | 16,383 | ✓ uint16 valid |
Special tokens at ids 0–6: <|bos|> <|eos|>
<|pad|> <|unk|> <|user|>
<|assistant|> <|system|>
Fourteen workers encoded the corpus, appended <|eos|> after every document,
buffered across documents, and emitted complete 1,024-token uint16 windows —
every 100th routed to validation. Document counts reconcile exactly with Phase 2, confirming
the modulo sharding neither duplicated nor dropped a document.
| Source | Workers | Documents | True tokens | Share | Reference |
|---|---|---|---|---|---|
| case-law | 4 | 183,259 | 580,531,200 | 31% | 40% |
| sec | 6 | 43,849 | 820,358,144 | 44% | 40% |
| fineweb-edu | 4 | 418,280 | 467,992,576 | 25% | 20% |
| Total | 14 | 645,388 | 1,868,881,920 | 100% | — |
75% legal overall, against a reference of 78%.
.bin files stat to 3,737,763,840 bytes.
Every file size divides evenly by 2 and by the 1,024-token window; disk windows total
1,825,080, matching index.json exactly. No discrepancies.
| Metric | Achieved | Reference | Assessment |
|---|---|---|---|
| Documents streamed | 718,785 | ~718,000 | match |
| Documents kept (Phase 1) | 697,869 | ~698,000 | match |
| Near-duplicates | 1,606 | ~1,600 | match |
| SEC exact duplicates | 2,000 | ~2,000 | match |
| Contaminated | 48,814 | ~24,000 | 2× — decontaminates against test and validation |
| Training tokens | 1.85B | ~2.19B | −15%, following from the above |
| Validation share | 1.00% | ~1% | match |
The shortfall is concentrated in case law and is a deliberate trade: decontaminating against both the test and validation splits removes more real training data in exchange for stronger protection against benchmark leakage. The error direction is the safe one.