Pretrained from random weights on US case law, SEC filings and educational web text.
This is a base text completer, not an instruction-tuned chat assistant — it continues whatever
you give it, plausibly but without following instructions or holding a conversation.
Read before you type: this checkpoint is the best-validation-perplexity save from a run that
stopped at its GPU spend cap 58% through a planned 5-epoch schedule (step 10,253 of 17,645). It has no
instruction tuning, no factual grounding, and can fabricate plausible-sounding case names, statutes
and figures. Nothing it generates is legal or financial advice. The backend sleeps when idle, so the
first reply after a quiet period can take up to ~15 seconds.
temp 0.8length 80
Pick an example above, or write your own legal/financial opening — the model continues from there.
Training run — full 5-epoch pretrain, stopped at its spend cap
Optimizer steps10,253 / 17,645 · 58%
Tokens seen5.38B / 9.25B target
Best validation perplexity
8.92
Final train loss
2.17
Steps completed
10,253 / 17,645
Tokens seen
5.38B / 9.25B target
Training loss
every 50 steps
Validation perplexity
every 1,000 steps
Architecture & recipe
Architecture
Llama (SwiGLU, RoPE, RMSNorm)
Parameters
125,847,552
Layers
12
Hidden size
768
Attention / KV heads
12 / 12
Intermediate size
3,072
Context length
1,024
Vocabulary
16,384
Optimizer
AdamW (0.9, 0.95)
Peak / floor LR
6e-4 / 6e-5
Warmup
200M tokens
Precision
bf16 autocast
Global batch
524,288 tokens/step
Hardware
8× H100, single-node DDP
Schedule
5 epochs (planned)
Training data
A 16,384-token byte-level BPE vocabulary and this model were both
trained from scratch on the same ~1.85B-token packed corpus — cleaned, deduplicated (MinHash + exact) and
decontaminated against the CaseHOLD and LexGLUE evaluation splits before packing. See the
full data-pipeline report for Phases 0-4.
case law 31%SEC 44%web 25%
Source
Share of train tokens
US case law (HFforLegal/case-law)
31.1%
SEC filings (PleIAs/SEC)
43.9%
Educational web text (fineweb-edu)
25.0%
Reference completions
Sampled at this checkpoint with temperature 0.8 — illustrative,
not curated.
"The court held that" → …the defendant was not prejudiced by the admission of
the testimony relating to the statement that he was not under arrest. In so holding, the court stated:
We agree. In State v. Hensley…
"Net revenues for the fiscal year increased" → …$2.5 million, or 7.1%, to
$31.9 million in fiscal 1997 from $31.9 million in fiscal 1996. The increase in total revenues was
attributable to the Company…