Selected workFebruary to December 2023

Superintelligence

A decoder-only transformer I trained from raw text in 2023, and served to 1.2 million people in its first 30 hours.

The tokenizer, the pretraining corpus, the architecture, the training runs, the evals and the serving stack were all built here.

Fox News, May 2023 What I build now

Arvin Bhangu on Fox News, the lower third reading Founder, Superintelligence. — Live on Fox News, May 2023, with the extended interview on Fox News Digital.

The build sheet.

The shape, the corpus, the cluster and the schedule, as they were set.

Architecture

Architecture specification
Field	Value
Family	Dense decoder-only transformerTrained from raw text, not a fine-tune of another checkpoint.
Parameters	37,993,521,15237.58B in the 56 blocks, 412M in one tied embedding matrix.
Layers	56
d_model	8,192
Attention heads	64 query, 1 key, 1 valueMulti-query attention. All 64 query heads read one shared key and one shared value head, which is what keeps the KV cache at 56 MiB for a full 2,048-token request.
Feed-forward ratio	2.65625× (21,760)SwiGLU, so three matrices rather than two. Sized to cost what a conventional 4× two-matrix layer costs.
Positional scheme	RoPE, base 10,000
Normalisation	Pre-RMSNorm, ε 1e-6One shared norm per block feeding attention and feed-forward in parallel rather than in series.
Context length	2,048 tokensThe same at training and inference. Nothing was extrapolated past what was trained.

Tokenizer

Tokenizer specification
Field	Value
Algorithm	SentencePiece byte-fallback BPE, trained on the corpusNot borrowed. Vocabulary matched to the training data.
Vocabulary size	50,30450,040 learned pieces, 256 byte tokens, 8 special. The byte tokens mean there is no such thing as an out-of-vocabulary input.

Data

Data specification
Field	Value
Raw corpus	24TB of extracted text and source codeBefore deduplication and quality filtering, which removed about 87% of it by bytes.
Training tokens	799,958,630,40021.06 tokens per parameter. One pass. No document was repeated on purpose.
Sources	Nine components45% filtered web, 12% code, 10% books, 8% scientific, 7% Q&A and forums, 5% each Wikipedia, government and reference, 3% news. By language: 70% English, 18% other languages, 12% code.
Deduplication	SHA-256 exact, then MinHash-LSH128 permutations over word 5-grams, rejected at Jaccard ≥0.80 for text and ≥0.85 for code. Benchmark decontamination ran separately on unique 13-grams.

Compute

Compute specification
Field	Value
Hardware	512× NVIDIA A100 80GB SXM64 nodes of 8. Tensor parallelism stays inside a node over NVLink; data parallelism crosses nodes.
Total GPU-hours	410,000 on the main runResumable checkpoints every two hours across the 512-GPU cluster.

Training

Training specification
Field	Value
Framework	PyTorch, with Megatron-LM and DeepSpeed ZeRO-1Tensor parallel 8, pipeline 1, data parallel 64. Optimiser state sharded across the 64 replicas.
Precision	BF16 forward and backward, FP32 master weightsBF16 rather than FP16 for the exponent range. Optimiser state and the reductions that matter stayed in FP32.
Optimiser	AdamW, β 0.9/0.95, ε 1e-8, weight decay 0.1No decay on embeddings or normalisation scales. Gradients clipped at global norm 1.0.
Peak learning rate	1e-4, decayed to 1e-5
Schedule	2,000-step linear warmup, then cosine decay on tokensToken-based rather than step-based, so doubling the batch part-way through does not bend the curve.
Tokens per batch	2,097,152, then 4,194,304The smaller batch for the first 10,000 steps, the larger for the remaining 185,725. 195,725 optimiser steps in total.

Serving

Serving specification
Field	Value
Optimisations	Multi-query KV caching, dynamic batching, quantization, streaming generation
Quantization	BF16 reference, INT8 weight-only for single-GPU serving76GB of BF16 weights across two A100s, or 38GB in 8-bit on one.
Users in first 30 hours	1.2 millionFollowing the Fox News feature, 13 May 2023.
Daily active users at shutdown	2.5 millionDecember 2023, when I decided to shut it down.

Four shapes, one budget.

The compute was fixed, so the question was never how much to spend. It was what shape to spend it on. All four of these cost about the same.

Candidate model shapes at a fixed compute budget, and why three were rejected
Strategy	Parameters	Tokens	Tokens/param	Compute cost	Verdict
Small, heavily trained	24B	1.0T	41.7	$1.44M	Cheapest to serve and the data goes furthest. Too little room to hold what it reads.
Balanced / built	37.99B	800B	21.1	$1.83M	Closest to compute-optimal, and still fits on two A100s to serve.
Large, undertrained	65B	450B	6.9	$1.76M	The impressive number on a slide. Starved of data, and slower and dearer to serve for it.
Sparse mixture-of-experts	120B total, 24B active	1.0T	41.7 per active	$1.54M	Most capacity per FLOP and the most ways to fail: routing collapse, load imbalance, all-to-all traffic, and no mature GPU implementation at this scale in 2023.

Chinchilla had come out ten months earlier and shown that the large models of the day were badly undertrained for their size. The 65B is the one people wanted to hear about. It gets 6.9 tokens for every parameter it has to store them in.

1.2 million people, in a day and a bit.

A model I trained alone stayed up underneath that traffic, on capacity I was still adding while the segment aired. Usage kept climbing from there: by the end, 2.5 million people were using it every day.

The decisions worth the space.

Why not the bigger model

Every version of this that people wanted to hear about was larger. A 65B would have cost roughly the same compute as the 38B, because compute is parameters multiplied by tokens and you can trade one against the other freely on a slide.

You cannot trade them freely in the model. Chinchilla had shown ten months earlier that the large models of the day were badly undertrained, and that a smaller one fed more data beat them. The 65B option would have seen 6.9 tokens for every parameter it had to store them in. The 38B saw 21.

Tokens per parameter

21.06 tokens per parameter, against 6.9 for the 65B alternative at the same compute

One key/value head instead of sixty-four

Standard attention keeps a separate key and value head for every query head, and every one of them has to be cached for every token in the conversation. That cache is what limits how many people can be served at once, and it grows as the conversation does.

Multi-query attention shares one key head and one value head across all 64 query heads. It is a decision made months before launch that only pays off on the day the traffic arrives, which is the kind that is easy to skip.

KV cache per full-context request

56 MiB of KV cache for a full 2,048-token request, against roughly 3.5 GiB for the same model with 64 KV heads

Why I trained the tokenizer instead of borrowing GPT-2's

Borrowing a tokenizer is the default, and for most projects it is correct. It is also the first place a model quietly inherits somebody else's assumptions: a vocabulary fitted to one corpus segments a different corpus badly, and every extra token is context you paid for and cannot spend on reasoning.

So the tokenizer was trained on the corpus the model would actually see. The test of whether that was worth doing is fertility, meaning tokens per word on held-out text, because a lower number is directly more effective context at the same sequence length.

The refusal problem in post-training

A base model has no refusal behaviour; refusals are installed during instruction tuning. The work was getting instruction-following without acquiring them, which is delicate because the data that teaches a model to follow directions is the same data that teaches it to decline.

The model still had to follow instructions, so the constraint was to get instruction-following while keeping direct answering intact, and then to measure whether it held rather than assert that it did.

Reading a loss curve with nothing to compare it to

The recipes were unwritten in 2023. You chose the data mixture, chose the learning-rate schedule, and read your loss curve without any shared sense of what healthy looked like at that scale, because nobody had published one.

Which meant the errors were the kind that only surface after the loss curve has already looked fine for a long time, and the cost of finding them was measured in GPU-hours, not minutes.

Receipts.

Dates with texture beat adjectives.

architecture:
  family: dense_decoder_only_transformer
  total_parameters: 37993521152
  layers: 56
  d_model: 8192
  attention:
    type: full_causal_multi_query
    query_heads: 64
    kv_heads: 1
    head_dimension: 128
  ffn:
    type: swiglu
    dimension: 21760
  block:
    layout: parallel_residual
    normalization: pre_rmsnorm
  positional_encoding: { type: rope, theta: 10000 }
  context: { training_tokens: 2048 }
  biases: false
  weight_tying: input_output

training:
  parallelism: { tensor: 8, pipeline: 1, data: 64 }
  precision: { forward_backward: bfloat16, master_weights: float32 }
  optimizer: { type: adamw, beta1: 0.9, beta2: 0.95, weight_decay: 0.1 }
  learning_rate: { peak: 1.0e-4, final: 1.0e-5, warmup_steps: 2000 }
  gradient_clipping: 1.0
  batch_schedule:
    - { steps: 10000,  global_tokens: 2097152 }
    - { steps: 185725, global_tokens: 4194304 }
  total_steps: 195725
  total_tokens: 799958630400
  tokens_per_parameter: 21.0551
  epochs: 1

How it ended.

It shipped as a consumer app in February 2023, Fox News covered it that May, and it drew 1.2 million users in the 30 hours after. By December 2023 it had 2.5 million daily active users. It had also drawn the attention of people working on AI safety and alignment, and I took the conversations. My view flipped. I shut it down at its peak, because I no longer believed in what I was scaling.

The stack is the part that carried forward. Training pipelines, eval discipline, and the reflex to measure a system rather than describe it are the same instincts Roam runs on.

What I build now Timeline