Selected workFebruary to December 2023

Superintelligence

A decoder-only transformer I trained from raw text in 2023, and served to 1.2 million people in 30 hours.

I trained the tokenizer, assembled the pretraining corpus, chose the architecture, ran the training, built the evals and stood up the serving stack.

Fox News on Superintelligence, 13 May 2023

Arvin Bhangu on Fox News, the lower third reading Founder, Superintelligence. — Live on Fox News, May 2023, with the extended interview on Fox News Digital.

What it was made of.

These are the settings I ran in 2023.

Architecture

Architecture specification
Field	Value
Family	Dense decoder-only transformerTrained from raw text, not a fine-tune of another checkpoint.
Parameters	37,993,521,15237.58B in the 56 blocks, 412M in one tied embedding matrix.
Layers	56
d_model	8,192
Attention heads	64 query, 1 key, 1 valueMulti-query attention. All 64 query heads read one shared key and one shared value head, which is what keeps the KV cache at 56 MiB for a full 2,048-token request.
Feed-forward ratio	2.65625× (21,760)SwiGLU, so three matrices rather than two. Sized to cost what a conventional 4× two-matrix layer costs.
Positional scheme	RoPE, base 10,000
Normalization	Pre-RMSNorm, ε 1e-6One shared norm per block feeding attention and feed-forward in parallel rather than in series.
Context length	2,048 tokensThe same at training and inference. Nothing was extrapolated past what was trained.

Tokenizer

Tokenizer specification
Field	Value
Algorithm	SentencePiece byte-fallback BPE, trained on the corpus
Vocabulary size	50,30450,040 learned pieces, 256 byte tokens, 8 special. The byte tokens mean there is no such thing as an out-of-vocabulary input.

Data

Data specification
Field	Value
Raw corpus	24TB of extracted text and source codeBefore deduplication and quality filtering, which removed about 87% of it by bytes.
Training tokens	799,958,630,40021.06 tokens per parameter. One pass. No document was repeated on purpose.
Sources	Nine components45% filtered web, 12% code, 10% books, 8% scientific, 7% Q&A and forums, 5% each Wikipedia, government and reference, 3% news. By language: 70% English, 18% other languages, 12% code.
Deduplication	SHA-256 exact, then MinHash-LSH128 permutations over word 5-grams, rejected at Jaccard ≥0.80 for text and ≥0.85 for code. Benchmark decontamination ran separately on unique 13-grams.

Compute

Compute specification
Field	Value
Hardware	512× NVIDIA A100 80GB SXM64 nodes of 8. Tensor parallelism stays inside a node over NVLink; data parallelism crosses nodes.
Funded by	$2.5M in NVIDIA compute creditsThe main run cost $1.83M of it, which is what made the shape decision a real choice rather than a formality.
Total GPU-hours	410,000 on the main runResumable checkpoints every two hours across the 512-GPU cluster.

Training

Training specification
Field	Value
Framework	PyTorch, with Megatron-LM and DeepSpeed ZeRO-1Tensor parallel 8, pipeline 1, data parallel 64. Optimizer state sharded across the 64 replicas.
Precision	BF16 forward and backward, FP32 master weightsBF16 rather than FP16 for the exponent range. Optimizer state and the reductions that matter stayed in FP32.
Optimizer	AdamW, β 0.9/0.95, ε 1e-8, weight decay 0.1No decay on embeddings or normalization scales. Gradients clipped at global norm 1.0.
Peak learning rate	1e-4, decayed to 1e-5
Schedule	2,000-step linear warmup, then cosine decay on tokensToken-based rather than step-based, so doubling the batch part-way through does not bend the curve.
Tokens per batch	2,097,152, then 4,194,304The smaller batch for the first 10,000 steps, the larger for the remaining 185,725. 195,725 optimizer steps in total.

Serving

Serving specification
Field	Value
Optimizations	Multi-query KV caching, dynamic batching, quantization, streaming generation
Quantization	BF16 reference, INT8 weight-only for single-GPU serving76GB of BF16 weights across two A100s, or 38GB in 8-bit on one.
Users in 30 hours	1.2 millionThe 30 hours following the live Fox News segment, 13 May 2023.
Daily active users at shutdown	2.5 millionDecember 2023, when I decided to shut it down.

Four shapes I could have trained.

NVIDIA gave me $2.5 million in compute credits to train it, and that budget was the constraint. These were the four shapes it could buy, and picking one meant giving up the other three.

Four model shapes the budget could buy, and the verdict on each
Strategy	Parameters	Tokens	Tokens/param	Compute cost	Verdict
Small, heavily trained	24B	1.0T	41.7	$1.44M	Cheapest to serve and the data goes furthest. Too little room to hold what it reads.
Balancedbuilt	37.99B	800B	21.1	$1.83M	Closest to compute-optimal, and still fits on two A100s to serve.
Large, undertrained	65B	450B	6.9	$1.76M	The impressive number on a slide. Starved of data, and slower and dearer to serve for it.
Sparse mixture-of-experts	120B total, 24B active	1.0T	41.7 per active	$1.54M	Most capacity per FLOP and the most ways to fail: routing collapse, load imbalance, all-to-all traffic, and no mature GPU implementation at this scale in 2023.

1.2 million people, in a day and a bit.

A model I trained alone stayed up underneath that traffic, on capacity I was still adding while the segment aired. It kept climbing after that. By December 2023, 2.5 million people were using it every day.

The decisions I still remember.

Why not the bigger model

Every version of this that people wanted to hear about was larger. A 65B would have cost roughly the same compute as the 38B, because compute is parameters multiplied by tokens and you can trade one against the other freely on a slide.

You cannot trade them freely in the model. Chinchilla had shown ten months earlier that the big models of that year were badly undertrained, and that a smaller one fed more data beat them. The 65B would have seen 6.9 tokens for every parameter it had to store them in. The 38B I built saw 21.

Tokens per parameter

21.06 tokens per parameter, against 6.9 for the 65B alternative at the same compute

One key/value head instead of sixty-four

Standard attention keeps a separate key and value head for every query head, and every one of them has to be cached for every token in the conversation. That cache is what limits how many people can be served at once, and it grows as the conversation does.

Multi-query attention shares one key head and one value head across all 64 query heads. I made that call months before launch, and it did nothing for me at all until the day the traffic arrived.

KV cache per full-context request

56 MiB of KV cache for a full 2,048-token request, against 3.5 GiB for the same model with 64 KV heads

Why I trained the tokenizer instead of borrowing GPT-2's

Borrowing a tokenizer was the default in 2023, and for most projects it was the right call. It was also the first place I would have inherited somebody else's assumptions. A vocabulary fitted to one corpus segments a different corpus badly, and every extra token it spends is context I had paid for and could not spend on reasoning.

So I trained the tokenizer on the corpus the model would actually see. That took a week. What it bought was fertility, fewer tokens per word on held-out text, so the same 2,048 positions carried more of the document.

The refusal problem in post-training

A base model has no refusal behaviour. Refusals arrive during instruction tuning, and the hard part was getting instruction-following without picking them up, because the data that teaches a model to follow directions is the same data that teaches it to decline.

It still had to follow instructions, so I had to carry direct answering through post-training and then go back and check whether it had survived.

Reading a loss curve with nothing to compare it to

The recipes were unwritten in 2023. I chose the data mixture, chose the learning-rate schedule, and read the loss curve with no shared sense of what healthy looked like at that scale, because nobody had published one.

The mistakes that hurt were the ones that only showed up after the curve had already looked fine for days. Finding them cost GPU-hours, and there was no way to get those back.

What I still have.

architecture:
  family: dense_decoder_only_transformer
  total_parameters: 37993521152
  layers: 56
  d_model: 8192
  attention:
    type: full_causal_multi_query
    query_heads: 64
    kv_heads: 1
    head_dimension: 128
  ffn:
    type: swiglu
    dimension: 21760
  block:
    layout: parallel_residual
    normalization: pre_rmsnorm
  positional_encoding: { type: rope, theta: 10000 }
  context: { training_tokens: 2048 }
  biases: false
  weight_tying: input_output

training:
  parallelism: { tensor: 8, pipeline: 1, data: 64 }
  precision: { forward_backward: bfloat16, master_weights: float32 }
  optimizer: { type: adamw, beta1: 0.9, beta2: 0.95, weight_decay: 0.1 }
  learning_rate: { peak: 1.0e-4, final: 1.0e-5, warmup_steps: 2000 }
  gradient_clipping: 1.0
  batch_schedule:
    - { steps: 10000,  global_tokens: 2097152 }
    - { steps: 185725, global_tokens: 4194304 }
  total_steps: 195725
  total_tokens: 799958630400
  tokens_per_parameter: 21.0551
  epochs: 1

How it ended.

It shipped as a consumer app in February 2023, I went on Fox News live that May, and it drew 1.2 million users in the 30 hours after. By December 2023 it had 2.5 million daily active users. It had also drawn the attention of people working on AI safety and alignment, and I took the conversations. My view flipped. I shut it down at its peak, because I no longer believed in what I was scaling.

What I kept was the way of working. I still build the eval before the thing it is supposed to measure.