Jack Morris

Scaling Compute on Context — Jack Morris, Engram

Aug 12, 2026 · 19:42

Jack Morris of Engram frames scaling compute on context as the pursuit of depth in AI, contrasting it with the breadth of public-data pre-training. He argues models trained on public data know nothing about your emails, meetings, or company, and that with a fixed private corpus, compute is the only scalable axis. He critiques naive fine-tuning (loss 0.00001 on 10K financial reports then collapse), KV compaction, on-policy distillation, and synthetic continued pretraining, noting each hits a synthetic data wall. The goal is self-improvement like AlphaGo, where better models generate harder training questions, enabling indefinite compute scaling on your context.

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.