NanoGPT

First Steps Toward Automated AI Research — Richard Socher, CEO Recursive AI

Jul 30, 2026 · 20:24

Richard Socher, CEO of Recursive AI, presents his vision of a "Eureka machine" that automates scientific discovery through recursive self-improvement, arguing that automating research can compress centuries of progress into decades. He frames science as an evolutionary process driven by Popperian falsification, and proposes a four-pillar system covering existing knowledge, measurement, simulation, and physical experimentation. Socher shows early proof points from his lab: a NanoChat model improved from 0.93 to 0.91 bits per byte, a NanoGPT speedrun cut by over two seconds to 70 seconds, and CUDA kernels that beat NVIDIA's benchmark leaderboard across all categories. He emphasizes that while these are early wins, the direction points toward fully autonomous AI research that could ultimately tackle problems in medicine, economics, and astrophysics.

How Autoresearch is changing ML research — Zhengyao Jiang, Weco

Jul 16, 2026 · 16:16

Zhengyao Jiang, CEO of Weco AI, details how their autonomous agent Aiden became the top contributor in OpenAI's Parameter Golf competition, setting 7 leaderboard records—more than twice any human's 3. Over 22 days, Aiden ran 1,300 experiments on a single H100 node, using just 4% of total compute while contributing 15% of records and achieving a 28% leaderboard hit rate (six times the community average). The agent excelled at finding and implementing ideas from human researchers, combining them (e.g., gated attention from a paper with a tokenizer improvement) to produce breakthrough results. Jiang argues that auto-research does not replace human creativity but shifts it upward: humans design evals and code abstractions that bias the agent's search, a new craft he compares to training neural networks. The talk reveals that Aiden's H-index in the community was 10 versus the next human's 7, proving autonomous agents can produce work that others build on.

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.