Laurie Voss

How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

Sep 9, 2026 · 22:26

Laurie Voss, head of developer relations at Arize AI and co-founder of npm, argues the 200-instruction ceiling for skills files collapsed 10x in a year. Rerunning the IFScale benchmark, which asks a model to write a report containing exact words and counts how many appear, he replicated last year's 200-to-300 drop, then watched current models ace it. He had to raise the test from 500 words to 10,000 before anything broke: the boundary now sits near 2,000 instructions, closer to 5,000 for the best. Failures diverged: DeepSeek forgets, Claude refuses when random words trip its safety filter, Gemini overthinks and answers nothing, and GPT-5.5 writes half a report then calls the request stupid. Compression is solved; verification—checking outputs with evals—is now the hard part.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

Effective agent design patterns in production — Laurie Voss, LlamaIndex

Jun 27, 2025 · 15:38

Laurie Voss, VP of Developer Relations at LlamaIndex, presents five agent design patterns—chaining, routing, parallelization, orchestrator workers, and evaluator optimizers—for building production-ready agents. He argues that agents improve RAG by enabling introspection, self-correction, and better accuracy, and that RAG remains essential because it's cheaper and faster than feeding full context. LlamaIndex provides a framework, LlamaParse for parsing complicated documents, and LlamaCloud for managed retrieval endpoints, integrating with 400 models over 80 providers. He demonstrates how workflows in LlamaIndex enable concurrent event emitting to implement parallelization and voting to reduce hallucination, noting that three different LLMs seldom hallucinate the same answer. The evaluator optimizer pattern uses an LLM to check its own output and loop back for improvement. Multi-agent systems can be built with a single line of code by passing an array of function agents.