RAG Evaluation

Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai

Aug 12, 2026 · 13:04

Stefania Druga, a research scientist at Sakana AI in Tokyo, presents experiments on memory harnesses for long-running research agents running on local models like Qwen 27B and DeepSeek V4 Flash on an M3 Ultra. She frames memory as a write-manage-read control loop, not a database, and tests a recall ladder—no recall, vector RAG, a ranked decisions ledger, and an oracle—across 68 xbench questions. The ranked ledger performed best, beating even gating on whether memory is needed, while the oracle didn't hit max because giving the right memory doesn't force its use. When tasks fit in context, memory only added cost with no accuracy gain, but for long-horizon tasks where answers sit far outside the window, good recall policy became essential and cheaper. She urges treating recall policy as a first-class metric and highlights the broader memory technique landscape, including over 30 runnable cookbooks from Diamond, while noting local models run serially, which is why her Tokyo machine…

On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

Jul 17, 2026 · 17:35

Pablo Castro, Distinguished Engineer and CVP for AI Knowledge at Microsoft, argues that AI agents require three knowledge types—intrinsic (model training), extrinsic (RAG and grounding), and learned (optimization loops)—and demonstrates how Microsoft's Foundry IQ, Azure AI Search, and Agent Optimizer deliver them. He shows that combining retrieval methods outperforms individual ones, citing Azure AI Search evaluations where combined methods improve evidence recall and answer completeness. He introduces Foundry IQ as a layered system with agentic retrieval that reflects on data to satisfy information needs before returning results. Castro demonstrates creating a knowledge base that connects unstructured data, structured Parquet tables, and the web, then linking it to an agent via MCP server. He also showcases the Agent Optimizer, which uses hill climbing on baseline evaluations to automatically refine agent instructions, tool definitions, and skills based on runtime traces, enabling continuous learning loops that capture organizational differentiation.

How to look at your data — Jeff Huber (Chroma) + Jason Liu (567)

Aug 6, 2025 · 19:23

Jeff Huber (Chroma CEO) and Jason Liu argue that AI practitioners should look at their data—both inputs and outputs—with fast evals and conversation analysis to systematically improve retrieval and product decisions. Huber advocates for fast evals using golden datasets of query-document pairs over public benchmarks or expensive LLM judges, showing how synthetic queries aligned to real user behavior can empirically compare embedding models; in a Weights & Biases chatbot case study, Voyage 3 large outperformed text-embedding-3-small and others. Jason Liu focuses on outputs: extract structured metadata from conversations, cluster them to find segments, then compare KPIs like performance across clusters to prioritize what to fix, build, or ignore—for example, if 40% of conversations involve data visualization and the agent performs poorly, invest in better plotting tools. This population-level analysis enables impact-weighted decisions, turning vague metrics like 0.5 factuality into actionable insights by segment. The episode emphasizes that retrieval improvements are foundational, and once you have users, looking at conversation structure drives a data-driven product roadmap.

Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)

Jul 29, 2025 · 20:22

David Karam of Pi Labs (formerly Google Search) walks through layering every technique in RAG, from in-memory retrieval to planet-scale search with 70+ corpus mix of token, embeddings, and knowledge graphs, jointly retrieved and re-ranked at 160,000 queries per second in under 200msec. He advocates a Quality Engineering Loop: baseline with simplest methods, analyze losses, then apply incremental techniques based on complexity-adjusted impact. Karam explains why queries like "falafel" are notoriously hard due to ambiguous intent, highlights failures of chunking documents, and shows when BM25 suffices, when relevance embeddings are needed, and when custom embeddings or domain-specific signals (price, popularity, user clicks) become essential. He covers query orchestration via fan-out, supplementary retrieval across back ends, and distillation for cost optimization, concluding that at sufficient complexity, problems must be punted to LLM or UX for graceful degradation.

Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer

Apr 22, 2025 · 12:01

Writer CTO Waseem Alshikh presents FailSafe, a benchmark for evaluating LLMs in real-world finance, challenging the notion that general-purpose models suffice. Tests on query failures (misspelling, incomplete, out-of-domain) and context failures (missing context, OCR errors, irrelevant context) reveal that reasoning models like o1 and o3 perform 50% to 60% worse on context grounding than smaller domain-specific models, despite higher answerability. Even the best model achieves only 81% combined robustness and grounding, meaning roughly 20% of responses are wrong. Alshikh argues that domain-specific models remain essential, and that full-stack systems—RAG, guardrails, and grounding—are necessary for reliable deployment in high-stakes finance.

How Codeium Breaks Through the Ceiling for Retrieval: Kevin Hou

Jul 31, 2024 · 18:42

Kevin Hou, AI engineer at Codeium, argues that embedding-based retrieval is hitting a ceiling for AI code generation and that Codeium breaks through by throwing large language models directly at the problem via a system called MQuery. He outlines the limitations of long context, fine-tuning, and embeddings, noting that embeddings are cheap but fail to reason over multiple documents needed for real-world coding tasks. Codeium developed a product-driven benchmark using recall 50 and commit-message datasets to measure retrieval quality. Because Codeium is vertically integrated—training custom models and building its own infrastructure—it can afford to run thousands of LLM calls in parallel per query, giving users 100x more compute than competitors. MQuery retrieves context from thousands of files in seconds, resulting in more accepted completions and better chat satisfaction. Hou compares this shift to autonomous driving: early heuristics like embeddings are giving way to large models that handle more data directly.