RAG & Retrieval

Your agents lack context: Here's how to fix "You're absolutely right!" — Brandon Waselnuk, Unblocked

Sep 9, 2026 · 14:09

Brandon Waselnuk of Unblocked argues that AI-generated code should feel like it was written by someone who has been on your team for years, and that the gap is no longer intelligence but context. He traces how bad context compounds as teams move from tab completion to parallel and background agents, producing correction doom loops, wasted search tokens, and a review tax. Two common fixes stall: the curated context trap, where markdown files rot and someone must curate them for everyone, and the MCP plateau, where an agent may never call the server or stops at the first plausible answer due to satisfaction of search bias, missing last night's Slack correction. A context engine must resolve conflicts between an old architecture diagram and a fresh CTO Slack thread, personalize relevance, enforce permissions, and deliver token-optimized context. Running the same prompt with and without context cut tokens from 21 million to 10.8 million and finished about two hours sooner. He closes by demoing three open source tools, including a workshop that builds a relational context engine from scratch.

It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners

Sep 9, 2026 · 20:48

Kevin Madura, director of advanced technology at AlixPartners, argues that recursive language models (RLMs) mark a fundamentally different way for LLMs to handle context: instead of attending to every token, the model treats its context as a variable in a Python REPL and delegates subtasks to sub-LMs, including itself. He traces RLMs to early work by Omar Khattab and Alex Zhang, citing the Oolong and BrowseComp benchmarks, where an RLM beats GPT-5 tool calling at lower cost, and the long chain of thought benchmark, where accuracy jumps from 2.6 to 45.4 percent. Unlike RAG, which stuffs the context window, or agents that pass JSON strings back and forth, an RLM keeps logic, execution, and results in one environment, avoiding context rot. Madura demonstrates a cohort retention analysis on three data frames where the model reasons in its own REPL and decides when to submit a typed answer. Case studies include Trampoline AI consolidating long invoices without chunking or embeddings, an AWS engineer surfacing patterns in raw logs, Halo optimizing an agent harness from its own traces, and a security report generated across 500,000 lines of code. His closing bet: models post-trained to…

Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Sep 4, 2026 · 20:48

MiniMax's M3 pairs a functional one-million-token context window with coding and multimodal abilities in a 400-billion-parameter, 20-billion-activated model, explain Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder of Hugging Face. Song says short context fails when agents handle multi-round tool responses; MiniMax Sparse Attention — an index branch selecting what matters plus a sparse branch computing on selected blocks — was designed by an intern. She argues multimodal training from the very first step beats post-hoc adapters that harm text performance and risk collapse; interleaved data and reward modeling solved this. Its apps reach over 300 million people in 200 countries; anyone can propose projects, and community feedback and internal agent harnesses now drive M3.1.

Agents' next frontier: agent-to-agent and network effects — Jean-Denis Greze, Town

Sep 3, 2026 · 21:17

Jean-Denis Greze, CTO of Town and former Plaid CTO, argues agent-to-agent is not a useful concept: every LLM system is a search problem, and what matters is whether the right information sits in the context window at the moment of a tool call. The ideal is a single agent with access to all the world's information; privacy, not context length, blocks it — his Coase framing: privacy is the transaction cost. He grades five strategies by how closely each approximates that impossible agent: a shared trust boundary like an HR agent scoped to the most junior access, custom tools that read everyone's mail but return only a connection score, shared silos fed by a sweeper agent, humans as the conduit, and a black box that searches every silo unasked, seeking approval only from the information's owner.

Agentic Sites: Building Hyper Personalized Websites — Carlos Sanchez, Adobe

Aug 29, 2026 · 20:43

Carlos Sanchez, principal scientist at Adobe, argues that agentic sites can deliver hyper-personalized web pages in real time by treating the whole site as a corpus and generating only specific blocks. He demonstrates a coffee-machine site that builds a personalized camping page in 1.64 seconds using Google's Gemma 4 on Cerebras at 2,300 tokens per second. Model choice must be evaluated per site for accuracy and speed, he says, showing 1.1 seconds against 4.6 for the runner-up, and this doesn't need a frontier model because work is choosing and arranging blocks. Marketers define personas in natural language, browsing signals feed an 'audience of one' loop for pre-generated recommendations, and a tool builds an agentic site for any URL in less than an hour.

The Half Life of Agent Infrastructure — Ben Kus, Box

Aug 29, 2026 · 19:26

Ben Kus, CTO of Box, explains why AI agent infrastructure has a half-life of months, not the three-to-five years typical of enterprise software. He retracts his own graph-based agent approach from last year's talk, and recounts asking an engineer to rebuild working agentic search twice in quick succession. At Box's scale — over an exabyte of data and around a trillion tokens — models, agent design, and retrieval have each shifted repeatedly. His advice: prepare teams to expect change, build abstractions that let underlying layers be swapped, and switch only when eval sets show measurable gains in cost, speed, quality, or capability. Box now reviews every AI technology on a six-month clock, and he suggests judging vendors by how well they handled past changes.

How to Generate Mergeable Code with a Context Engine — Peter Werry, Unblocked

Aug 27, 2026 · 18:36

Peter Werry of Unblocked argues that access to information does not equal understanding—organizational context is the bottleneck for AI agents. He likens agents to expert engineers resetting knowledge every task, falling prey to radiology's 'satisfaction of search'; million-token windows distract. Werry demos the context engine generating an architecture diagram, then runs the same optimization plan in Claude Code twice: with it, under a dollar and about a minute; without, roughly double the time and more cost, because later steps loop on wrong assumptions. He also shows a review agent boosting comments by seniority, a drop in flagged issues traced to a Slack thread, and open-source tools: a GitHub-history query engine and a social graph showing thin review coverage.

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

Aug 27, 2026 · 21:48

Yuchen Fama and Ashish Kamra of Red Hat argue that public inference benchmarks hide the chaotic reality of agentic workloads, and show how KV cache-aware routing plus prefill/decode disaggregation in the open source llm-d framework tackles it. Red Hat's traces show agentic sessions running from a few turns to 3,000, cache hit rates over 90%, and input-output token ratios past 100:1, making a 10x cost gap between cached and uncached tokens. A live demo shows routing reusing cache on the same pod cutting time from 3 seconds to 1, while a fresh system prompt pays the full 3 again. For prefill/decode disaggregation, they report P99 inter-token latency dropping from roughly 900 milliseconds to about 100 across 16 H100s serving gpt-oss, but note it only wins in the middle concurrency band and requires RDMA or RoCE to move KV caches. Their closing case study runs GLM 5.2 on H200s with three prefill workers to one decode, achieving 4x faster time to first token and 60% more requests.

GTM Engineering: The Technical Bits — Everett Berry, Clay

Aug 26, 2026 · 19:04

Everett Berry of Clay explains GTM engineering, the discipline of letting go-to-market teams ship as fast as engineering teams, by covering its hardest technical areas. Data first: accounts never hold still, so teams waterfall across hundreds of vendors—Forager alone yields half of phone numbers—and must run evals to trust providers. Orchestration is the second problem, a data engineering puzzle across 20–30 tools that sync behind your back; Clay's fix is a graph of general purpose nodes. Third are long-running agents, one per account, dormant until triggered, holding state across a deal cycle; human-agent interface is the hardest problem. Execution is the fourth: with cold email reply rates at 0.5–1%, reps spread domains, route replies home, and suppress channels when meetings book.

Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

Aug 21, 2026 · 13:22

Jeff Ng, founding engineer at Unblocked, argues building agents is trivial but missing context makes them confidently wrong. Cloud primitives and frameworks have removed plumbing that six months ago took a team a quarter. In a demo, an agent enriching a Linear ticket recommended re-enabling async dispatch, missing that a support engineer had disabled it after an outage; it lacked the Slack thread and postmortem. Background agents fail silently when no human supplies that context. Ng's context engine connects docs, code, tickets, and conversations, reconciles conflicts, and returns a synthesized understanding scoped to permissions. MCP provides access, not understanding; rerunning the same agent grounded in the engine flips its recommendation from repeating the outage to preventing it.

The Last Human Code Review: Building Trust in AI-Generated Code — Itamar Friedman, Qodo

Aug 20, 2026 · 18:54

Itamar Friedman, CEO and co-founder of Qodo, argues the bottleneck in AI-generated code is now context, not models; teams shipping code faster than humans can review are already inside the problem. He splits leaders into two camps — trust every line vs. ship bugs and fix fast — and says code-review benchmarks barely moved, proving context, not reasoning, matters. Qodo's answer is a context engine that codifies tribal knowledge — instruction files, Slack threads, and senior developers' heads — into interfaces humans can audit and agents can consume, shifting review from a single PR to a graph of PRs and contracts. Qodo targets 2027 with zero critical bugs in production, moving from artificial intelligence to artificial wisdom.

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Aug 17, 2026 · 1:03:26

Louis-François Bouchard, Omar Solano, and Samridhi Vaid of Towards AI test context engineering on their AI tutor: keeping full history beat every compaction technique on recall, cost, and latency because 97% of tokens were served from cache (up to 50x cheaper), making summarization a trap unless it shrinks context by more than 50x. Full history recovered specific details 95% vs 32% after summarizing, and distinctive facts survived 800k tokens. Local hardware changes it: a 32k window can't keep everything, and larger models don't widen context. Dense retrieval hit 0% recall at 400k tokens where BM25 got 100%, so they use hybrid retrieval. Rule: name the constraint before compacting.

The Rise of CaaS: Context-as-a-Service for Agentic AI — Omer Primor, Bright Data

Aug 14, 2026 · 22:20

Omer Primor of Bright Data argues that rented AI search and context-as-a-service (CaaS) lose to a self-built pipeline once query volume passes a tipping point. The web is context, not data, and it decays: social content goes stale within a day, news/finance/retail within 30 days, so context is never a one-time snapshot. His test enriching 100 sponsor companies across 25 fields found general search beat dedicated CaaS vendors on coverage, because CaaS only answers from data it already holds; costs were similar, but frequency is the real killer — every repeated query costs the same when nothing changed. He built scrapers for LinkedIn, jobs, and Crunchbase in a day, priced setup at $5,000, and put the crossover just over 15,000 entities. Owned context compounds while rented decays.

Scaling Compute on Context — Jack Morris, Engram

Aug 12, 2026 · 19:42

Jack Morris of Engram frames scaling compute on context as the pursuit of depth in AI, contrasting it with the breadth of public-data pre-training. He argues models trained on public data know nothing about your emails, meetings, or company, and that with a fixed private corpus, compute is the only scalable axis. He critiques naive fine-tuning (loss 0.00001 on 10K financial reports then collapse), KV compaction, on-policy distillation, and synthetic continued pretraining, noting each hits a synthetic data wall. The goal is self-improvement like AlphaGo, where better models generate harder training questions, enabling indefinite compute scaling on your context.

Memory Harnesses for Long-Running Research Agents — Stefania Druga, Sakana.ai

Aug 12, 2026 · 13:04

Stefania Druga, a research scientist at Sakana AI in Tokyo, presents experiments on memory harnesses for long-running research agents running on local models like Qwen 27B and DeepSeek V4 Flash on an M3 Ultra. She frames memory as a write-manage-read control loop, not a database, and tests a recall ladder—no recall, vector RAG, a ranked decisions ledger, and an oracle—across 68 xbench questions. The ranked ledger performed best, beating even gating on whether memory is needed, while the oracle didn't hit max because giving the right memory doesn't force its use. When tasks fit in context, memory only added cost with no accuracy gain, but for long-horizon tasks where answers sit far outside the window, good recall policy became essential and cheaper. She urges treating recall policy as a first-class metric and highlights the broader memory technique landscape, including over 30 runnable cookbooks from Diamond, while noting local models run serially, which is why her Tokyo machine…

Building Turbopuffer: Gergely Orosz (@pragmaticengineer ) × Simon Eskildsen (CEO)

Aug 3, 2026 · 56:30

Gergely Orosz talks with Turbopuffer CEO Simon Eskildsen about how napkin-math engineering took him from high-school Shopify hire to building the S3-based vector database that cut Cursor's bill by 95%. Eskildsen recounts scaling Shopify's infra to 1M RPS, building failure-injection proxy ToxyProxy, and launching Turbopuffer at $1 per million vectors on S3 with an Nginx cache. Cursor became the first customer after he helped debug Postgres/autovacuum issues; he also tells how Jensen Huang ribbed him for choosing CPUs over GPUs. He argues RL workloads are gobbling CPU capacity, making CPU SKUs scarce, and lists six reasons to raise capital, noting his first raise funded R&D and the second let employees cash out. He closes on Turbopuffer's remote culture of campfires and turbo credits for business-class flights.

AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j

Jul 23, 2026 · 1:59:10

Zach Blumenfeld of Neo4j argues that AI agents need context in shapes rather than queries, building three reusable graph shapes on lakehouse data to solve agent hallucinations and missed connections. The shapes include a connection semantic layer on top of BigQuery (or Databricks/Snowflake) that helps agents navigate join paths across hundreds of tables, a deterministic table-of-contents tree that lets agents traverse document folders and links without vector search, and a Leiden community-detection theme shape that surfaces unknown patterns and documentation gaps. Blumenfeld demonstrates with an auto-repair chain scenario, showing how these shapes enable an agent to answer specific repair questions and estate-level questions like what documentation is missing or what failure patterns exist, by treating context as navigable structure rather than a single query.

Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI

Jul 23, 2026 · 20:54

Daniel Chalef argues that provenance for knowledge graphs built by LLMs must itself be a graph, not a simple source ID, because LLM synthesis destroys the paper trail. In Graphiti, the open-source temporal graph framework behind Zep, sources become episodes and derived facts link back to them, enabling a graph walk to trace any fact. This handles merges (merged entities keep all source links), invalidation (invalid-at dates on mutated edges), and metadata projection (tags on episodes inherit to derived facts). Deletion follows the same edges: a fact survives GDPR erasure only if other supporting episodes remain. Benefits include compliance, veracity evaluation, and debuggability for agentic systems.

From Systems of Record to Systems of Context — Omri Bruchim & Tomer Ast, monday.com

Jul 22, 2026 · 15:58

Omri Bruchim and Tomer from monday.com argue that AI assistants fail to understand users because the bottleneck is understanding, not retrieval—so they are building a 'Monday world model' that precomputes context before the user asks. The system uses two engines: a slow engine that mines weeks of activity into a durable user profile (knows you), and a fast engine that reads recent signals for urgent items (knows your day). This split mirrors neuroscience’s hippocampus-neocortex and data architecture’s lambda architecture. The context is served to their Sidekick assistant, which degrades gracefully by falling back to verified context and compounds as every new day sharpens the profile. The result: Sidekick can answer 'what should I focus on right now' with understanding, not just a list of disconnected bullets.

Your Moat Is Your Data Model — Mike Phipps, Gates Foundation

Jul 22, 2026 · 20:30

Mike Phipps of the Gates Foundation argues that as AI commoditizes frontends and agent frameworks, the durable moat is your data model and the tacit knowledge of how your questions are answered. At the foundation, he and his team modeled 25 years of grantmaking—$7 billion a year across 2,000 grants and 4,000 people—into a single Neo4j knowledge graph served to Claude through one MCP server. The graph is built for agents, not dashboards: hierarchies become traversable paths, and unstructured documents are chunked, tagged, and mapped to structured entities at ingestion. Phipps details the curation pipeline, engaging data owners to capture reporting conventions and safeguard constraints, and explains how the graph connects siloed systems (funding, management, org charts) with unstructured meeting documents. Retrieval evals with LLM-as-judge measure pass-at-one and stability, surfacing gaps that feed back into the data model. The talk makes the case that a small team's efforts compound in the data layer, not the layers above it, offering a practical architecture for enterprise agentic retrieval.

CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens — Stephen Chin, Neo4j

Jul 22, 2026 · 20:42

Stephen Chin of Neo4j introduces CrabRAG, a graph-based memory system that outperforms vector databases for AI agent reasoning. He demonstrates that markdown-based memory wastes over 100,000 tokens per round and that vector similarity fails at multi-hop questions. Using a home lab digital twin, he shows a graph agent correctly identifies his daughter's Minecraft server running outdated OS and exposed management ports, while the vector agent returns vague answers. Chin explains that graphs store relationships and enable precise, explainable, and auditable results, and that Claude can write Cypher queries for graph traversal. He announces his book 'GraphRAG: The Definitive Guide' and free training at Neo4j's Graph Academy.

Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla

Jul 20, 2026 · 12:08

Ishita Daga, a senior machine learning engineer at Tesla, argues that enterprise agents fail because of three structural problems — ambiguity, staleness, and preference — rather than needing bigger models or more RAG. For ambiguity, she proposes a hierarchy of sources of truth: a curated semantic layer (best for known KPIs), canonical tables (parametric queries for flexibility), and a database graph (full schema but hard to maintain). To solve staleness, she recommends a context lifecycle embedding live data sources (GitHub, CRM, semantic layers) and a feedback loop that logs events, evaluates agent performance, and updates context automatically. On preference, she notes that different teams calculate the same metric differently (e.g., average milestone time by start vs. completion) and that current solutions like semantic layers or agent memory still fail to capture user-level routing, calling this an open problem requiring further research.

A Practitioner's Guide to Graphs - Tim Ainge, Good Collective

Jul 18, 2026 · 14:18

Tim Ainge from Good Collective presents a practitioner's guide to graphs, covering extraction from unstructured text, schema-first design, and graph-native algorithms to make AI applications smarter, cheaper, and more reliable. He demonstrates that giving extractors a schema (e.g., recipe with ingredients and steps) yields more meaningful graphs, and using ontology instructions standardizes units and ingredient names. Embedding models solve the potato–potato problem by flexibly matching duplicate nodes. Personalized PageRank, inspired by Pinterest's Pixie paper and HIPORAG, finds authoritative nodes in dense graphs, e.g., identifying Miranda v. Arizona as a landmark case not directly cited. Shortest path algorithms reduce tool calls by 40% in code search by retrieving intermediate nodes missed by vector search. Subgraph matching detects software design patterns like the decorator by querying graph shape without specific node details.

On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

Jul 17, 2026 · 17:35

Pablo Castro, Distinguished Engineer and CVP for AI Knowledge at Microsoft, argues that AI agents require three knowledge types—intrinsic (model training), extrinsic (RAG and grounding), and learned (optimization loops)—and demonstrates how Microsoft's Foundry IQ, Azure AI Search, and Agent Optimizer deliver them. He shows that combining retrieval methods outperforms individual ones, citing Azure AI Search evaluations where combined methods improve evidence recall and answer completeness. He introduces Foundry IQ as a layered system with agentic retrieval that reflects on data to satisfy information needs before returning results. Castro demonstrates creating a knowledge base that connects unstructured data, structured Parquet tables, and the web, then linking it to an agent via MCP server. He also showcases the Agent Optimizer, which uses hill climbing on baseline evaluations to automatically refine agent instructions, tool definitions, and skills based on runtime traces, enabling continuous learning loops that capture organizational differentiation.

RLM: Recursive Language Models for Large Codebases - Shashi, Superagentic AI

Jul 12, 2026 · 17:27

Shashi from Superagentic AI explains how Recursive Language Models (RLM), a pattern from MIT, solve the context window problem in large codebases by externalizing context management into a programmable REPL where the model writes code to inspect, slice, and compute relevant chunks, and recursively delegates sub-questions via llm_query. The RLM loop loads the repo as data, produces bounded observations, and can recursively call another model for deeper insights. Shashi demonstrates RLM Code, an open-source independent implementation running locally and on Gemini with a Docker sandbox, showing two tool calls and a complete trajectory. He notes that similar RLM concepts are used in proprietary systems like Codex, Claude managed agents, and dynamic workflows, making it a practical pattern for AI engineers dealing with monorepos.

Semantic Blindness: 500,000 Sensors Confused an LLM - Raahul Singh & Vanč Levstik, Phaidra

Jul 12, 2026 · 16:25

Raahul Singh and Vanč Levstik of Phaidra argue that LLMs fail at scale with physical infrastructure data, a problem they call 'Semantic Blindness.' They show how deploying LLMs on 500,000 real sensors led to three failure modes: the topology trap (embeddings ignore physical causality), the illusion of scale (attention degrades with context), and the repetition kill switch (identical naming triggers model's repetition penalty). Their solution replaces raw context with a hierarchical tree, letting an LLM plan a search pattern and a deterministic resolver execute set operations via pre-indexed subtrees. This achieved 100% accuracy across 460,000 GPUs versus 30% with old approaches, cut tokens from 116 million to 390,000 per eval, and kept query cost flat at 9,000 tokens. They advocate starting with pure LLM for demos then migrating deterministic logic into code, inverting the traditional software evolution path.

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Jul 11, 2026 · 55:19

Elizabeth Fuentes (AWS) presents five code-based techniques to stop AI agent hallucinations, each with measurable before/after metrics. Semantic Tool Selection filters 29 tools to the 3 most relevant per query, cutting token usage from thousands to under 300 per call. Graph-RAG replaces vector similarity with structured graph queries (using Neo4j), enabling precise aggregation and multi-hop reasoning that vanilla RAG fabricates. Multi-Agent Validation uses an Executor-Validator-Critic swarm to catch fabrications, achieving a 92% detection rate. Neurosymbolic Guardrails enforce business rules in Python hooks that the agent cannot skip, achieving zero rule violations. Agent Steering guides agents to self-correct when soft rules fire, completing tasks without hard failures—demonstrated by booking 50 guests by intelligently splitting into two rooms.

How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI

Jul 7, 2026 · 14:28

Mixedbread AI co-founder Amir and AI engineer Hanna Lichtenberg argue the main bottleneck for knowledge agents is not reasoning but retrieval, coining the 'Oracle Gap'—the performance drop when agents use default tools. On BrowsComPlus, Oracle accuracy is 93% but Codex drops 9 points; with Mixedbread's search they recover to within 3 points. They show models write keyword gibberish (e.g., 'Senator, women, questions, billionaires, not a company') because they were trained on code tools like grep. To fix this, they built a search agent with 4 tools (overview, semantic, filter, grep) using a harness that encourages natural query sentences. Training involved supervised fine-tuning from a larger teacher and reinforcement learning with a reward combining retrieval metrics and trajectory quality (e.g., judging if queries are natural sentences). Their agent achieved NDCG@10 of 0.4 on Oblique Congress (vs. 0.18 for GPT multihop) and topped Snowflake's MedQA benchmark with 93.4% accuracy using Gemini 3.5 Flash.

We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

Jun 28, 2026 · 10:43

Rajkumar Sakthivel and his friend Faz built Code Context Engine (CCE) to cut AI coding costs after their bill jumped from £15 to £200 in one month. They found that 45,000 tokens were sent per query but only 5,000 were needed, with 90% of cost being input context. Their solution is a local retrieval layer that uses AST-aware chunks, combined vector and keyword search, and a weighted scoring heuristic (50% meaning, 30% keyword, 20% recency) that runs in 0.4 milliseconds. On a FastAPI test, tokens per question dropped from 83k to 4.9k—a 94% reduction—with 90% accuracy. They emphasize that fixing the input, not the model, yields the biggest savings, and their open-source tool shares a single index across Claude Code, Cursor, Copilot, and Codex.

Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Jun 28, 2026 · 45:48

Abed Matini from Ogilvy demonstrates bypassing the multimodal tax by building a local-first hybrid RAG system that converts documents to clean Markdown via Docling, eliminating cloud vision token overhead. Using Ollama (Qwen 2.5 0.5B), PostgreSQL with pgvector, and raw SQL with Reciprocal Rank Fusion, he implements a FAQ assistant for an employee handbook with four chunking strategies—heading-based, paragraph, fixed 512-char with 64% overlap, and sentence-based. The system combines dense embedding vectors and sparse keyword indices (BM25) in a single query, retrieves top 2 chunks, and uses Python functions for speed and testability. Guardrails block prompt injections and out-of-scope questions before reaching the LLM, and LangFuse tracks token usage and latency. Matini argues that a small, local model and code-controlled pipelines reduce hallucination and cost while maintaining full observability.

When All Context Matters: Extended Cache Augmented Generation - Luis Romero-Sevilla, Orbis

Jun 28, 2026 · 5:52

Luis Romero-Sevilla, VP of AI at Orbis Operations, presents Extended Cache Augmented Generation (ECAG) as a solution for knowledge representation when all documents in a collection are relevant and rapidly replaced. He argues that Simple RAG fails because it cannot pass all relevant documents to the LLM, and GraphRAG is computationally expensive due to constant graph recomputation. ECAG distributes documents across parallel KV-cache contexts (CAGs) and uses a supervisor model to interrogate each bucket, progressively building understanding without full recomputation. This approach balances speed and accuracy, outperforming GraphRAG on dynamic datasets while avoiding the degradation of single-context CAG. Romero-Sevilla notes that KV-cache costs can be reduced by optimizing cache lifetime.

RAG is dead, right?? — Kuba Rogut, Turbopuffer

Jun 9, 2026 · 11:13

Kuba Rogut (Turbopuffer) argues that RAG isn't dead—it's evolving into agentic retrieval, contrasting Cursor's upfront indexing (which boosted answer accuracy 24% in their composer model) with Claude Code's per-session grep approach. He frames embeddings as cached compute, citing Jeff Dean: 'You don't need a trillion at once, you need the right million.'

Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

May 29, 2026 · 20:12

Zach Blumenfeld from Neo4j argues that context graphs—a graph-based layer storing decision traces, precedence, and causal chains—extend standard RAG to make AI agents more accurate and explainable. A financial analyst agent, for example, can reject or accept a request by retrieving past decision traces and structurally similar precedents via graph embeddings, not just semantic similarity. Neo4j's one-command tool `uvx create-context-graph` scaffolds a full-stack app with backend, frontend, demo data, and MCP server, supporting 22 built-in domains or custom ontology generation. The underlying `neo4j-agent-memory` package handles entity extraction through a spaCy-to-GLiNER-to-LLM pipeline with deduplication and integrates with pydantic AI, LangGraph, Crew, Google ADK, and others. The episode also covers data connectors for GitHub, Notion, Jira, and Slack, and the ability to store and query decision traces with timestamps, though automated trace quality evaluation is still evolving.

Stop babysitting your agents... — Brandon Waselnuk, Unblocked

May 26, 2026 · 18:54

Brandon Waselnuk of Unblocked argues that the bottleneck for AI coding agents is not access (MCPs, tools) but understanding—they need a context engine that builds a research packet from codebase, Slack, PRs, and org structure before generating code. He debunks three myths: naive RAG suffers from satisfaction of search, more MCPs don't provide reasoning, and million-token windows don't enable effective reasoning. His team's context engine reduced a Zendesk integration task from 2.5 hours and 20.9M tokens (with MCPs only, but code that would have broken production) to 25 minutes and 10.8M tokens, earning a senior engineer's approval with one nitpick. Three hard lessons: optimize for understanding not access, resolve conflicts rather than hide them (e.g., Slack thread where CTO says code is wrong), and never cache answers because context changes daily. He demonstrates a social graph tool (open-sourced Monday) that maps engineers to code areas and collaborators, and shows a demo where the agent's MCP calls the context engine to generate a plan covering factory patterns, library modules, and client registration.

Connecting the Dots with Context Graphs — Stephen Chin, Neo4j

May 16, 2026 · 17:39

Stephen Chin of Neo4j argues that retrieval alone is insufficient for AI systems, because context graphs—which store relationships, reasoning traces, and decision provenance—enable grounded, auditable answers. He demonstrates with a healthcare example where a generic RAG system returns generic advice while a graph-grounded system knows the patient smokes and has had surgery, tailoring recommendations. The episode walks through Lenny’s Podcast memory demo and a financial services loan-decision app that surfaces prior rejections, margin trades, and fraud risk patterns, making the graph traversal visible. Chin notes Gartner has placed context graphs on the AI hype cycle and Foundation Capital called them a $3 trillion startup opportunity. Neo4j’s open-source agent memory package (short‑term, long‑term, reasoning memory) powers these context graphs, aiming to help engineers escape fragmented enterprise data and build explainable, policy‑aware AI.

Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked

May 3, 2026 · 1:41:25

Peter Werry of Unblocked argues that context engines—systems that supply AI agents with only the relevant organizational context—are critical to avoid agent doom loops and wasted tokens. He debunks three myths: naive RAG, connecting MCP servers, and bigger context windows do not solve the context problem. Werry describes building a social engineering graph to identify experts and distill team best practices, and shares hard lessons including hiding conflicts and caching answers. In a benchmark task, Unblocked's context engine reduced a 2.5-hour, 21-million-token task to 25 minutes and 10 million tokens. The talk offers a practitioner's guide to building context engines with conflict resolution, personalization, and access control.

How Google DeepMind is researching the next Frontier of AI for Gemini — Raia Hadsell, VP of Research

Apr 18, 2026 · 20:37

Google DeepMind VP of Research Raia Hadsell presents three non-language-model frontiers of AI: Gemini Embeddings 2, an omnimodal embedding model that unifies text, video, audio, and PDFs into a single semantic vector for fast retrieval; weather forecasting models including GraphCast (spherical GNN for 15-day forecasts), GenCast (probabilistic model 97% more accurate than benchmarks), and FGN (directly predicts cyclones, used by the US National Hurricane Center); and Genie world models that create interactive, memory-rich 3D environments from prompts, enabling real-time dynamic changes for gaming and education.

OpenRAG: An open-source stack for RAG — Phil Nash

Apr 8, 2026 · 15:52

Phil Nash introduces OpenRAG, an open-source RAG stack from IBM combining Docling, OpenSearch, and Langflow, arguing that RAG remains hard and custom despite claims it's dead — every business has unique data requiring more than simple vector search. Docling parses PDFs, audio, and more with specialized pipelines, outputting hierarchically chunked text. OpenSearch provides hybrid vector and keyword search using JVector for live indexing and disk-based ANN. Langflow enables visual agentic retrieval where the LLM decides searches with tools like a calculator and MCP servers. The stack supports local models via Ollama, cloud connectors for Google Drive and SharePoint, and an API. Nash demonstrates adding guardrails in Langflow and invites contributions to the open project.

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

Small Bets, Big Impact Building GenBI at a Fortune 100 – Asaf Bord, Northwestern Mutual

Dec 23, 2025 · 22:50

Asaf Bord, AI Product Lead at Northwestern Mutual, shares how his team built GenBI, an LLM-powered analytics copilot, by flipping the logic from a single big bet to an incremental roadmap of small, fundable projects. Using real, messy data from the 160-year-old company, they deployed a modular architecture with metadata, RAG, SQL, and BI agents, each productizable independently. The RAG agent alone automated 80% of the 20% of BI team capacity spent on finding and sharing reports, saving roughly two full-time employees. Bord explains how a crawl-walk-run release strategy built trust with both users and leadership, starting with BI experts before expanding to business managers, and how each six-week sprint delivered tangible business value—like proving the ROI of enriched metadata through A/B tests against a semantic layer initiative. He also explores the future of SaaS pricing in the GenAI era, questioning whether per-seat models still make sense when individuals become 10x more effective.

VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response — Suman Debnath, AWS

Dec 6, 2025 · 1:23:52

Suman Debnath, a Principal ML Advocate at AWS, demonstrates how Colpali—a vision-based retrieval model that treats each document page as an image and generates multi-vector embeddings via patch-based late interaction—can be combined with voice synthesis for a more intuitive RAG system. He explains that Colpali bypasses traditional OCR and preprocessing by directly embedding document images, then scoring query-page relevance through dot-product similarity across patches. The workshop shows how to embed pages, store them in Qdrant with multivector support, and retrieve top pages. Debnath then wraps this retrieval pipeline using the Strands Agent framework, adding a speak tool to output answers in natural voice. A live demo answers a textbook question about trophic levels, first using Bedrock to generate text and then speaking the answer in a female voice, all without a predefined system prompt. The talk positions Colpali as a complementary technique for complex visual documents like IKEA instructions or scanned forms, not a full replacement for traditional RAG.

Context Engineering: Connecting the Dots with Graphs — Stephen Chin, Neo4j

Nov 24, 2025 · 26:50

Stephen Chin, VP of Developer Relations at Neo4j, argues context engineering powered by knowledge graphs transforms AI from prompt engineering to information architecture, enabling agents with structured memory and retrieval. He demonstrates GraphRAG using Neo4j's knowledge graph builder with supply chain and VEX documents, showing how vector and graph algorithms retrieve specific Jackson library vulnerabilities with CVE details, severity, and remediation versions. Chin contrasts fast two-pass vector+graph retrieval with agentic multi-step traversal via the Neo4j MCP server and Claude Code, which yields deeper results like CVE number, attack type, and upgrade paths. He explains that graphs excel when relationships span two or more facts, using a presentation update example involving himself, colleague Sid, and the GIDS event. Practical applications include role-based access overlays and explainable AI via visualized conversation flows. Resources like free Graph Academy courses, Nodes AI 2026 conference, and graphrag.com support implementation.

How BlackRock Builds Custom Knowledge Apps at Scale — Vaibhav Page & Infant Vasanth, BlackRock

Aug 23, 2025 · 18:47

Infant Vasanth and Vaibhav Page of BlackRock explain how the world's largest asset manager scales custom AI knowledge apps for investment operations using a modular, Kubernetes-native framework. They detail challenges like prompt engineering for complex financial instruments, choosing LLM strategies (RAG, chain-of-thought), and deploying apps with cost control and compliance. Their sandbox lets domain experts rapidly iterate on extraction templates with inter-field dependencies and validations, while the app factory turns definitions into production apps. Key takeaways: invest in prompt engineering for domain experts, educate on LLM strategies, evaluate ROI, and design for human-in-the-loop in regulated environments. The framework compressed app development from eight months to days for use cases like setting up securities after IPOs or stock splits.

Wisdom-Driven Knowledge Augmented Generation at Scale - Chin Keong Lam, Patho AI

Aug 22, 2025 · 18:43

Chin Keong Lam, founder and CEO of Patho.ai, argues that Wisdom-Driven Knowledge Graphs (KAG) significantly outperform traditional RAG systems for complex quantitative analysis and expert-level advice. He presents a wisdom-decision-situation feedback loop where knowledge, experience, and insight feed into wisdom, mapped to a practical competitive analysis chatbot for a marketing client. The system uses N8n and a multi-agent architecture with a supervisory Wisdom Agent overseeing insight, knowledge, and strategy agents, all updating a centralized knowledge graph. Lam cites five reasons knowledge graphs beat RAG: capturing complex relationships, improved accuracy, scalability, rich multi-hop query capability, and seamless data integration. He demonstrates how vector RAG fails on numerical reasoning (e.g., Apple revenue query) while KAG returns precise evidence-based answers. Lam recommends a hybrid graph extraction combining LLM extraction with expert pruning, and reports benchmark results of 91% accuracy, 85% flexibility, and strong traceability and scalability.

How to look at your data — Jeff Huber (Chroma) + Jason Liu (567)

Aug 6, 2025 · 19:23

Jeff Huber (Chroma CEO) and Jason Liu argue that AI practitioners should look at their data—both inputs and outputs—with fast evals and conversation analysis to systematically improve retrieval and product decisions. Huber advocates for fast evals using golden datasets of query-document pairs over public benchmarks or expensive LLM judges, showing how synthetic queries aligned to real user behavior can empirically compare embedding models; in a Weights & Biases chatbot case study, Voyage 3 large outperformed text-embedding-3-small and others. Jason Liu focuses on outputs: extract structured metadata from conversations, cluster them to find segments, then compare KPIs like performance across clusters to prioritize what to fix, build, or ignore—for example, if 40% of conversations involve data visualization and the agent performs poorly, invest in better plotting tools. This population-level analysis enables impact-weighted decisions, turning vague metrics like 0.5 factuality into actionable insights by segment. The episode emphasizes that retrieval improvements are foundational, and once you have users, looking at conversation structure drives a data-driven product roadmap.

The 2025 AI Engineering Report — Barr Yaron, Amplify

Aug 1, 2025 · 12:33

Barr Yaron presents early findings from the 2025 State of AI Engineering survey at the AI Engineer World's Fair. The survey reveals that while titles vary widely, the community is broad and technical, with nearly half of seasoned software engineers having worked with AI for three years or less. Over half of respondents use LLMs for both internal and external use cases, with 3 of the top 5 models for customer-facing products from OpenAI. RAG is the most popular customization method (70%), and fine-tuning is surprisingly common, with 40% using LoRA or QLoRA. More than 50% update models monthly, and 70% update prompts at least monthly. A 'multimodal production gap' exists, with image, video, and audio usage lagging text. 80% say LLMs work well, but less than 20% say the same about agents, though most plan to use them. Evaluation is the top pain point.

Scaling Enterprise-Grade RAG: Lessons from Legal Frontier - Calvin Qi (Harvey), Chang She (Lance)

Jul 29, 2025 · 16:40

Calvin Qi of Harvey and Chang She of LanceDB detail how Harvey builds enterprise-grade RAG for legal use cases, tackling massive scale (tens of millions of documents per corpus), spikey workloads, and non-negotiable privacy. They explain why domain-specific reranking is essential for ambiguous, expert queries like 'What is the applicable regime to covered bonds issued before 9 July 2022 under Directive EU 2019/2062?' and stress that evaluation-driven development—from expert reviews to automated metrics—is key to accuracy. Chang shows how LanceDB's AI-native multimodal lakehouse, built on the open-source Lance format, enables low-latency, high-throughput retrieval across billions of vectors (indexing 3–4 billion in under 3 hours) while supporting offline analytics, GPU indexing, and compute-storage separation. The talk emphasizes that creative data understanding, iteration speed via good evals, and flexible infrastructure are critical for scaling RAG in domain-specific, privacy-sensitive environments.

Building Alice’s Brain: an AI Sales Rep that Learns Like a Human - Sherwood & Satwik, 11x

Jul 29, 2025 · 22:18

Sherwood Callaway and Satwik from 11x describe building 'Alice's brain' — a knowledge base that lets their AI sales development representative learn from seller materials as a human SDR would, using a full RAG pipeline. They outsourced parsing to vendors like LlamaParse (documents/images), Firecrawl (websites), and Cloud Glue (audio/video), converting resources into Markdown. Their chunking strategy splits on Markdown headers then sentences and tokens. Storage uses Pinecone vector database for similarity search, with bundled embedding models. For retrieval, they built a deep research agent with Letta that plans and executes context retrieval for email generation. A 3D UMAP visualization lets users inspect Alice's knowledge nodes. Key lessons: RAG is complex, get to production before benchmarking, and leverage vendors for expertise. Future plans include hybrid RAG with a graph database, hallucination tracking, and cost reduction.

Layering every technique in RAG, one query at a time - David Karam, Pi Labs (fmr. Google Search)

Jul 29, 2025 · 20:22

David Karam of Pi Labs (formerly Google Search) walks through layering every technique in RAG, from in-memory retrieval to planet-scale search with 70+ corpus mix of token, embeddings, and knowledge graphs, jointly retrieved and re-ranked at 160,000 queries per second in under 200msec. He advocates a Quality Engineering Loop: baseline with simplest methods, analyze losses, then apply incremental techniques based on complexity-adjusted impact. Karam explains why queries like "falafel" are notoriously hard due to ambiguous intent, highlights failures of chunking documents, and shows when BM25 suffices, when relevance embeddings are needed, and when custom embeddings or domain-specific signals (price, popularity, user clicks) become essential. He covers query orchestration via fan-out, supplementary retrieval across back ends, and distillation for cost optimization, concluding that at sufficient complexity, problems must be punted to LLM or UX for graceful degradation.

Building a Smarter AI Agent with Neural RAG - Will Bryk, Exa.ai

Jul 29, 2025 · 18:42

Will Bryk, CEO of Exa.ai, argues that traditional keyword search engines like Google are ill-suited for AI agents, which need semantic understanding, complex multi-paragraph queries, and comprehensive results. He explains Exa's neural search approach using embeddings to capture ideas and context, unlike keyword-based systems that miss nuances like negation. Bryk demonstrates a live-coded agent combining neural searches (e.g., finding personal sites of SF engineers who like information retrieval) with keyword searches (e.g., retrieving specific GitHub profiles), showing how Exa's API exposes toggles for date ranges, domains, and result counts. He emphasizes that AI agents require search engines that return exactly what they ask for, not what humans click on, and can handle thousands of results. The episode also previews Exa's new research endpoint for automated deep research.

Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Jul 27, 2025 · 1:48:07

Philipp Krenn of Elastic demonstrates that vector search is only a feature of information retrieval, not its foundation, in this hands-on workshop on retrieval for RAG. He contrasts classic keyword search—using tokenization, stemming, stopword removal, and BM25 scoring with the inverted index—with sparse (Elser/Splade) and dense (OpenAI text-embedding-small) vector embeddings. Krenn explains why hybrid search, combining lexical and semantic methods via reciprocal rank fusion (RRF), outperforms any single approach, especially for brand or exact-match queries where keyword search remains superior. He also covers scoring normalization, chunking strategies, and the new `text_similarity_reranker` retriever, showing how Elasticsearch's retrievers API enables two-stage retrieval with a cross-encoder for re-ranking. The session emphasizes that retrieval quality depends on evaluating against a golden dataset or LLM-as-judge, and that 'it depends' is the correct answer for choosing between PgVector and dedicated search engines.

Ship Production Software in Minutes, Not Months — Eno Reyes, Factory

Jul 25, 2025 · 16:06

Eno Reyes, cofounder and CTO of Factory, argues that AI agents can orchestrate the entire software development lifecycle, moving beyond vibe coding to agent-native development where enterprises delegate planning, coding, testing, and incident response to autonomous droids. He explains that AI tools are only as good as the context they receive—missing context from meetings, whiteboards, or Slack is the primary cause of failure, not LLM quality. Factory's droids search codebases, leverage organizational memory, and question unclear tasks before executing, from generating PRDs and tickets to creating runbooks and RCAs from sentry alerts. Reyes demonstrates how an agent can convert user transcripts and ad-hoc notes into a full feature plan, then break it into parallel tickets for multiple code droids. For incident response, droids pull logs, historical runbooks, and team discussions to produce mitigation plans in minutes, cutting response times in half and shifting from reactive to predictive operations. He emphasizes that the future belongs to engineers who manage agents—thinking clearly and communicating effectively—rather than those writing every line of code.

How Intuit uses LLMs to explain taxes to millions of taxpayers - Jaspreet Singh, Intuit

Jul 23, 2025 · 18:59

Jaspreet Singh, Senior Staff Engineer at Intuit, explains how TurboTax uses Anthropic's Claude and OpenAI models to generate personalized tax explanations for 44 million customers. The system, built on Intuit's GenOS platform, combines prompt-engineered static explanations with dynamic question-answering using RAG and graphRAG for tax-specific queries. Singh details their fine-tuning of Claude Haiku on AWS Bedrock, which improved quality but proved too specialized. A key focus is evaluation: manual reviews by tax analysts followed by automated LLM-as-a-judge scoring for accuracy, relevancy, and coherence, with a golden dataset and safety guardrails to prevent hallucinated numbers. He highlights challenges like latency spikes on tax day (3–10 seconds), vendor lock-in through expensive contracts, and the difficulty of upgrading models even within the same vendor. Singh emphasizes that evaluations are essential for launching any GenAI feature in a regulated domain like tax.

POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent

Jul 23, 2025 · 19:16

Randall Hunt from Caylent shares hard lessons from over 200 enterprise GenAI deployments, arguing that evals, embeddings, and prompt engineering matter far more than fine-tuning. He emphasizes that speed and UX are critical; a slow inference kills adoption, while techniques like generative UI and caching can mitigate latency. Hunt details real-world examples: using audio amplitude spectrographs for sports highlight reels, pooling multimodal embeddings for nature footage search, and noting that nurses prefer chat over voice bots in noisy hospitals. He reports zero regressions moving from Claude 3.7 to 4, and advises optimizing context and economics, such as leveraging Amazon Bedrock batch for 50% cost reduction. The talk underscores that knowing your end customer and minimizing irrelevant context are key to production success.

The Billable Hour is Dead; Long Live the Billable Hour — Kevin Madura + Mo Bhasin, Alix Partners

Jul 23, 2025 · 17:04

Kevin Madura and Mo Bhasin from AlixPartners argue that AI is reshaping knowledge work by compressing upfront data ingestion from 50% to 10-20% human effort, enabling analysis of 100% of data rather than the top 20%. They detail three use cases: categorization via structured outputs achieving 95% accuracy on 10,000 vendors in minutes; enterprise-scale RAG that democratizes access to siloed data by teaching LLMs to call APIs; and structured extraction from documents using schemas and log-prob-based confidence scoring. They caution that AI investments face a paradox—89% of CEOs plan agentic AI but NBER finds no earnings impact—and stress that success requires people skills, demos, and a focus on NPS and ROI over chasing shiny new tools. The episode concludes that once Excel-powered LLMs work reliably, AGI will be here.

Top Ten Challenges to Reach AGI — Stephen Chin, Andreas Kollegger

Jul 22, 2025 · 4:23

Stephen Chin and Andreas Kollegger, curators of the GraphRAG track at AI Engineer World's Fair, use ten sci-fi memes to expose top challenges to AGI—from Memento's short-term memory as prompt engineering to Skynet's unintended consequences, the Matrix's agent simulation, Hal's trust issues, emotions as bug/feature, Frankenstein's creator obligations, Terminator's time travel, Star Wars' language nuance, the Borg's assimilation, and Deep Thought's right questions. They argue graphs and graph technology can solve some of these challenges, inviting the audience to the GraphRAG track to learn which ones. The episode delivers a playful yet pointed framework connecting pop culture to core AGI obstacles.

Practical GraphRAG: Making LLMs smarter with Knowledge Graphs — Michael, Jesus, and Stephen, Neo4j

Jul 22, 2025 · 19:46

Michael Hunger and Stephen Chin of Neo4j present GraphRAG as a method to enhance LLMs by integrating knowledge graphs, achieving more accurate, contextual, and explainable answers than standard RAG. They highlight limitations of vector search, showing GraphRAG improves relevance and reduces hallucinations. They detail a three-step construction process: lexical graphs, entity extraction, graph enrichment with algorithms. They demonstrate open-source tools like Neo4j Knowledge Graph Builder and GraphRAG Python package, and show an agentic approach using domain-specific retrievals. They cite studies showing three times improvement in accuracy.

When Vectors Break Down: Graph-Based RAG for Dense Enterprise Knowledge - Sam Julien, Writer

Jul 22, 2025 · 15:47

Sam Julien, Director of Developer Relations at Writer, explains how the company built a graph-based RAG system that achieved 86.31% accuracy on the RobustQA benchmark and sub-second response times, significantly outperforming vector search approaches for dense enterprise knowledge. The team moved from simple vector retrieval to a graph-based approach to handle concentrated data where similar terms appear frequently, solving problems like inaccurate chunking and failure with similar documents. They built a specialized model to convert data into graph structures, stored as JSON in a Lucene-based search engine, and incorporated Fusion-in-Decoder with knowledge graphs to lower hallucination rates. Key decisions included focusing on customer needs over hype, staying flexible based on team expertise, and letting research challenge assumptions, leading to features like multi-hop reasoning and complex data format handling.

HybridRAG: A Fusion of Graph and Vector Retrieval - Mitesh Patel, NVIDIA

Jul 22, 2025 · 20:24

Mitesh Patel, Developer Advocate Manager at NVIDIA, presents HybridRAG, a fusion of knowledge graph-based GraphRAG and vector-based VectorRAG for improved question-answering from complex texts. He emphasizes that ontology engineering consumes 80% of development time and is critical for accurate triplet extraction, where fine-tuning LLaMA 3.1 with LoRA boosted triplet accuracy from 71% to 87% on 100 documents. Patel also highlights retrieval strategies like multi-hop graph traversal, which provides richer context but increases latency, and recommends QGraph acceleration via Networx to reduce latency. For evaluation, he suggests RAGAS for end-to-end pipeline metrics and the LLaMA-Nimotron reward model for response quality. Ultimately, he advises using GraphRAG when data has inherent structure or complex relationships, but notes it is compute-heavy, so the choice between GraphRAG, semantic RAG, or hybrid depends on the use case.

Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Jul 21, 2025 · 15:27

Stephen Chin of Neo4j argues that Agentic GraphRAG — combining graph databases with retrieval-augmented generation — overcomes LLM hallucinations and biases by providing structured, relational context. He demonstrates how LLMs fail on reasoning tasks like calculating classroom capacity due to inaccurate anchoring on irrelevant data, and proposes an architecture where vector search first identifies relevant nodes, then graph traversal retrieves related context for the LLM. Chin highlights Neo4j’s MCP server for cypher query generation and memory modules, and cites Klarna’s success: 250K employee questions answered in the first year, 2,000 daily queries, and 85% adoption, replacing their entire SaaS stack. He recommends the Neo4j Certified Developer Program and the Nodes Conference for further learning.

Transforming search and discovery using LLMs — Tejaswi & Vinesh, Instacart

Jul 16, 2025 · 21:10

Vinesh Gudla and Tejaswi Tenneti from Instacart detail how they use LLMs to overhaul search and discovery. They address challenges with conventional search: broad queries suffer cold start, tail queries lack engagement data. Using LLMs with Instacart's domain knowledge—e.g., top-converting categories as context—they improved precision by 18 percentage points and recall by 70% for tail queries, and cut zero-result queries. For discovery, LLMs generate complementary/substitute items; but pure LLM suggestions like 'chicken' for 'protein' missed user intent, so they augmented prompts with behavioral data (top categories, subsequent queries) to boost engagement and revenue. They use a hybrid approach: pre-compute offline for head/torso, distilled Llama 8B for long-tail, and LLMs as judges for evaluation. Key takeaways: combining LLM world knowledge with domain-specific data is critical, and evaluation is as hard as generation.

What We Learned from Using LLMs in Pinterest — Mukuntha Narayanan, Han Wang, Pinterest

Jul 16, 2025 · 18:13

Pinterest search engineers Han Wang and Mukuntha Narayanan present four key learnings from integrating LLMs into the platform's search relevance system. Fine-tuned LLM cross-encoders improve relevance prediction by 12% over multilingual BERT and 20% over SearchSage using an 8B model. VLM-generated image captions and user action features enrich pin text representations and further boost performance. Knowledge distillation into a bi-encoder student model, trained on 100x more data via semi-supervised learning, enables production serving with real-time query embeddings and 85% cache hit rate. Relevance-tuned embeddings serve as general-purpose semantic representations across multiple surfaces, and the system yields relevance gains in multiple languages and countries despite a predominantly US training set.

A year of Gemini progress + what comes next — Logan Kilpatrick, Google DeepMind

Jul 10, 2025 · 11:58

Logan Kilpatrick, head of product for Google AI Studio at DeepMind, announces the final update to Gemini 2.5 Pro, which achieves state-of-the-art results on Aider and HLE benchmarks. He details Google's 50x increase in AI inference over the past year, driven by merging research and product teams into DeepMind. The episode outlines Gemini's evolution toward a universal assistant that unifies Google products, with upcoming features including proactivity, native audio and video capabilities (Veo), and smaller models. Kilpatrick also previews developer-focused updates: a SOTA embeddings model, a deep research API, and Veo 3 and Imagine 4 in the API, alongside repositioning AI Studio as a dedicated developer platform.

Intro to GraphRAG — Zach Blumenfeld

Jun 30, 2025 · 1:18:35

Zach Blumenfeld introduces GraphRAG using Neo4j, showing how to build knowledge graphs from employee and skill data, combine structured and unstructured data, and use graph traversal with vector search for retrieval. The workshop covers Cypher query patterns for multi-hop similarity, entity extraction from resumes via LLMs, and creating semantic similarity relationships. It demonstrates Leiden community detection for skill clustering and builds a LangGraph agent with four custom tools that balance exact skill matches with vector similarity. Blumenfeld argues that knowledge graphs provide controlled, explainable retrieval logic for agentic workflows, enabling developers to decompose data into graph models that expose domain logic for more accurate retrieval than pure vector search.

Data is Your Differentiator: Building Secure and Tailored AI Systems — Mani Khanuja, AWS

Jun 27, 2025 · 20:10

Mani Khanuja from AWS explains how data differentiates generative AI applications, emphasizing that foundation models require careful data handling specific to each use case. She demonstrates Amazon Bedrock's capabilities: Bedrock Data Automation for custom pipelines with a single API, Knowledge Bases for RAG with out-of-box chunking and hybrid search, and Guardrails for responsible AI with PII filtering. Using a travel agent example, she details data needs—customer profiles, travel policies—and introduces the 'coconuts' framework: chunking, optimization via semantic cache and re-ranking, observability, evaluation (context relevance), test suites, and iterative updates. She argues that organizations must crack their coconuts—rigorously prepare data and evaluate applications—before scaling into production securely.

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Jun 27, 2025 · 1:15:43

Rajiv Shah, Nina Lopatina, and Matthew from Contextual AI demonstrate how developers can build production-ready RAG agents in minutes using Contextual AI's managed RAG platform, emphasizing that RAG should be treated as a managed service to avoid reinventing infrastructure. They walk through ingesting documents like NVIDIA financials and spurious correlation reports, then querying the agent with questions requiring quantitative reasoning across tables. The platform handles extraction, layout analysis, image captioning, hybrid retrieval, and a state-of-the-art reranker, capped by a grounded language model that avoids hallucinations and provides attribution. Evaluation is done via LM Unit, a model-as-judge that scores responses on criteria like accuracy and causation. The episode also shows integrating the agent with Claude Desktop via MCP and answers audience questions on pricing (consumption-based with a $25 credit), scalability, entitlements, HIPAA, and domain-specific language.

RAG in 2025: State of the Art and the Road Forward — Tengyu Ma, MongoDB (acq. Voyage AI)

Jun 27, 2025 · 18:48

Tengyu Ma, Chief AI Scientist at MongoDB (acquired Voyage AI), argues that retrieval-augmented generation (RAG) will outlast fine-tuning and long-context models for enterprise AI because it mirrors how humans use libraries—retrieving only relevant information rather than memorizing or reprocessing entire corpora. He reports that Voyage's embedding models now achieve ~80% average accuracy across 100 datasets, with half exceeding 90%, and that 100x storage compression via matryoshka learning and quantization loses only 5–10% accuracy. Ma advocates hybrid search with rerankers and domain-specific embeddings (e.g., for code) to further improve retrieval. He predicts the model layer will absorb current 'tricks' like query decomposition and contextual chunking, simplifying RAG pipelines. New products like multimodal embeddings (accepting screenshots of PDFs, tables, videos) and auto-chunking embeddings that incorporate cross-chunk metadata are designed to shift complexity from users to model providers.

The State of AI Powered Search and Retrieval — Frank Liu, MongoDB (prev Voyage AI)

Jun 27, 2025 · 12:35

Frank Liu from Voyage AI (now part of MongoDB) presents the state of AI-powered search and retrieval, arguing that embedding quality is core but not sufficient—systems must integrate structured data filtering and agentic query decomposition for real-world RAG applications. He details three live use cases: chatting with codebases (where Continue.dev found Voyage Code 3 best, emphasizing domain-specific evaluation), legal document search requiring pre-filtering by metadata, and agentic retrieval where LLMs decompose queries into multiple sub-queries. Looking ahead, Liu predicts three trends: multimodal embeddings that combine text and images (e.g., Voyage Multimodal 3 embedding Claude 3.5 blog posts), instruction tuning to steer embeddings toward specific document aspects, and the rise of agent-native databases (like MongoDB integrating Voyage for embedding, reranking, and query augmentation within a single platform).

Why Your Agent’s Brain Needs a Playbook: Practical Wins from Using Ontologies - Jesús Barrasa, Neo4j

Jun 27, 2025 · 13:54

Jesús Barrasa, AI Field CTO at Neo4j, argues that ontologies are the secret weapon for building robust GraphRAG applications, providing both a formal, implementation-agnostic schema for knowledge graph creation and a dynamic data layer that drives retriever behavior. He explains how ontologies replace rigid code-orchestrated workflows and chaotic LLM-generated ones by guiding entity extraction from unstructured data and mapping structured sources into a property graph. During retrieval, storing the ontology in the graph enables dynamic Cypher queries that automatically navigate contextualizing relationships — such as "acted-in" rather than "directed" — based on annotated subproperties, without hard-coded logic. This allows developers to change retrieval behavior on the fly by modifying the ontology as a data artifact. The talk covers two practical wins: using ontologies as a shared domain model for graph construction, and leveraging them to build adaptive, schema-driven retrievers that improve result completeness and relevance.

Graph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics - Alison & Andreas, Neo4j

Jun 27, 2025 · 1:41:16

Alison Cossette and Andreas Kollegger from Neo4j demonstrate how graph data science algorithms enhance GraphRAG systems by connecting, clustering, and curating unstructured data. They walk through running KNN similarity (K=25) on document embeddings, then applying Louvain community detection to identify clusters of near-identical chunks—one community had 49 documents with 0.98 average similarity—which can be collapsed via APOC to improve retrieval efficiency while preserving lineage. Using page rank and betweenness centrality, they show how to surface influential documents and track conversation paths across communities. The session uses a preloaded Neo4j AuraDB database with 17K nodes and 774K relationships from the Agent Neo project, and includes practical cipher queries for projecting graphs and running algorithms. They emphasize using graph analytics to diversify retrieval, manage content hygiene, and build accountable agent systems.

GraphRAG methods to create optimized LLM context windows for Retrieval — Jonathan Larson, Microsoft

Jun 27, 2025 · 15:09

Jonathan Larson of Microsoft Research presents GraphRAG methods, arguing that LLM memory with structure is a key enabler for effective AI applications, and that agents paired with these structures amplify their power. He demonstrates GraphRAG for Code, which uses graph-based understanding to outperform regular RAG on code comprehension, translation from Python to Rust, and even multi-file feature addition like adding jumping to the original DOOM codebase. Larson announces Benchmark QED, an open-source evaluation framework for measuring local and global query quality across comprehensiveness, diversity, empowerment, and relevance. He introduces Lazy GraphRAG, which beats vector RAG on 92% of local and 90% of global queries at one-tenth the cost of million-token context windows, and is being integrated into Azure Local and Microsoft Discovery.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo

Jun 3, 2025 · 9:49

Charlie Guo explains how his team at Pully used Claude 3.5 Sonnet to analyze 10,000 sales call transcripts in two weeks, turning a task that would take 625 days manually into a $500 project. They chose Claude over smaller models due to unacceptable hallucination rates, and reduced costs by up to 90% using prompt caching and extended outputs. The system combined RAG enrichment, chain-of-thought prompting, and structured JSON outputs with citations to ensure accuracy. The analysis, originally for the executive team, became a company-wide resource, enabling marketing to pull customer quotes and sales to automate transcript downloads, saving dozens of hours weekly. Guo emphasizes that good engineering—JSON outputs, database schemas, and thoughtful integration—matters as much as AI capability, and challenges listeners to mine their own untapped customer data.

open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Jun 3, 2025 · 5:03

Ofer Mendelevitch from Vectara presents Open-RAG-Eval, an open-source framework that enables RAG evaluation without requiring golden answers or golden chunks, solving a major scalability problem. Backed by research with the University of Waterloo's Jimmy Lin Lab, it uses UMBRELA for retrieval scoring on a 0–3 scale that correlates well with human judgment, and AutoNuggetizer for generation via nugget creation, Vital/OK ratings, and an LLM judge analyzing the top 20 nuggets for support. Additional metrics include citation faithfulness and Vectara's HHEM hallucination detection model. Connectors are available for Vectara, LangChain, and LlamaIndex, with results viewable through an intuitive UI at OpenEvaluation.ai.

Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer

Apr 22, 2025 · 12:01

Writer CTO Waseem Alshikh presents FailSafe, a benchmark for evaluating LLMs in real-world finance, challenging the notion that general-purpose models suffice. Tests on query failures (misspelling, incomplete, out-of-domain) and context failures (missing context, OCR errors, irrelevant context) reveal that reasoning models like o1 and o3 perform 50% to 60% worse on context grounding than smaller domain-specific models, despite higher answerability. Even the best model achieves only 81% combined robustness and grounding, meaning roughly 20% of responses are wrong. Alshikh argues that domain-specific models remain essential, and that full-stack systems—RAG, guardrails, and grounding—are necessary for reliable deployment in high-stakes finance.

RAG Agents in Prod: 10 Lessons We Learned — Douwe Kiela, creator of RAG

Apr 10, 2025 · 16:56

Douwe Kiela, CEO of Contextual AI and creator of RAG, shares 10 lessons from deploying enterprise RAG systems at scale. He argues that language models are only 20% of a larger system; success comes from focusing on systems, not models, and specializing over AGI to unlock domain expertise. Enterprise data is the real moat, but pilots are easy while production is hard—design for production from day one. Speed beats perfection: ship barely functional to real users early and iterate. Avoid boring engineering chores like chunking; instead, integrate AI into existing workflows to drive adoption. Accuracy is table stakes; handle inaccuracy with observability and attribution. Be ambitious: aim for transformative ROI, not low-hanging fruit like basic HR questions.

Anchoring Enterprise GenAI with Knowledge Graphs: Jonathan Lowe (Pfizer), Stephen Chin (Neo4j)

Apr 7, 2025 · 20:59

Stephen Chin (Neo4j) and Jonathan Lowe (Pfizer) explain how Pfizer uses knowledge graphs with GraphRAG to accelerate drug manufacturing technology transfer, cutting time from years to weeks. They argue graph databases provide superior accuracy and explainability over vector-only RAG, crucial for life-saving drugs. Jonathan details navigating organizational silos in a 100,000-person company, from C-suite taglines to client partners demanding cost savings. Gartner's prediction of 30% GenAI project failure is addressed with a concrete business case: manufacturing worker tenure plunged from 20 to 3 years, making AI essential to capture lost expertise. The architecture combines vector and graph retrieval to deliver contextually relevant, auditable answers, reducing data consolidation from three months to three weeks.

OpenAI for VP's of AI + Advice for Building Agents

Mar 5, 2025 · 16:52

OpenAI's Toki Sherbakov and Prashant Mital explain how enterprises adopt AI through a three-phase journey: building an AI-enabled workforce with ChatGPT, automating operations with APIs, and infusing AI into end products. They detail a Morgan Stanley case study where retrieval methods improved an internal knowledge assistant's accuracy from 45% to 98%. The pair define agents as models with instructions, tools, and self-terminating execution loops, then share four field lessons: build with primitives before frameworks, start with a single purpose-built agent, graduate to a network of specialized agents with handoffs for complex tasks, and keep prompt instructions simple while running guardrails in parallel using fast models like GPT-4o mini for safety and reliability.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

Your LLM Ran Out of Knowledge — Now What?

Feb 22, 2025 · 12:51

Speaker 1 presents a technique for applying LLMs to low-knowledge domains like corporate negotiations and geopolitics, where structured training data is scarce. The method uses a parsing engine to identify the problem type and match domain-specific heuristics—binary rules like 'must prioritize agreements with highest combined value' for negotiations or 'must have three independent paths for critical resources' for geopolitics. After reformatting the input for consistency, the system sends it to a reasoning model (e.g., Anthropic's Claude with a world sim) that applies the rule set alongside powerful reasoning capabilities. The demo shows an intelligence estimate scenario where the model identifies geopolitics, reformats the query, and evaluates options against rules—eliminating those that breach heuristics (e.g., scenario five violating a rule on control thresholds). The approach enables generating dozens of scenarios in a fraction of the time a human expert would take, allowing practitioners to explore more options and find novel solutions while keeping a human in the loop for subject-matter oversight.

The Hidden Costs of Building Your Own RAG Stack — Ofer Vectara

Feb 22, 2025 · 15:14

Ofer from Vectara argues that building your own RAG stack comes with seven hidden costs: irrelevant responses and hallucinations, high latency, scaling and cost issues, security and compliance challenges, vendor chaos from multiple components, unsustainable expertise requirements, and limited non-English language support. He details how each pitfall compounds at enterprise scale, citing the need for hybrid search, reranking, and continuous evaluation to maintain quality. Vectara offers a turnkey RAG-as-a-service platform that handles parsing, chunking, embedding, vector search, retrieval, and hallucination detection. Ofer highlights Vectara’s open-source HHEM model for hallucination evaluation, downloaded over 3 million times, and a leaderboard showing some LLMs hallucinate at rates above 10%. The platform supports deployment in SaaS, VPC, or on-premises, with built-in access control and explainability.

The Adversarial Path to the Personal Assistant: Sumit Agarwal

Feb 15, 2025 · 18:46

Sumit Agarwal, founder of Ario AI, discusses building a personal AI assistant that uses adversarial ETL to extract users' data from services like Google, Amazon, and Doordash, enabling immediate personalization without manual input. He announces $16M in funding and demonstrates how the assistant generates data portraits, recommends travel and routines based on personal history, and manages schedules by detecting conflicts. Agarwal shares RAG insights: avoid LLMs for trivial math, use search and pre-processed profiles, and present the right data at the right time. Ario Boost, a browser extension, lets users download their data locally in developer mode without creating an account.

RAG at scale: production ready GenAI apps with Azure AI Search

Feb 13, 2025 · 21:53

Pablo, a Distinguished Engineer at Microsoft, explains how Azure AI Search enables production-grade RAG applications at scale by addressing key scaling challenges: data volume, query load, workflow complexity, and diverse data types. He demonstrates the platform's hybrid retrieval combining vector search (HNSW, exhaustive), keyword search (BM25), and filters, with a two-stage system using cross-encoder re-ranking for improved quality, achieving 100ms re-ranking latency. Azure AI Search supports multi-billion vector indexes through 10–12x storage limit increases and quantization (int8, single-bit) preserving 90–95% precision, with oversampling for full-precision re-ranking. Integrated vectorization automates ingestion from Azure sources (blob, Cosmos DB, OneLake) with change tracking and format handling (PDFs, Office documents, images). OpenAI uses Azure AI Search to back ChatGPT file uploads and the Assistants API, scaling user limits 500x after Microsoft's infrastructure upgrades.

RAG and the MongoDB Document Model: Ben Flast

Feb 8, 2025 · 13:13

Ben Flast, Director of Product at MongoDB, explains how the MongoDB document model and Atlas Vector Search enable more efficient and scalable Retrieval Augmented Generation (RAG). He contrasts MongoDB's JSON-based documents with relational databases, highlighting that documents store application objects directly without stitching tables. Flast details MongoDB's HNSW-based vector search, which stores embeddings alongside transactional data in the same document, supporting up to 4,096 dimensions. He introduces Search Nodes, which decouple vector search infrastructure from transactional databases for independent scaling. Flast covers AI integrations with LangChain, LlamaIndex, and others, enabling features like semantic caching and chat history memory within a single database. He cites the startup 4149, which uses MongoDB to store user data, meeting notes, and vector embeddings for an AI teammate that assists with tasks by combining transactional and semantic search.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

EyeLevel Launch: Your RAG is Tripping, Here's the Real Reason Why

Feb 6, 2025 · 6:13

Benjamin Over, co-founder of EyeLevel.ai, argues that 95% of RAG hallucinations stem from content ingestion problems, not LLMs or prompts, and introduces his platform's approach to solve this. EyeLevel avoids vector databases entirely, instead creating semantic objects with auto-generated metadata and rewriting text for search and completion. Its pipeline uses a fine-tuned vision model to extract tables, figures, and text, then applies nine fine-tuned models for ingestion and search, including a re-ranking LLM. Air France uses EyeLevel to build a call-center co-pilot, achieving over 95% accuracy on hundreds of thousands of complex documents. In a study, EyeLevel reached 98% accuracy on real-world documents, outperforming popular solutions by up to 120%.

BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

Feb 5, 2025 · 5:47

BotDojo founder Paul Henry demonstrates how his platform uses synthetic data generation and evaluations to productionize AI assistants. He shows a chatbot template with a low-code editor, node tracing, and JSON schema support. By running batch evaluations, he identifies knowledge gaps in the vector database. Then, he generates synthetic test data from live support tickets by extracting question-answer pairs and writing new documents to the index. Running the evaluations again, the scores improve from having red failures to all green, measurably boosting the chatbot's performance. Henry stresses that while hooking up a vector database to an LLM is easy, getting it production-ready requires systematic evaluation and data augmentation.

Building efficient hybrid context query for LLM grounding: Simrat Hanspal

Feb 5, 2025 · 11:35

Simrat Hanspal from Hasura explains how to build efficient hybrid context queries for LLM grounding using Hasura's GraphQL API, demonstrated via an e-commerce product search use case. The talk covers three query types: semantic search (e.g., 'essential oils for relaxation'), structured search (e.g., 'products less than $500'), and hybrid queries combining both (e.g., 'essential oil diffusers between $500 to $1,000'). Hanspal shows how Hasura unifies relational (Postgres) and vector (Weaviate) databases into a single GraphQL API, auto-vectorizing records via events. A critical security demonstration highlights how role-based permissions (e.g., 'Product Search Bot' role with only select access) prevent malicious insert mutations like adding a fake product with ID 7001. The approach enables secure, dynamic data retrieval for Retrieval-Augmented Generation (RAG) pipelines without building separate APIs.

Cohere for VPs of AI: Vivek Muppalla

Feb 5, 2025 · 16:11

Vivek Muppalla, Director of Engineering at Cohere, details the company's enterprise AI strategy centered on security, customization, and deployment flexibility. He presents Cohere's product line: Command R and R+ for generation, plus advanced retrieval models like embeddings and the ReRanker, which reduces RAG costs by narrowing context. Key claims include a focus on enterprise-specific eval suites (health, HR, finance), out-of-the-box citations, and multilingual performance. Partnerships with Accenture and McKinsey bridge the last-mile gap, while wins often stem from private cloud deployment and data control. In Q&A, he recommends purpose-built classifiers for high-throughput production and notes current 128k context windows.

Navigating Challenges and Technical Debt in LLMs Deployment: Ahmed Menshawy

Dec 31, 2024 · 16:15

Ahmed Menshawy, VP of AI Engineering at Mastercard, argues that deploying LLMs requires overcoming massive technical debt, with over 95% of the work outside the ML code. He details how Mastercard boosted fraud detection by 300% using LLMs while adhering to seven responsible AI principles. The episode covers RAG as a solution for hallucination, attribution, and access control, noting that 80% of organizational data is unstructured and 71% of companies struggle to manage it. Menshawy highlights that GPU compute for inference now exceeds training, and that fine-tuning open-source models enables joint training of retriever and generator. He also cautions against AGI hype, quoting Ada Lovelace and a Nature article urging focus on current AI risks.

RAG for VPs of AI: Jerry Liu

Dec 31, 2024 · 26:51

Jerry Liu, CEO of LlamaIndex, argues that building production RAG systems requires a new data processing stack distinct from traditional ETL, emphasizing data quality through advanced parsing (Llama Parse) and rigorous eval over chunk size tuning. He explains that while longer context windows may eliminate fine-grained chunking, retrieval from external storage remains vital for multi-doc enterprise systems. Liu addresses data privacy with VPC deployment options, and notes that Llama Cloud processes but does not store data. He also highlights Llama Agents, an open-source multi-agent framework for deploying agentic microservices, and shares that Llama Parse processes tens of millions of pages monthly.

Understanding AI Stakes to Break Production Code: Philip Rathle

Dec 31, 2024 · 23:24

Philip Rathle, CTO of Neo4j, argues that the level of 'stakes' in an AI application determines the production barriers and appropriate solutions, from low-stakes summarization to high-stakes uses requiring knowledge graphs and human-in-the-loop systems. He distinguishes pilot (full autonomy) from co-pilot (human oversight), showing how vector RAG solves moderate stakes but fails for high-stakes needs like tightening bolts on a 737 MAX 9. He promotes Graph RAG for deterministic reasoning and fact retrieval. Attendees share learnings: using LLMs to write code rather than reason directly, adapting to user keyword habits, focusing on few projects with achievable accuracy, building eval pipelines early, and for regulated loans offering three options with explanations instead of a single recommendation.

Navigating RAG Optimization with an Evaluation Driven Compass: Atita Arora and Deanna Emery

Nov 12, 2024 · 18:14

Atita Arora (Qdrant) and Deanna Emery (Quotient AI) demonstrate how evaluation-driven optimization improves Retrieval Augmented Generation (RAG) systems, using Qdrant's vector database and Quotient's evaluation platform. They detail a 10-experiment pipeline on Qdrant documentation, starting with naive RAG and incrementally testing chunk sizes, embedding models, LLMs (Mistral, GPT-3.5), re-rankers (MixedBread, Cohere, ColBERT), and hybrid search. Key metrics—faithfulness (hallucination reduction), context relevance, and chunk relevance—guide decisions: increasing chunk size hurt faithfulness (0.76 to 0.72), while smaller chunks with larger retrieval windows improved it. Switching to GPT-3.5 lifted all metrics, but poor context relevance revealed retrieval issues. Adding Cohere re-ranking boosted faithfulness to 0.82, and hybrid search with re-ranking achieved 0.85, proving domain-specific terminology demands tailored search strategies. The episode stresses avoiding over-engineering without metrics, keeping evaluation datasets current, and using domain understanding for systematic RAG improvement.

Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

Nov 1, 2024 · 1:39:52

In this workshop, Zach Blumenthal of Neo4j demonstrates how to build a GraphRAG application using Neo4j, OpenAI, and LangChain on an H&M fashion dataset. He argues that combining knowledge graphs with vector search and graph embeddings improves retrieval and personalization for LLMs. The session covers creating a Neo4j sandbox, loading data, performing vector search with OpenAI embeddings, adding collaborative filtering via graph traversal patterns, using graph data science (FastRP) to generate graph embeddings for recommendations, and integrating everything into a LangChain chain that generates personalized marketing emails. Attendees build a Gradio app that, given a customer ID and season, outputs a tailored email with product recommendations, demonstrating how structured graph data enhances RAG systems beyond pure vector search.

Realtime Data Connectivity for AI: Tanmai Gopal

Oct 11, 2024 · 7:14

Tanmai Gopal from Hasura introduces Pacha DDN, an AI-powered data access layer that lets LLMs securely query live data from multiple sources, arguing that the key is treating all data—structured, unstructured, and APIs—with a unified SQL-based query language. He demonstrates with a Blockbuster example: writing an email to a top customer by querying database transactions and recent rentals via natural language. The system uses an object model for authorization that applies rules based on data schema and session properties, regardless of data origin. To overcome LLM reasoning limitations, Pacha DDN asks the LLM to write Python code to retrieve data instead of reasoning directly. The talk concludes that for AI to be useful, it needs realtime data access, and Pacha DDN provides a secure, explainable query planner to achieve that.

We accidentally made an AI platform: Jamie Turner

Oct 8, 2024 · 5:36

Jamie Turner, co-founder of Convex, explains how his company's backend platform accidentally became an AI platform because its reactive data flow paradigm—extending React's state reactivity to the server with subscribable queries and mutations—perfectly suits generative AI workflows. He describes Convex's architecture as seamlessly syncing state between backend steps and the application, enabling concurrent chains of tasks like automatic speech recognition, summarization, embedding generation, and finding related notes. Turner notes that post-ChatGPT, over 90% of Convex projects are generative AI. To support this shift, Convex added native vector indexes for schema fields and launched a startups program with discounts and exclusive forums. Upcoming high-level components will encapsulate state machines for sophisticated workflows. Turner presents Convex as a way for generative AI engineers to ship quickly and confidently.

LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries: Hubert Misztela

Sep 23, 2024 · 20:00

Hubert Misztela, an AI researcher at Novartis, argues that LLMs require more than naive RAG to achieve scientific reasoning capable of Nobel-level discoveries. He uses the 1990s petunia flower experiment — where three separate biological phenomena (including RNA interference) went unexplained for eight years until their common cause was found — as a benchmark. Misztela classifies question complexity from one-to-one to multi-needle problems and demonstrates that reasoning before retrieval (e.g., routing, GraphRAG) and after retrieval (e.g., relevance classifiers) improves hypothesis generation. His experiments show that strict prompting and a relevance classifier that evaluates each paper's contribution to advancing a hypothesis can recover the correct RNA/DNA link without post-discovery knowledge. He concludes that harder problems demand reasoning steps and that brute-force checking with LLMs may outperform embedding distances.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Going beyond RAG: Extended Mind Transformers - Phoebe Klett

Sep 11, 2024 · 16:04

Phoebe Klett presents Extended Mind Transformers (EMT), a modification to transformer attention that lets models retrieve relevant memory tokens during generation without fine-tuning. EMT uses relative position embeddings (RoPE in LLaMA, ALiBi in MPT) to enable zero-shot generalization. On a counterfactual retrieval benchmark up to 16K tokens, EMT outperforms fine-tuned models and, combined with RAG, surpasses GPT-4. It provides granular citations by showing exactly which memory tokens were attended to, and reduces hallucinations via active learning: when token-level entropy signals uncertainty, the model retrieves more memories. EMT is open-sourced on Hugging Face and GitHub with configurable parameters like stride length, top K, similarity masking, and unknown token elimination.

GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem

Aug 28, 2024 · 19:15

Emil Eifrem, Neo4j co-founder and CEO, argues that GraphRAG—combining knowledge graphs with vector search—significantly improves RAG application accuracy, ease of development, and explainability. Citing studies, he reports accuracy gains of 3x (Data.org), 75-77% (LinkedIn), and Microsoft's finding that GraphRAG enables answering entirely new question types. He demonstrates Neo4j's Knowledge Graph Builder, which auto-generates graphs from PDFs, Wikipedia, and YouTube, making graph creation accessible. Eifrem frames this as the next evolution in search after PageRank and Google's knowledge graph, urging developers to adopt GraphRAG for richer context and better AI outcomes.

Building State of the Art Open Weights Tool Use: The Command R Family: Sandra Kublik

Aug 26, 2024 · 15:03

Sandra Kublik of Cohere presents the Command R family of open-weight models optimized for retrieval-augmented generation and tool use. Released in March 2024, Command R and Command R+ achieved 150,000 Hugging Face downloads within two weeks and now serve nearly 500,000 developers. The models overcome RAG challenges such as prompt sensitivity and citation accuracy through post-training, delivering fine-grained citations and low hallucination. Cohere open-sourced a toolkit UI with plug-and-play components for RAG and tool use, supporting cloud, local, and Hugging Face access. The new Multi-Step API enables sequential reasoning with automatic retry and reflection. Command R+ matches GPT-4 Turbo and Claude Opus on complex reasoning while being three to five times cheaper, positioning it as a scalable enterprise solution.

Git push get an AI API: Ryan Fox-Tyler

Aug 23, 2024 · 45:01

Ryan Fox-Tyler and Matt Johnson Pint from Hypermode present a hands-on workshop demonstrating how to build and iteratively improve AI features using their platform, focusing on a GitHub issue triage app. They first illustrate the process with a multiplayer game (Hypercategories) that uses AI for classification and scoring. Then they build a trend summary function using OpenAI GPT-4 to summarize repository issues, and a classify issue function with a Hugging Face DistilBERT model for labeling issues as bug, feature, or question. Finally, they add natural language search for similar issues by creating an embeddings model (MiniLM) and a Hypermode collection, enabling vector search without external databases. The workshop emphasizes incremental iteration, mixing AI models with traditional code, and using Hypermode's automatic GraphQL generation and observability to speed development.

Hypermode Launch: Kevin Van Gundy

Aug 23, 2024 · 5:03

Kevin Van Gundy, founder of Hypermode, argues that iteration velocity is the compound interest of software and key to succeeding with AI, and presents Hypermode as a runtime that makes AI approachable by integrating models and data into AI functions with minimal friction. Drawing from his experience at Vercel, he explains how rapid iteration helped them win against competitors, and applies the same philosophy to AI development. Hypermode removes common pain points: it offers one-request RAG with in-memory embedding and search, model comparison and fine-tuning export, and strong defaults that work with existing stacks. The platform de-risks getting it wrong by making model switching and prompt changes frictionless. Kevin invites listeners to a workshop to build a demo and offers $1000 in Hypermode credits to get started.

Disrupting the $15 Trillion Construction Industry with Autonomous Agents: Dr. Sarah Buchner

Aug 22, 2024 · 5:32

Dr. Sarah Buchner, Founder & CEO of Trunk Tools, argues that vertical AI agents are the future, targeting the $15 trillion construction industry where 10% rework costs $1.5 trillion annually. Her company deploys an agent every 45 days, starting with TrunkText, a RAG-powered Q&A tool that saves field professionals 1-2 hours daily by accessing up to 3.6 million pages of documentation per skyscraper. The episode reveals how their system uncovers costly data discrepancies—such as a door needing power-actuated hardware that contradicts specs—and automatically generates RFIs to resolve them. Buchner insists RAG is commoditized, but keeping humans at the center with an army of agents solving real-world problems is the true impact. The talk highlights Trunk Tools' brain behind construction, backed by Sequoia, Accel, and others, and notes the company is hiring as it scales.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.

How Codeium Breaks Through the Ceiling for Retrieval: Kevin Hou

Jul 31, 2024 · 18:42

Kevin Hou, AI engineer at Codeium, argues that embedding-based retrieval is hitting a ceiling for AI code generation and that Codeium breaks through by throwing large language models directly at the problem via a system called MQuery. He outlines the limitations of long context, fine-tuning, and embeddings, noting that embeddings are cheap but fail to reason over multiple documents needed for real-world coding tasks. Codeium developed a product-driven benchmark using recall 50 and commit-message datasets to measure retrieval quality. Because Codeium is vertically integrated—training custom models and building its own infrastructure—it can afford to run thousands of LLM calls in parallel per query, giving users 100x more compute than competitors. MQuery retrieves context from thousands of files in seconds, resulting in more accepted completions and better chat satisfaction. Hou compares this shift to autonomous driving: early heuristics like embeddings are giving way to large models that handle more data directly.

The Future of Knowledge Assistants: Jerry Liu

Jul 13, 2024 · 16:55

Jerry Liu, CEO of LlamaIndex, explains how to move beyond simple RAG to build production-grade knowledge assistants. He details three steps: advanced data processing with LlamaParse for accurate PDF parsing, single-agent flows with query planning and tool use, and multi-agent task solvers via the newly announced Llama Agents framework. Llama Agents treats each agent as a deployable microservice that communicates through a central API, enabling specialization, parallelism, and easier production deployment. Jerry also highlights that naive RAG is insufficient for complex queries, and that good data quality—like proper parsing of tables and charts—is essential to reduce hallucinations.

Building Production-Ready RAG Applications: Jerry Liu

Nov 15, 2023 · 18:35

Jerry Liu, CEO of LlamaIndex, explains how to productionize Retrieval Augmented Generation (RAG) systems by moving beyond naive implementations. He identifies key challenges: low retrieval precision causing hallucination, low recall from insufficient top-K, and lost-in-the-middle problems. Liu advocates starting with 'table stakes' improvements like tuning chunk sizes (showing optimal values per dataset), adding metadata filters (e.g., year=2021 for SEC 10Q queries), and hybrid search. More advanced techniques include 'small-to-big retrieval', embedding smaller chunks for precision then expanding windows for synthesis, and using reranking to improve recall. Finally, he explores agent architectures where each document becomes a tool for summarization or QA, and fine-tuning—generating synthetic query datasets from raw text to fine-tune embeddings, or distilling GPT-4's chain-of-thought into GPT-3.5 Turbo for better reasoning.

Retrieval Augmented Generation in the Wild: Anton Troynikov

Nov 15, 2023 · 12:20

Anton Troynikov, co-founder of Chroma, explains that retrieval augmented generation (RAG) requires more than simple vector search—it needs human feedback, self-updating memory, and agent interaction to handle dynamic data. He covers challenges like choosing the right embedding model, chunking strategies (including using language model perplexity), and determining result relevance without distractors. Chroma is building a horizontally scalable cluster, a cloud technical preview by December, and support for multimodal data. The episode argues that a capable memory system is key to making AI agents truly functional, citing the Voyager paper where Chroma stored learned skills for Minecraft agents.

Building Reactive AI Apps: Matt Welsh

Nov 9, 2023 · 17:02

Matt Welsh announces AI.JSX, an open-source TypeScript framework he describes as 'React for LLMs,' which lets developers build reactive AI applications by composing JSX components that render to an LLM rather than the DOM. He argues AI.JSX makes LLM app development accessible to JavaScript developers, not just Python back-end engineers, by handling RAG, tool invocation, and real-time voice interaction out of the box. Welsh demonstrates parallel streaming of multiple LLM calls for story generation, a kid-safe wrapper component that rewrites content, and a 10-line RAG implementation. He showcases a live voice demo where an AI takes a donut order with sub-second latency, powered by Fixie's cloud platform for hosting, managed RAG pipelines, and conversational state. He emphasizes the framework's full React integration and ability to generate UI from AI, positioning AI.JSX as a simplification for building sophisticated LLM-powered apps.

The Hidden Life of Embeddings: Linus Lee

Nov 7, 2023 · 18:15

Linus Lee, a Research Engineer at Notion, presents a tour of embedding visualization and manipulation at the AI Engineer Summit 2023. He demonstrates an encoder-decoder model fine-tuned from T5 that can reconstruct text from embeddings, allowing direct manipulation of features like length and sentiment by moving in latent space. Lee shows that mixing embeddings by splicing dimensions from two texts produces a semantic blend, and a linear adapter can decode text from OpenAI's text-embedding-ada-002 embedding space. Using CLIP, he interpolates between photographic and cartoon images and performs vector arithmetic to modify facial expressions. Lee releases these custom text embedding models on Hugging Face, enabling others to explore and interact with latent spaces. He argues that making model internals visible and manipulable fosters deeper understanding and more humane interfaces to generative AI.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Supabase Vector: The Postgres Vector database: Paul Copplestone

Nov 3, 2023 · 16:05

Paul Copplestone, CEO of Supabase, makes the case for pgvector as an embedded vector database within Postgres, arguing it offers production-grade performance and unique advantages for AI applications. He recounts how pgvector was contributed by a single developer, Andrew Kane, and how Supabase integrated it, leading to 12,000 new databases launched weekly with 10-15% using pgvector. Addressing benchmarks that claimed pgvector was 20x slower, Copplestone shows that adding HNSW indexing brought accuracy to 0.99, matching specialized vector databases, and a cost comparison vs. Pinecone ($410 vs. $480 for comparable queries/sec) demonstrates Postgres competitiveness. He demonstrates Postgres's extensibility with a cat image filtering example using partitions and triggers to separate good cats (similarity >0.8) from bad, all in 14 lines of SQL. Finally, he outlines future work on sharding with Citus to handle billions of vectors, and invites design partners for enterprise use cases.

Building Blocks for LLM Systems & Products: Eugene Yan

Nov 2, 2023 · 17:24

Eugene Yan presents practical patterns for building LLM systems and products, emphasizing evals as the foundation. He argues that automated evals—starting with as few as 40 domain-specific questions—enable faster iteration and safer deployment, while academic benchmarks like MMLU may not fit real tasks. For retrieval-augmented generation, he warns that LLMs perform worse when the answer is in the middle of retrieved documents, and even with perfect retrieval, accuracy tops out at 75%. Guardrails against hallucination can be built via natural language inference at the sentence level or by sampling multiple summaries to check consistency. Finally, he highlights UX design—like Copilot’s accept/reject or Midjourney’s upscale/vary—as a way to collect implicit feedback that builds a data flywheel for evals and fine-tuning.

Building Context-Aware Reasoning Applications with LangChain and LangSmith: Harrison Chase

Nov 1, 2023 · 18:54

Harrison Chase, CEO of LangChain, argues that building context-aware reasoning applications on LLMs requires treating the model as part of a larger system, and he outlines key approaches and challenges. He categorizes context provision as instruction prompting, few-shot examples, retrieval-augmented generation (RAG), and fine-tuning. For reasoning, he describes a spectrum from single LLM calls to chains, routers, agents with cycles, and autonomous agents like Auto-GPT. Chase then details engineering challenges: choosing the right cognitive architecture, data engineering for context, prompt engineering across multi-step systems, evaluation using LLM-assisted metrics and user feedback, and enabling collaboration between technical and non-technical team members. He emphasizes that the field remains early and that LangChain and LangSmith help prototype, debug, and iterate on these systems.