LangChain

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Aug 17, 2026 · 1:03:26

Louis-François Bouchard, Omar Solano, and Samridhi Vaid of Towards AI test context engineering on their AI tutor: keeping full history beat every compaction technique on recall, cost, and latency because 97% of tokens were served from cache (up to 50x cheaper), making summarization a trap unless it shrinks context by more than 50x. Full history recovered specific details 95% vs 32% after summarizing, and distinctive facts survived 800k tokens. Local hardware changes it: a 32k window can't keep everything, and larger models don't widen context. Dense retrieval hit 0% recall at 400k tokens where BM25 got 100%, so they use hybrid retrieval. Rule: name the constraint before compacting.

Beyond the Harness: A Journey Towards Adaptative Engineering - Rajiv Chandegra, Annicha Labs

Jul 7, 2026 · 37:01

Rajiv Chandegra of Annicha Labs introduces adaptive engineering as a design philosophy that moves beyond fixed AI harnesses (like Claude Code, Cursor, or Py) to let multi-agent systems self-organize during runtime. He argues that as models become exponentially more powerful and AI collides with the messy, dynamic real world, pre-engineered static harnesses become brittle. Drawing on complexity science, he contrasts the factory model—reliable but novelty-suppressing—with adaptive engineering, where agents interact locally and the harness emerges as an ongoing output, not a predetermined input. The engineer's role shifts to designing constraints (e.g., rate of coupling, goal reward) and sensing the emergent order, while failures include drift, monoculture, and legibility collapse. This approach targets 'horizontal intelligence'—decentralized coordination among agents—as the key to handling complex, real-world problems that cannot be decomposed into fixed parts.

User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch

Jun 28, 2026 · 15:37

Sonam Pankaj, CEO of StarlightSearch, argues that production agents fail because retrieval is static and eval signals never cross into runtime context. She introduces utility-ranked memory, where memories are re-ranked by a utility score combining semantic similarity with outcome history—passing runs raise a memory's score, failing runs lower it. In a product SQL agent demo, a search for 'gaming mouse' initially failed, but after marking that output as a failure and noting a wireless mouse was relevant, the agent's trajectory updated in real time to find the correct product. Benchmarks on Tao Bench show reflect memory improves performance from 66% to 76% without skills and 80% with skills; on agentic tasks, reflect achieves 61.3% versus 35.7% baseline and 58.2% with other memory systems. Pankaj explains that after enough reviews, memories can be baked into skills, allowing continuous improvement without manual prompt rewriting or model fine-tuning.

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Jun 18, 2026 · 37:06

Sandipan Bhaumik, a technical lead for Data and AI at Databricks, presents a five‑pillar playbook for taking AI agents to production: evaluation (define numerical success before touching code), observability (trace every decision for regulators and debugging), data foundation (agents do not forgive bad data), multi‑agent orchestration patterns (orchestrator‑worker, choreography, human‑in‑the‑loop), and governance (PII pre‑validation, prompt versioning as change management). He recounts a retail bank that spent £85,000 over six months on a chatbot PoC that failed because no one could measure or trace it. His team reversed the order: they built the evaluation dataset and tracing infrastructure first, selected the model in week 7 of an 8‑week engagement, and launched successfully. Six weeks post‑launch, when the bank updated interest rate policies, the tracing system caught that the new document had not been re‑embedded, so the agent served stale answers—a production incident the five pillars were designed to handle. The evaluation dataset is a living system that grows from 200 test cases; a production incident playbook connects all pillars: detect via eval dashboard, diagnose with…

Why More Context Makes Your Agent Dumber and What to Do About It — Nupur Sharma, Qodo

Jun 8, 2026 · 26:27

Nupur Sharma from Qodo explains why giving an AI agent more context often makes it dumber, describing the 'U curve' where models attend only to the start and end of inputs while dropping the middle. She covers practical fixes: iterative retrieval, hierarchical summarization, and self-correction with honest cost tradeoffs. The talk introduces the 'orchestration paradox'—smart models waste most tokens figuring out how to solve a problem rather than solving it—and Qodo's 80/20 hybrid: high-reasoning models for open-ended discovery, lighter deterministic models for validation. Sharma walks through Qodo's code review architecture: a context collector feeds specialized agents (security, code quality, etc.), a judge node recombines results against PR history, and every accepted or rejected suggestion shifts weighting for future reviews.

Don't Build Slop (4 Levels of AI Agent Maturity) - Ara Khan, Cline

May 19, 2026 · 18:52

Ara Khan of Cline presents four levels of AI agent maturity, arguing that most builders suffer from mass psychosis and should focus on thoughtful architecture. Level one is using existing frameworks for rapid prototyping. Level two introduces five rules: treat every agent as a state machine, keep system prompts minimal (e.g., GPT-5.3's prompt is one-third the size of GPT-5's to avoid sensory overload), integrate with a CLI for pseudo-RL pipelines, avoid building slop by designing architecture manually, and master frontier model APIs to leverage reasoning traces correctly. Level three advocates Kanban boards as the ideal UX form factor for managing parallel inference-bound agents, a prediction he made on March 26 that Claude Code shipped ten hours before this talk. Level four involves shipping agents to the cloud for scalability, enabling long-running tasks and parallel execution without local dependencies.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft

May 14, 2026 · 1:20:07

Amy Boyd and Nitya Narasimhan of Microsoft explain how to close the gap between agent behavior and requirements using Microsoft Foundry's observability stack. They demonstrate tracing via OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics (e.g., intent resolution, task adherence), and red teaming where a second AI attacks the agent to reveal vulnerabilities. The showcase is the observe skill: pointed at an agent with no eval data, it generates a dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one—all from a single prompt. The skill surfaces failures developers didn't know existed, accelerating the optimize loop with human-in-the-loop guidance.

Agentic Search for Context Engineering — Leonie Monigatti, Elastic

May 8, 2026 · 1:03:13

Leonie Monigatti from Elastic argues that context engineering is 80% agentic search—the search tool that decides what to pull from files, databases, memory, and the web. She identifies three failure modes: the agent not calling any tool, calling the wrong tool, or generating incorrect parameters, and shows how detailed tool descriptions and agent skills (progressive disclosure) reduce these. In demos, semantic search fails for keyword 'JEPA' due to embedding similarity; a general-purpose ESQL tool with an agent skill correctly queries the database. Shell/bash tool enables file system search but requires iterative grep, while the custom Gina Grap CLI provides semantic grep for fuzzy queries. Practical recommendations: start with general-purpose tools, log agent behavior, and add specialized tools for frequent queries to balance low floor (easy successes) and high ceiling (complex queries). Hybrid agents combining shell and database tools achieve higher accuracy by verifying results.

Scaling GitHub for your Agents — Sam Morrow, GitHub

Apr 27, 2026 · 20:35

Sam Morrow, GitHub's MCP server lead, details the architectural challenges and solutions for scaling a remote MCP server to 7 million weekly tool calls. He explains how tool proliferation degraded agent performance—LangChain's research confirmed more tools confuse agents—leading to innovations like tool sets and dynamic discovery, though 100% of users stuck with defaults. To reduce context, GitHub cut tool descriptions by 49% and trimmed output tokens by 75% on list pull requests. Security is addressed via OAuth 2.1 with PKCE and step-up auth; they rejected dynamic client registration to avoid unbounded app databases and rate-limiting issues. The stateless server uses Redis for session storage and builds a fresh server instance per request, enabling horizontal scaling without session affinity. Metrics include 11 million Docker downloads, 30,000 stars, 4,000 forks, and 126 contributors. Morrow predicts compositional tools and automatic server discovery will make thousands of tools the norm.

The Future of MCP — David Soria Parra, Anthropic

Apr 19, 2026 · 18:46

David Soria Parra from Anthropic argues that MCP (Model Context Protocol) is the key to connecting agents to tools and data in production, with 110 million monthly downloads—outpacing React's growth at the same stage. He lays out a 2026 connectivity stack combining Skills, MCP, and CLI/Computer Use, each suited for different needs, and emphasizes that best agents will use all three seamlessly. To improve client harnesses, he introduces Progressive Discovery—deferring tool loading via Tool Search to reduce context usage—and Programmatic Tool Calling, where models write scripts to compose tool outputs efficiently. Upcoming MCP protocol improvements include stateless transport (with Google) for easier scaling, async agent-to-agent tasks, enterprise features like Cross App Access and Server Discovery via well-known URLs, and a Skills-over-MCP extension for shipping usage instructions with servers. He calls for community feedback on these directions.

One Registry to Rule them All - Sonny Merla, Mauro Luchetti, & Mattia Redaelli, Quantyca

Apr 10, 2026 · 22:47

Amplifon's AI transformation led to the Amplify program, for which Quantyca built an enterprise-grade registry system for MCP servers and A2A agents. Sonny Merla, Mauro Luchetti, and Mattia Redaelli explain how three registries—MCP, A2A, and use case—are linked via a catalog to provide full lineage, including ownership, environment, authentication, cost attribution, and use case linkage. The solution uses an AI gateway for unified LLM access with Entra ID authentication and budgeting, and provides template repositories on GitHub with CI/CD pipelines that automatically publish agent cards and server metadata to the registries. This enables discovery, governance, and impact analysis across 26 countries and multiple teams, letting developers focus on business logic while avoiding reinventing security and deployment infrastructure.

Bending a Public MCP Server Without Breaking It — Nimrod Hauser, Baz

Apr 8, 2026 · 40:50

Nimrod Hauser, founding engineer at Baz, presents a hands-on guide to adapting third-party MCP servers for production, using Playwright's MCP server as an example. He demonstrates five best practices—curation, wrapping descriptions, deterministic guardrails, composing new tools, and treating tools as deterministic functions—to transform a brittle setup into resilient infrastructure. Hauser shows how curating tools from 21 to 16 reduces context load, while enhanced descriptions guide agents to use accessibility snapshots before clicking. Deterministic path validation prevents agents from saving screenshots outside a designated folder, and a new 'evidence tool' with tailored instructions improves spec review accuracy. The talk culminates with Baz's Spec Reviewer, an agentic code review tool that compares requirements against implementation, successfully passing a test after applying all optimizations.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Wisdom-Driven Knowledge Augmented Generation at Scale - Chin Keong Lam, Patho AI

Aug 22, 2025 · 18:43

Chin Keong Lam, founder and CEO of Patho.ai, argues that Wisdom-Driven Knowledge Graphs (KAG) significantly outperform traditional RAG systems for complex quantitative analysis and expert-level advice. He presents a wisdom-decision-situation feedback loop where knowledge, experience, and insight feed into wisdom, mapped to a practical competitive analysis chatbot for a marketing client. The system uses N8n and a multi-agent architecture with a supervisory Wisdom Agent overseeing insight, knowledge, and strategy agents, all updating a centralized knowledge graph. Lam cites five reasons knowledge graphs beat RAG: capturing complex relationships, improved accuracy, scalability, rich multi-hop query capability, and seamless data integration. He demonstrates how vector RAG fails on numerical reasoning (e.g., Apple revenue query) while KAG returns precise evidence-based answers. Lam recommends a hybrid graph extraction combining LLM extraction with expert pruning, and reports benchmark results of 91% accuracy, 85% flexibility, and strong traceability and scalability.

Agents vs Workflows: Why Not Both? — Sam Bhagwat, Mastra.ai

Aug 1, 2025 · 15:37

Sam Bhagwat, co-founder of Mastra and author of 'Principles of AI Agents', argues that the debate between agents and workflows is misguided—developers should combine both. He criticizes OpenAI's anti-workflow stance as 'being that guy' and LangChain's graph/node/edge APIs as harmful, advocating for fluent syntax over graph theory. Bhagwat defines agents as turn-based games and workflows as rules-engine dependency chains, noting that nondeterminism makes workflow tracing 10x more important in AI engineering. He demonstrates composition patterns: agents can be steps, workflows can be tools, and nested workflows handle dynamic tool injection. He recommends starting with agent power, then adding workflow control for reliability—e.g., breaking one LLM call into 12 for medical PDFs. Bhagwat concludes that practice trumps theory in this young field.

Human seeded Evals — Samuel Colvin, Pydantic

Jul 25, 2025 · 12:02

Samuel Colvin, creator of Pydantic, demonstrates how type safety and validation loops in Pydantic AI build reliable GenAI applications. He argues that type-safe frameworks enable confident refactoring, a necessity for evolving AI apps, and shows how returning validation errors to the model fixes simple mistakes like incorrect date formats. The agentic loop with tools and final result tools ensures structured extraction, while type-safe dependencies on tools prevent runtime errors. Colvin also highlights Logfire observability, which traces calls, shows pricing, and helps debug failures like failed memory retrieval. The talk covers agent definitions, validation error retries, and the importance of explicit exit conditions in agent loops.

Rise of the AI Architect — Clay Bavor, Cofounder, Sierra w/ Alessio Fanelli

Jul 24, 2025 · 18:55

Clay Bavor, cofounder of Sierra, and Alessio Fanelli discuss the rise of the AI Architect—a new role combining technology, brand, and business outcomes to build customer-facing AI agents. Sierra serves hundreds of millions of consumers this year. Bavor defines the AI Architect as wearing three hats: understanding AI capabilities, defining the agent's voice (e.g., Chubbies' irreverent Duncan Smothers), and driving business outcomes. Successful AI Architects embrace risk, start with narrow problems like processing a single return, and re-architect teams to coach the AI. On build vs. buy, Bavor warns of the "agent iceberg"—hundreds of hidden complexities like regression testing and model migration. He advises tracking model improvement in a Google Doc and anticipating future capabilities, predicting glasses as the ultimate interface for trusted personal AI.

How to Build Planning Agents without losing control - Yogendra Miraje, Factset

Jul 23, 2025 · 15:58

Yogendra Miraje from FactSet explains how to build controllable planning agents using agentic workflows and planning by subgoal division, arguing they balance reliability and flexibility for enterprises. He distinguishes workflow agents (static) from agentic workflows (dynamic), and introduces a blueprint generator that pre-selects tools and provides high-level natural language plans to reduce planner cognitive load. He details an architecture using LangGraph nodes (blueprint generator, planner, executor, joiner) and emphasizes designing tools from an agent's perspective with clear purpose, description, and contracts. Miraje stresses evals using code-based and LLM-as-judge techniques, and advises against agentic workflows for fixed tasks, strict compliance, or low-latency needs. The talk includes a concrete example of preparing for NVIDIA's earnings call, showing how blueprints enable structured responses and interpretability.

Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Jul 21, 2025 · 15:27

Stephen Chin of Neo4j argues that Agentic GraphRAG — combining graph databases with retrieval-augmented generation — overcomes LLM hallucinations and biases by providing structured, relational context. He demonstrates how LLMs fail on reasoning tasks like calculating classroom capacity due to inaccurate anchoring on irrelevant data, and proposes an architecture where vector search first identifies relevant nodes, then graph traversal retrieves related context for the LLM. Chin highlights Neo4j’s MCP server for cypher query generation and memory modules, and cites Klarna’s success: 250K employee questions answered in the first year, 2,000 daily queries, and 85% adoption, replacing their entire SaaS stack. He recommends the Neo4j Certified Developer Program and the Nodes Conference for further learning.

Full Spec MCP: Hidden Capabilities of the MCP spec — Harald Kirschner, Microsoft/VSCode

Jul 18, 2025 · 14:53

Harald Kirschner from Microsoft/VSCode argues that MCP's full specification unlocks powerful stateful interactions beyond the common 'tools-only' implementations, transforming AI assistants into more contextual and efficient agents. He highlights underused primitives like resources for rich data context and sampling for server-requested LLM completions, demonstrated via a dungeon game where dynamic tool discovery adapts to game state. VS Code's upcoming full spec support includes dynamic tool discovery, user-defined tool sets, a debug mode for server development, and support for streamable HTTP to reduce stateful server churn. Upcoming features like elicitations will allow tools to request user input directly. Kirschner calls on developers to build progressive, full-spec servers and contribute feedback to the open ecosystem, emphasizing that client and SDK support will follow as usage grows.

Intro to GraphRAG — Zach Blumenfeld

Jun 30, 2025 · 1:18:35

Zach Blumenfeld introduces GraphRAG using Neo4j, showing how to build knowledge graphs from employee and skill data, combine structured and unstructured data, and use graph traversal with vector search for retrieval. The workshop covers Cypher query patterns for multi-hop similarity, entity extraction from resumes via LLMs, and creating semantic similarity relationships. It demonstrates Leiden community detection for skill clustering and builds a LangGraph agent with four custom tools that balance exact skill matches with vector similarity. Blumenfeld argues that knowledge graphs provide controlled, explainable retrieval logic for agentic workflows, enabling developers to decompose data into graph models that expose domain logic for more accurate retrieval than pure vector search.

Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier

Jun 30, 2025 · 16:15

Zapier AI Tech Lead Rafal Willinski and Staff Engineer Vitor Balocco explain how Zapier's evaluation system turns agent failures into targeted improvements through a data flywheel. They detail collecting explicit feedback at critical moments and mining implicit signals like testing behavior, cursing, and user follow-ups. The pair advocates building unit test evals for specific failure modes, then trajectory evals and LLM-as-judge with rubrics to avoid overfitting and capture multi-turn criteria. They share that over-indexing on unit tests hurt model benchmarking, and reasoning models can compare model runs, revealing differences like Claude as a decisive executor versus Gemini's yapping. Ultimately, they argue that the goal is user satisfaction, so A/B testing on a small traffic fraction is the ultimate verification.

Why Your Agent’s Brain Needs a Playbook: Practical Wins from Using Ontologies - Jesús Barrasa, Neo4j

Jun 27, 2025 · 13:54

Jesús Barrasa, AI Field CTO at Neo4j, argues that ontologies are the secret weapon for building robust GraphRAG applications, providing both a formal, implementation-agnostic schema for knowledge graph creation and a dynamic data layer that drives retriever behavior. He explains how ontologies replace rigid code-orchestrated workflows and chaotic LLM-generated ones by guiding entity extraction from unstructured data and mapping structured sources into a property graph. During retrieval, storing the ontology in the graph enables dynamic Cypher queries that automatically navigate contextualizing relationships — such as "acted-in" rather than "directed" — based on annotated subproperties, without hard-coded logic. This allows developers to change retrieval behavior on the fly by modifying the ontology as a data artifact. The talk covers two practical wins: using ontologies as a shared domain model for graph construction, and leveraging them to build adaptive, schema-driven retrievers that improve result completeness and relevance.

Graph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics - Alison & Andreas, Neo4j

Jun 27, 2025 · 1:41:16

Alison Cossette and Andreas Kollegger from Neo4j demonstrate how graph data science algorithms enhance GraphRAG systems by connecting, clustering, and curating unstructured data. They walk through running KNN similarity (K=25) on document embeddings, then applying Louvain community detection to identify clusters of near-identical chunks—one community had 49 documents with 0.98 average similarity—which can be collapsed via APOC to improve retrieval efficiency while preserving lineage. Using page rank and betweenness centrality, they show how to surface influential documents and track conversation paths across communities. The session uses a preloaded Neo4j AuraDB database with 17K nodes and 774K relationships from the Agent Neo project, and includes practical cipher queries for projecting graphs and running algorithms. They emphasize using graph analytics to diversify retrieval, manage content hygiene, and build accountable agent systems.

Agentic GraphRAG: Simplifying Retrieval Across Structured & Unstructured Data — Zach Blumenfeld

Jun 27, 2025 · 15:25

Zach Blumenfeld (Neo4j) demonstrates how a knowledge graph simplifies agentic retrieval across structured and unstructured data, using an employee skill graph as an example. He shows that a plain document search fails to answer questions like "how many Python developers" (returns 5 instead of 28) or "who is most similar to Lucas Martinez" — but after extracting entities into a Neo4j graph, the agent can run precise Cypher queries via an MCP server, correctly reporting 28 Python devs and identifying Sarah as the closest match via skill overlap. He also shows how to flexibly add new relationships (e.g., project collaborators from an HR system) without schema changes, enabling questions like "which individuals collaborated most on AI projects" to return exact pairings like Sarah and Amanda. The talk emphasizes accuracy, explainability, and the ability to start with a simple data model and expand as agents ingest more data.

Building Agentic Applications w/ Heroku Managed Inference and Agents — Julián Duque & Anush Dsouza

Jun 27, 2025 · 52:35

In this workshop, Heroku Principal Developer Advocate Julián Duque and Product Manager Anush Dsouza introduce Heroku Managed Inference and Agents, a platform designed to make every software engineer an AI engineer by simplifying the attachment of agents and AI to applications. They demonstrate an opinionated, curated set of models (e.g., Claude 4) and an agentic control loop running on Heroku's trusted compute (Dynos) that provides first-party tools like code execution (Python, Node, Go, Ruby), Postgres schema inspection and querying, and document conversion, all streaming responses in real time. The workshop walks through provisioning inference via a single CLI command or add-on, using a Jupyter notebook to chain multiple tools (e.g., HTML-to-markdown then Python execution), and attaching custom MCP servers (like Brave Search) that spin up one-off Dynos and scale to zero. They also show how to expose MCPs remotely via a server-sent events endpoint for use with Cursor or other clients, emphasizing security (read-only database followers, bearer tokens) and future OAuth support. The episode concludes with a call to try the platform via a free trial team valid through the weekend.

Case Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP - Dan Mason

Jun 22, 2025 · 1:56:13

Dan Mason of Stride presents a case study on building autonomous agents for telemedicine support, replacing a human-driven button-pushing workflow with LangGraph, Claude, MCP, and a Node.js/React/MongoDB stack, achieving roughly 10X capacity increase. The LLM-driven virtual operations associate ("Ava") assesses patient messages, updates state with anchors and scheduled messages, and passes proposals to an evaluator agent that scores confidence and complexity, escalating to humans below 75% confidence. Mason explains how treatment "blueprints" in Google Docs are read directly by the LLM instead of using RAG, enabling new treatments to be added without writing code. He details the eval system using LLM-as-a-judge via PromptFu and retry logic for tool-calling errors, and discusses trade-offs around confidence scoring, prompt caching, and model selection (Claude for steerability). The team includes two software engineers, one designer, and Mason, who wrote most LangGraph code with AI assistance (Kline).

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

Cognitive Shield Real Time Real Smart - Rachna Srivastava

Jun 3, 2025 · 44:58

Rachna Srivastava presents Cognitive Shield, a three-layer AI defense system against sophisticated financial fraud such as deepfake voice cloning, synthetic identities, and AI-powered crypto scams that have surged 375% since 2023. Layer one secures user data and uses AI to guide licensing and examination processes. Layer two employs eight detection modules including GAN-based deepfake detection, graph neural networks for fraud ring visualization, and NLP for phishing, all integrated via CrewAI multi-agent orchestration. Layer three provides a unified intelligence console with natural language search, real-time dashboards, and automated case escalation with compliance-ready reporting. The system leverages Neo4j graph databases and Graph RAG to uncover hidden connections, and is built with Streamlit, FastAPI, and Postgres. Srivastava warns that by 2027, 90% of cyber attacks will be AI-driven and fraud losses will surpass $100 billion annually.

open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Jun 3, 2025 · 5:03

Ofer Mendelevitch from Vectara presents Open-RAG-Eval, an open-source framework that enables RAG evaluation without requiring golden answers or golden chunks, solving a major scalability problem. Backed by research with the University of Waterloo's Jimmy Lin Lab, it uses UMBRELA for retrieval scoring on a 0–3 scale that correlates well with human judgment, and AutoNuggetizer for generation via nugget creation, Vital/OK ratings, and an LLM judge analyzing the top 20 nuggets for support. Additional metrics include citation faithfulness and Vectara's HHEM hallucination detection model. Connectors are available for Vectara, LangChain, and LlamaIndex, with results viewable through an intuitive UI at OpenEvaluation.ai.

Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Apr 19, 2025 · 1:19:34

Charles Packer, lead author of the MemGPT paper and co-founder of Letta, argues that statefulness (memory) is the most important problem to solve for building useful AI agents, since LLMs are inherently stateless transformers. He presents MemGPT's LMOS (Language Model Operating System) approach, which treats memory management as a context compilation problem solved by the LLM itself using tool calling to read and write structured memory blocks. The workshop demonstrates Letta's open-source stack (FastAPI, Postgres, Python) where agents persist state on a server, enabling long-running, learning interactions without context overflow—e.g., an agent can update its core memory (e.g., correcting a user's name) or search archival memory (e.g., recalling user preferences after a reset). Packer also shows multi-agent communication via async message passing between agents running as independent services, and highlights that tools are sandboxed by default with support for Composio integrations. The session includes a live notebook exercise and a low-code UI (ADE) showing context window management and memory editing, emphasizing that true stateful agents improve continuously over time rather…

Building LinkedIn's GenAI Platform — Xiaofeng Wang

Apr 16, 2025 · 17:53

Xiaofeng Wang, manager of LinkedIn's GenAI Foundation, explains the evolution of LinkedIn's GenAI platform from simple prompt-in string-out applications to a multi-agent system for LinkedIn Hire Assistant, arguing that a unified platform is critical for bridging the gap between AI and product engineers in the era of compound AI systems. He details the platform's four-layer architecture—orchestration, prompt engineering, tools/skills invocation, and memory management—and key investments like a Python SDK, centralized skill registry, experiential memory across working, long-term, and collective layers, and observability built on OpenTelemetry. Wang discusses hiring philosophy: prioritize strong software engineers over AI expertise, hire for potential, and build diverse teams integrating full-stack engineers, data scientists, and AI engineers. He recommends solving immediate needs first, leveraging existing scalable infrastructure like messaging systems for memory, and focusing on developer experience to drive adoption.

Your Evals Are Meaningless (And Here’s How to Fix Them)

Feb 22, 2025 · 18:50

In this AI Engineering Summit talk, HoneyHive co-founder exposes why most LLM evaluations are meaningless due to criteria drift—where evaluator criteria misalign with user needs—and dataset drift, where test cases don't reflect real-world queries. He argues that static evaluation frameworks from tools like LangChain or Ffragas fail because they measure generic metrics rather than business-specific relevance, citing an e-commerce recommendation system that looked perfect in testing but broke in production. The fix is a three-step iterative alignment process: align evaluators with domain experts via continuous critique and few-shot examples; keep datasets alive by logging production underperformance and flowing those cases back into the test bank; and track alignment over time using F1 scores for binary judgments. Practical advice includes customizing the LLM evaluator prompt, starting with 20 domain expert examples in spreadsheets, and avoiding templated metrics. The speaker emphasizes that evals must evolve continuously, just like the LLM application itself, or they become meaningless.

OpenLLMetry is all you need

Feb 22, 2025 · 9:12

Nir, CEO of Trace Loop, introduces OpenLLMetry, an open-source project extending OpenTelemetry for tracing and monitoring GenAI applications. OpenTelemetry, maintained by CNCF, standardizes logging, metrics, and traces across cloud environments, supported by platforms like Datadog, New Relic, and Grafana. OpenLLMetry provides over 40 automatic instrumentations for foundation models (OpenAI, Anthropic, Cohere), vector databases (Pinecone, Chroma), and frameworks (LangChain, LlamaIndex, CrewAI). These instrumentations emit logs, metrics, and traces in OpenTelemetry format, allowing users to send data to any supported observability backend with a configuration change, avoiding vendor lock-in.

How to Improve Your Agents: Academic Lit Review

Feb 22, 2025 · 39:02

Joe from Columbia University and founder of Arklex AI explains research on AI agents, focusing on improving their reasoning and planning through self-reflection, test-time compute, and tree search methods, without relying on human supervision. He details the TriPath method, which uses larger models to edit smaller models' feedback for better self-improvement, achieving up to 48% accuracy on math benchmarks. He shows how multi-color tree search (MCTS) with contrastive reflection and multi-agent debate (RMCTS) outperforms other search methods on Visual Web Arena and OS World, achieving top non-trained results. He proposes exploratory learning, where models learn from search trajectories rather than optimal actions, improving performance under compute budgets. Finally, he presents the Arklex open-source agent framework, combining machine learning, systems, and security for practical multi-agent orchestration.

Tool Calling Is Not Just Plumbing for AI Agents — Roy Derks

Feb 22, 2025 · 25:18

Roy Derks argues that tool calling is the most critical yet overlooked component of AI agents, far more than mere 'plumbing.' He contrasts traditional tool calling—where developers manually manage callbacks, retries, and errors within the agent loop—with embedded tool calling, a black-box approach used by frameworks like LangChain’s createReactAgent. Derks advocates for separation of concerns via the Model Context Protocol (MCP) from Anthropic, which splits tool logic into MCP servers communicating with clients, and via standalone tool platforms such as IBM’s wxflows, Composio, and Toolhouse that let teams build tools once and reuse them across LangChain, CrewAI, or AutoGen. He also introduces dynamic tools, where an agent generates queries on the fly—e.g., using GraphQL or SQL schemas—instead of defining hundreds of static tools, noting that LLMs like Claude handle GraphQL well but may hallucinate on deeply nested schemas. The episode emphasizes that 'an agent is only as good as its tools' and provides practical guidance on designing tool descriptions (which act like system prompts) and output schemas to enable type-safe, chainable tool calls.

Cohere: Building enterprise LLM agents that work (Shaan Desai)

Feb 22, 2025 · 18:29

Shaan Desai, a machine learning engineer at Cohere, presents key strategies for building enterprise LLM agents that are scalable, safe, and seamless. He recommends using native or LangGraph frameworks for high observability in large-scale agents, while CrewAI or AutoGen suit quick proofs of concept. Core insights include starting with a single LLM and a handful of tools, simplifying tool specifications with clear descriptions and sharp examples, and caching chat history to prevent hallucinations beyond 20 turns. For multi-agent setups, the router needs clear routing instructions for edge cases, and sub-agents should be constrained to independent tasks. Safety is paramount, with human-in-the-loop triggered before or after tool calls based on codified rules. Evaluation uses a golden set of ground truth queries, expected tool calls, and outputs. Failure mitigation ranges from prompt engineering for low-severity issues to targeted annotation datasets for 10–20% failure rates and synthetic data fine-tuning for high failure rates. Cohere packages these learnings into NORTH, a single container deployment with RAG, vector DBs, and connectivity to Gmail, Outlook, Drive, and Slack,…

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Claude plays Minecraft!

Feb 15, 2025 · 18:16

Derek from AWS demonstrates building Rocky, an autonomous Minecraft agent using Amazon Bedrock, Claude 3 Haiku, and serverless AWS services, proving agentic workflows enable AI to reason and act beyond chatbots. He explains the architecture: Minecraft server on ECS with Mineflare bot framework, agents for Bedrock with return of control, and prompt engineering for tasks like digging and building. In a live demo, Rocky jumps, finds players, digs a hole, and builds a double-decker couch from a chat command, inferring parameters from natural language. The code is open-source and built with CDK/CloudFormation, emphasizing that agentic AI can operate in complex 3D environments through tool orchestration and managed prompt design.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.

RAG and the MongoDB Document Model: Ben Flast

Feb 8, 2025 · 13:13

Ben Flast, Director of Product at MongoDB, explains how the MongoDB document model and Atlas Vector Search enable more efficient and scalable Retrieval Augmented Generation (RAG). He contrasts MongoDB's JSON-based documents with relational databases, highlighting that documents store application objects directly without stitching tables. Flast details MongoDB's HNSW-based vector search, which stores embeddings alongside transactional data in the same document, supporting up to 4,096 dimensions. He introduces Search Nodes, which decouple vector search infrastructure from transactional databases for independent scaling. Flast covers AI integrations with LangChain, LlamaIndex, and others, enabling features like semantic caching and chat history memory within a single database. He cites the startup 4149, which uses MongoDB to store user data, meeting notes, and vector embeddings for an AI teammate that assists with tasks by combining transactional and semantic search.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

Cohere for VPs of AI: Vivek Muppalla

Feb 5, 2025 · 16:11

Vivek Muppalla, Director of Engineering at Cohere, details the company's enterprise AI strategy centered on security, customization, and deployment flexibility. He presents Cohere's product line: Command R and R+ for generation, plus advanced retrieval models like embeddings and the ReRanker, which reduces RAG costs by narrowing context. Key claims include a focus on enterprise-specific eval suites (health, HR, finance), out-of-the-box citations, and multilingual performance. Partnerships with Accenture and McKinsey bridge the last-mile gap, while wins often stem from private cloud deployment and data control. In Q&A, he recommends purpose-built classifiers for high-throughput production and notes current 128k context windows.

Knowledge Graphs & GraphRAG: Techniques for Building Effective GenAI Applications: Zach Blumenthal

Nov 1, 2024 · 1:39:52

In this workshop, Zach Blumenthal of Neo4j demonstrates how to build a GraphRAG application using Neo4j, OpenAI, and LangChain on an H&M fashion dataset. He argues that combining knowledge graphs with vector search and graph embeddings improves retrieval and personalization for LLMs. The session covers creating a Neo4j sandbox, loading data, performing vector search with OpenAI embeddings, adding collaborative filtering via graph traversal patterns, using graph data science (FastRP) to generate graph embeddings for recommendations, and integrating everything into a LangChain chain that generates personalized marketing emails. Attendees build a Gradio app that, given a customer ID and season, outputs a tailored email with product recommendations, demonstrating how structured graph data enhances RAG systems beyond pure vector search.

Build an AI Research Agent: Apoorva Joshi

Oct 25, 2024 · 27:33

In this workshop, Apoorva Joshi, an AI Developer Advocate at MongoDB, teaches how to build an AI research agent using MongoDB as the memory provider and knowledge store, open-source LLMs from Fireworks AI (Fire Function V1) as the agent’s brain, and LangChain to orchestrate the workflow. The agent searches for research papers, summarizes them, and answers questions based on past research, using tools like ArXivLoader and a MongoDB vector store. Joshi explains key agent concepts—planning with chain of thought and react patterns, short-term and long-term memory, and tool creation—then guides attendees through hands-on sections to build the agent step by step. Attendees learn to create agent tools, implement reasoning with react, and add short-term memory persisted to MongoDB. The workshop emphasizes that agents enable complex, multi-step tasks through iterative reasoning, tool use, and memory, trading higher cost and latency for improved accuracy.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Building State of the Art Open Weights Tool Use: The Command R Family: Sandra Kublik

Aug 26, 2024 · 15:03

Sandra Kublik of Cohere presents the Command R family of open-weight models optimized for retrieval-augmented generation and tool use. Released in March 2024, Command R and Command R+ achieved 150,000 Hugging Face downloads within two weeks and now serve nearly 500,000 developers. The models overcome RAG challenges such as prompt sensitivity and citation accuracy through post-training, delivering fine-grained citations and low hallucination. Cohere open-sourced a toolkit UI with plug-and-play components for RAG and tool use, supporting cloud, local, and Hugging Face access. The new Multi-Step API enables sequential reasoning with automatic retry and reflection. Command R+ matches GPT-4 Turbo and Claude Opus on complex reasoning while being three to five times cheaper, positioning it as a scalable enterprise solution.

Building Reliable Agentic Systems: Eno Reyes

Aug 20, 2024 · 18:14

Eno Reyes, CTO of Factory.ai, explains how his team builds reliable agentic systems—called Droids—for automating software development tasks by applying techniques from robotics and control systems to handle planning, decision making, and environmental grounding. Reyes describes a pseudo-Kalman filter that passes intermediate reasoning through plan steps, converging reasoning but risking error propagation. He advocates for explicit plan criteria and hard-coded logic to improve reliability, despite reducing generalizability. On decision making, he recommends consensus mechanisms like self-consistency, explicit reasoning with checklists, fine-tuning for out-of-distribution decisions, and simulation via Monte Carlo tree search. For environmental grounding, he emphasizes building custom AI computer interfaces for domain-specific workflows, designing explicit feedback processing (e.g., parsing CI/CD logs), balancing bounded exploration with long-context models, and incorporating human guidance to boost reliability from 30–40% to 90–100%.

Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Aug 14, 2024 · 1:29:02

In this workshop, Microsoft's Gabriela de Queiroz, Pamela Fox, and Harald Kirschner show how to deploy AI applications in minutes using AI templates, Azure OpenAI proxy, and GitHub Codespaces. They walk through deploying a simple chat app, then two RAG applications: one that queries a Postgres database with SQL filtering, and another that performs RAG on unstructured documents using Azure AI Search. Key decisions include using async frameworks like Quart and FastAPI, token-based chunking, and hybrid retrieval with semantic reranking for best results. They stress the importance of running evaluations with hundreds of samples and share production insights from Copilot Chat, where TFIDF sparse indexing and LLM reranking are used. The session includes free Azure credits and a proxy to bypass Azure OpenAI approval, allowing attendees to deploy everything without spending their own money.

Using agents to build an agent company: Joao Moura

Aug 8, 2024 · 13:46

Joao Moura, founder of crewAI, explains how he used AI agents to build his own company, sharing that over 10.5 million agents were executed with crewAI in the last 30 days. He describes starting by building a marketing crew that automated LinkedIn and content creation, which generated 10x more views in 16 days, then a lead qualification crew that led to 150+ customer calls in two weeks. Moura details new features: code execution with a single flag, trainable crews for consistent results, support for third-party agents, and crewAI Plus for deploying crews as auto-scaling APIs in minutes. He emphasizes starting simple with low-risk, high-impact use cases and encourages attendees to bring agents into production now.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

Retrieval Augmented Generation in the Wild: Anton Troynikov

Nov 15, 2023 · 12:20

Anton Troynikov, co-founder of Chroma, explains that retrieval augmented generation (RAG) requires more than simple vector search—it needs human feedback, self-updating memory, and agent interaction to handle dynamic data. He covers challenges like choosing the right embedding model, chunking strategies (including using language model perplexity), and determining result relevance without distractors. Chroma is building a horizontally scalable cluster, a cloud technical preview by December, and support for multimodal data. The episode argues that a capable memory system is key to making AI agents truly functional, citing the Voyager paper where Chroma stored learned skills for Minecraft agents.

Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Nov 14, 2023 · 25:09

Abi Aryan's talk covers domain adaptation and fine-tuning for large language models, contrasting prompting, RAGs, and three fine-tuning methods: adaptive, behavioral, and parameter-efficient. Adaptive fine-tuning adds small adapter modules (0.15% of parameters) for new domains like biochemical engineering; behavioral fine-tuning optimizes label space for a single task; and parameter-efficient methods like LoRA and QLoRA reduce model size via low-rank adaptation and four-bit precision, ideal for low-resource devices. Aryan emphasizes data quality—deduplication reduces memorization—and practical tips: batch size of 32 or 64, starting with 100 epochs, using Adam optimizer, gradient checkpointing for memory savings, and in-context learning with dynamic examples to handle drift. Evaluation should combine metric-based (Bleu, Rouge), tool-based (Weights & Biases), model-based, and human-in-the-loop approaches, though full pipeline considerations (data collection, base model choice, storage) are critical for robust applications.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Building Context-Aware Reasoning Applications with LangChain and LangSmith: Harrison Chase

Nov 1, 2023 · 18:54

Harrison Chase, CEO of LangChain, argues that building context-aware reasoning applications on LLMs requires treating the model as part of a larger system, and he outlines key approaches and challenges. He categorizes context provision as instruction prompting, few-shot examples, retrieval-augmented generation (RAG), and fine-tuning. For reasoning, he describes a spectrum from single LLM calls to chains, routers, agents with cycles, and autonomous agents like Auto-GPT. Chase then details engineering challenges: choosing the right cognitive architecture, data engineering for context, prompt engineering across multi-step systems, evaluation using LLM-assisted metrics and user feedback, and enabling collaboration between technical and non-technical team members. He emphasizes that the field remains early and that LangChain and LangSmith help prototype, debug, and iterate on these systems.