LlamaIndex

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Aug 17, 2026 · 1:03:26

Louis-François Bouchard, Omar Solano, and Samridhi Vaid of Towards AI test context engineering on their AI tutor: keeping full history beat every compaction technique on recall, cost, and latency because 97% of tokens were served from cache (up to 50x cheaper), making summarization a trap unless it shrinks context by more than 50x. Full history recovered specific details 95% vs 32% after summarizing, and distinctive facts survived 800k tokens. Local hardware changes it: a 32k window can't keep everything, and larger models don't widen context. Dense retrieval hit 0% recall at 400k tokens where BM25 got 100%, so they use hybrid retrieval. Rule: name the constraint before compacting.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

Agentic Search for Context Engineering — Leonie Monigatti, Elastic

May 8, 2026 · 1:03:13

Leonie Monigatti from Elastic argues that context engineering is 80% agentic search—the search tool that decides what to pull from files, databases, memory, and the web. She identifies three failure modes: the agent not calling any tool, calling the wrong tool, or generating incorrect parameters, and shows how detailed tool descriptions and agent skills (progressive disclosure) reduce these. In demos, semantic search fails for keyword 'JEPA' due to embedding similarity; a general-purpose ESQL tool with an agent skill correctly queries the database. Shell/bash tool enables file system search but requires iterative grep, while the custom Gina Grap CLI provides semantic grep for fuzzy queries. Practical recommendations: start with general-purpose tools, log agent behavior, and add specialized tools for frequent queries to balance low floor (easy successes) and high ceiling (complex queries). Hybrid agents combining shell and database tools achieve higher accuracy by verifying results.

Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Jul 21, 2025 · 15:27

Stephen Chin of Neo4j argues that Agentic GraphRAG — combining graph databases with retrieval-augmented generation — overcomes LLM hallucinations and biases by providing structured, relational context. He demonstrates how LLMs fail on reasoning tasks like calculating classroom capacity due to inaccurate anchoring on irrelevant data, and proposes an architecture where vector search first identifies relevant nodes, then graph traversal retrieves related context for the LLM. Chin highlights Neo4j’s MCP server for cypher query generation and memory modules, and cites Klarna’s success: 250K employee questions answered in the first year, 2,000 daily queries, and 85% adoption, replacing their entire SaaS stack. He recommends the Neo4j Certified Developer Program and the Nodes Conference for further learning.

Effective agent design patterns in production — Laurie Voss, LlamaIndex

Jun 27, 2025 · 15:38

Laurie Voss, VP of Developer Relations at LlamaIndex, presents five agent design patterns—chaining, routing, parallelization, orchestrator workers, and evaluator optimizers—for building production-ready agents. He argues that agents improve RAG by enabling introspection, self-correction, and better accuracy, and that RAG remains essential because it's cheaper and faster than feeding full context. LlamaIndex provides a framework, LlamaParse for parsing complicated documents, and LlamaCloud for managed retrieval endpoints, integrating with 400 models over 80 providers. He demonstrates how workflows in LlamaIndex enable concurrent event emitting to implement parallelization and voting to reduce hallucination, noting that three different LLMs seldom hallucinate the same answer. The evaluator optimizer pattern uses an LLM to check its own output and loop back for improvement. Multi-agent systems can be built with a single line of code by passing an array of function agents.

Why Your Agent’s Brain Needs a Playbook: Practical Wins from Using Ontologies - Jesús Barrasa, Neo4j

Jun 27, 2025 · 13:54

Jesús Barrasa, AI Field CTO at Neo4j, argues that ontologies are the secret weapon for building robust GraphRAG applications, providing both a formal, implementation-agnostic schema for knowledge graph creation and a dynamic data layer that drives retriever behavior. He explains how ontologies replace rigid code-orchestrated workflows and chaotic LLM-generated ones by guiding entity extraction from unstructured data and mapping structured sources into a property graph. During retrieval, storing the ontology in the graph enables dynamic Cypher queries that automatically navigate contextualizing relationships — such as "acted-in" rather than "directed" — based on annotated subproperties, without hard-coded logic. This allows developers to change retrieval behavior on the fly by modifying the ontology as a data artifact. The talk covers two practical wins: using ontologies as a shared domain model for graph construction, and leveraging them to build adaptive, schema-driven retrievers that improve result completeness and relevance.

Memory Masterclass: Make Your AI Agents Remember What They Do! — Mark Bain, AIUS

Jun 27, 2025 · 51:25

Mark Bain, Vasilia Markovits, Alex Gilmore, and Daniel Chalev demonstrate that AI memory requires causal relationships and graph databases, not just vector similarity, to solve hallucinations and enable agentic workflows. Bain argues that memory is any data affecting change, and that attention, diffusion, and VAEs follow the same geometric principles as gravity and entropy. Alex shows Neo4j's MCP server storing semantic memory as entities and relationships retrieved across conversations. Vasilia demos Cognee building semantic graphs from GitHub data for agentic hiring decisions. Daniel presents Graphiti's domain-aware memory using custom Pydantic schemas to filter irrelevant facts. Bain introduces a GraphRAG chat arena that switches between memory solutions on a single Neo4j graph, testing different implementations for episodic and temporal recall.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

open-rag-eval: RAG Evaluation without "golden" answers — Ofer Mendelevitch, Vectara

Jun 3, 2025 · 5:03

Ofer Mendelevitch from Vectara presents Open-RAG-Eval, an open-source framework that enables RAG evaluation without requiring golden answers or golden chunks, solving a major scalability problem. Backed by research with the University of Waterloo's Jimmy Lin Lab, it uses UMBRELA for retrieval scoring on a 0–3 scale that correlates well with human judgment, and AutoNuggetizer for generation via nugget creation, Vital/OK ratings, and an LLM judge analyzing the top 20 nuggets for support. Additional metrics include citation faithfulness and Vectara's HHEM hallucination detection model. Connectors are available for Vectara, LangChain, and LlamaIndex, with results viewable through an intuitive UI at OpenEvaluation.ai.

OpenLLMetry is all you need

Feb 22, 2025 · 9:12

Nir, CEO of Trace Loop, introduces OpenLLMetry, an open-source project extending OpenTelemetry for tracing and monitoring GenAI applications. OpenTelemetry, maintained by CNCF, standardizes logging, metrics, and traces across cloud environments, supported by platforms like Datadog, New Relic, and Grafana. OpenLLMetry provides over 40 automatic instrumentations for foundation models (OpenAI, Anthropic, Cohere), vector databases (Pinecone, Chroma), and frameworks (LangChain, LlamaIndex, CrewAI). These instrumentations emit logs, metrics, and traces in OpenTelemetry format, allowing users to send data to any supported observability backend with a configuration change, avoiding vendor lock-in.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.

RAG and the MongoDB Document Model: Ben Flast

Feb 8, 2025 · 13:13

Ben Flast, Director of Product at MongoDB, explains how the MongoDB document model and Atlas Vector Search enable more efficient and scalable Retrieval Augmented Generation (RAG). He contrasts MongoDB's JSON-based documents with relational databases, highlighting that documents store application objects directly without stitching tables. Flast details MongoDB's HNSW-based vector search, which stores embeddings alongside transactional data in the same document, supporting up to 4,096 dimensions. He introduces Search Nodes, which decouple vector search infrastructure from transactional databases for independent scaling. Flast covers AI integrations with LangChain, LlamaIndex, and others, enabling features like semantic caching and chat history memory within a single database. He cites the startup 4149, which uses MongoDB to store user data, meeting notes, and vector embeddings for an AI teammate that assists with tasks by combining transactional and semantic search.

Cohere for VPs of AI: Vivek Muppalla

Feb 5, 2025 · 16:11

Vivek Muppalla, Director of Engineering at Cohere, details the company's enterprise AI strategy centered on security, customization, and deployment flexibility. He presents Cohere's product line: Command R and R+ for generation, plus advanced retrieval models like embeddings and the ReRanker, which reduces RAG costs by narrowing context. Key claims include a focus on enterprise-specific eval suites (health, HR, finance), out-of-the-box citations, and multilingual performance. Partnerships with Accenture and McKinsey bridge the last-mile gap, while wins often stem from private cloud deployment and data control. In Q&A, he recommends purpose-built classifiers for high-throughput production and notes current 128k context windows.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Using agents to build an agent company: Joao Moura

Aug 8, 2024 · 13:46

Joao Moura, founder of crewAI, explains how he used AI agents to build his own company, sharing that over 10.5 million agents were executed with crewAI in the last 30 days. He describes starting by building a marketing crew that automated LinkedIn and content creation, which generated 10x more views in 16 days, then a lead qualification crew that led to 150+ customer calls in two weeks. Moura details new features: code execution with a single flag, trainable crews for consistent results, support for third-party agents, and crewAI Plus for deploying crews as auto-scaling APIs in minutes. He emphasizes starting simple with low-risk, high-impact use cases and encourages attendees to bring agents into production now.

Retrieval Augmented Generation in the Wild: Anton Troynikov

Nov 15, 2023 · 12:20

Anton Troynikov, co-founder of Chroma, explains that retrieval augmented generation (RAG) requires more than simple vector search—it needs human feedback, self-updating memory, and agent interaction to handle dynamic data. He covers challenges like choosing the right embedding model, chunking strategies (including using language model perplexity), and determining result relevance without distractors. Chroma is building a horizontally scalable cluster, a cloud technical preview by December, and support for multimodal data. The episode argues that a capable memory system is key to making AI agents truly functional, citing the Voyager paper where Chroma stored learned skills for Minecraft agents.