Observability

AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok

Aug 29, 2026 · 19:48

Salman Munaf, TikTok site reliability engineer, argues AI agents are distributed systems once they call external services; deterministic controls must surround the probabilistic coordinator. A refund timeout illustrates it: timeout means unknown, not failure, and retrying can double-refund without request IDs, idempotency keys, and status lookups. He urges persisting every step, defining compensating actions, treating action-influencing context as cacheable state with invalidation and provenance, and adding guardrails: circuit breakers, budgets, exponential backoff, scoped read/write credentials, and approvals bound to action, timestamp, actor, expiration. Observability must trace prompts, calls, writes; tool contracts embed idempotency; ask what system lets the agent do when wrong.

FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft

Aug 22, 2026 · 21:24

Microsoft engineers Tisha Chawla and Susheem Koul present TokenOps, a runaway token governance control plane for AI agents, arguing the agentic era's missing control surface sits between code and model calls, not at request-level gateways. The design has an out-of-band SDK with boundary annotations, a ledger, and a governor, plus a control plane that groups runs into segments, sets budgets, and applies policies. Enforcement comes in two flavors: halt is a circuit breaker, while steer uses a cost guard that watches budget consumed and token velocity and, predicting an overrun, injects an instruction to keep outputs succinct instead of killing the run. They show preview mode with enforcement off, then enforced halt and steering. Benchmarked on Browser Use and MetaGPT, the full policy suite cut average agent spend by about 78% and lifted run completion from 67% to roughly 96% versus throttling.

Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

Aug 20, 2026 · 24:13

Ameya Bhatawdekar, Field CTO at Braintrust, argues that evals are the durable asset across AI replatformings and must move with each architectural generation. He traces five generations—single prompt, retrieval chain, ReAct loop, workflow graphs, and the return of loops with newer models—grounded in a site reliability agent, showing how each added failure surfaces: parsers, retrievers, branch logic, node contracts, classifier misfires, and trajectory variance. New metrics like pass@k (succeeds at least once in k attempts) measure capability, while pass^k (how many of k succeed) measure reliability. He urges harvesting production data into evals via the flywheel, and says Braintrust's Topics clusters production data to surface unanticipated failure modes.

Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

Aug 19, 2026 · 20:02

Ayush Bhardwaj, who applied AI at a hedge fund before pharma tech startup Allos AI, argues vertical AI projects die from not being able to judge whether they work. His seven-step recipe: narrow tasks, proprietary data, expert-modeled prompts, observability, then hiring the user—a domain expert—because using LLM-as-judge is a "stupid mistake" that jargons its way out. Verifiable rewards fail in finance and pharma: no answer keys, and data is withheld—30% of firms never disclose clinical trials, and the FDA publicly reminded over 2,000 sponsors in 2026. He says 89% of enterprise AI agents never reach production; the fix is endless expert-in-the-loop learning, starting with error analysis as highest ROI. Moat is never model or infra; only curated domain expertise and proprietary data matter.

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

Aug 12, 2026 · 20:02

Vivek Trivedy, lead of applied research at LangChain, argues that improving agents is fundamentally a data mining problem: ship agents, collect traces, then mine them to drive continual learning. He claims observability and continual learning are the same problem because agents operating in environments produce trace data, which is the substrate for all improvement. Trivedy details how LangChain sends agents to read other agents' traces to find good/bad interactions, detect degradation after compactions, and test counterfactuals like swapping GPT-5.5 for GLM 5.2. He shares that with Harvey on a legal benchmark, an open model matched Opus's trace judging at one to two orders of magnitude lower cost, achieved through harness engineering informed by traces. His rule for when to stop prompt tuning and start fine-tuning is feedback speed: harness engineering answers in about two minutes, so exhaust that ceiling first, then fine-tune to break through, then return to harness engineering. He…

Claude Managed Agents and Evolution of Agentic Surfaces — Gagan Bhat & Isabella Kai He, Anthropic

Aug 11, 2026 · 31:24

Gagan Bhat and Isabella Kai He of Anthropic's Applied AI team explain how agentic surfaces evolved from the Messages API to the Claude Agent SDK and now Claude-managed agents, arguing that harnesses encode assumptions about model limits that go stale as models improve. They detail decoupling the brain (agent loop) from the hands (tool execution sandbox), which cut time to first token by 60% at P50 and over 90% at P95, and made failures recoverable via durable session logs. The session log also powers observability, context recovery, and 'dreaming,' a batch process that rewrites agent memory for self-improvement. They cover production lessons: keeping credentials in vaults, self-hosted sandboxes for VPC control, MCP tunnels for private servers, and the 'outcomes' feature that uses a grader agent to enforce success criteria.

The Model Was Right. The Harness Failed. — Vinoth Govindarajan, OpenAI

Jul 29, 2026 · 18:26

Vinoth Govindarajan of OpenAI argues that most agent failures are harness failures, not model failures. Using OpenClaw as a case study, he details five failure shapes: state hole (delivered but not remembered), overlapping writers (last write silently erases previous), dangling tool call (run waits for an event that never arrives), approval drift (expired approval blocks later work), and missing edge proof (internal success but user didn't see result). The through line: a model proposes, the harness commits, and the receipt proves it. He prescribes three invariants—own the state, order the mutation, prove the action—and a five-question run receipt audit: what woke it up, what state did it inherit, what authority did it use, what executed, and what evidence survived.

From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

Jul 24, 2026 · 20:36

Jason Lopatecki demonstrates how Arize's Signal agent transforms observability from human-clicked dashboards into telemetry for self-fixing systems. The key unlock is pulling production traces and logs as files into a repo—Claude Code works magically with files, not dashboards—so the agent sees the exact code path software took. Signal runs periodically or event-based in a sandbox (VPC-deployed for Uber, Booking), using composable skills to gather context, find root cause, and create a PR. Lopatecki argues you should trace and log ten times more because agents can read that smoke. For the question 'why not just point Claude Code at data?', he explains skills must be well-designed to fetch and format the right data into files. On evals, they run as LLM judges layered on production traces, pre-processing information for the agent to catch known failures and create new evaluators.

Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

Jul 23, 2026 · 19:38

Ritvik Pandya of JP Morgan Chase presents Learned Execution Graphs, a method that models each API request as a short-lived DAG of middleware steps learned from telemetry at over 1,600 requests per second. The system detects anomalies by comparing actual execution against a learned baseline, localizing deviations to exact nodes instead of whole endpoints. In production it flagged a 41x deviation at a single node that service-level monitoring missed, cutting root cause from hours to under 30 seconds. Pandya distinguishes one-off anomalies from drift, categorizing drift into structural (added/removed steps), volume (scaling needs), and covariate (shifting request demographics), using per-client baselines and KL divergence rather than a single threshold. The approach uses tiered checks: a cheap first check only escalates when the graph signals a real change, reducing false alarms and enabling faster automated responses.

Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest

Jul 20, 2026 · 11:21

Drasko Profirovic, a Staff Engineer at Pinterest, presents Medic for Apache Spark, an agentic diagnostics tool that automatically troubleshoots Spark job failures by ingesting logs, correlating context, and producing human-quality diagnoses and actionable recommendations in minutes. The talk covers the journey from a single React agent with unsustainable prompt tuning to a multi-agent architecture built on LangGraph's deep agent library, where specialized agents handle triage, research, and healing. Key improvements include an exception classifier pipeline that filters benign exceptions from logs, converting raw time-series metrics into annotated graphs for token efficiency, and an end-to-end test harness that snapshots production state for offline evaluations. Profirovic shares lessons on handling ambiguity, reducing hallucinations, and balancing automation with human oversight, plus unexpected failure modes that informed iterations. The system now extends to optimizing Spark SQL…

Feedback Loops are All You Need — Mehedi Hassan, Granola

May 10, 2026 · 10:11

Mehedi Hassan, a product engineer at Granola, argues that shipping AI features into production requires building feedback loops rather than one-shotting better prompts. Granola's chat feature for meeting notes revealed problems with web search — token costs ballooning to 10p per chat, overnight provider updates silently degrading results — and with prompt personalization, as a single prompt cannot serve salespeople, engineers, and HR managers equally. To close the gap, Granola built custom internal tracing exposing tool calls, search trails, reasoning, and cost in a UI accessible to non-engineers, not just CloudWatch queries. They also refactored their Electron app's renderer to run as a web app, enabling preview links on every PR and allowing Cursor to automatically test changes and upload screenshots. The result is faster iteration and confidence that shipped features actually work for customers.

How to look at your data — Jeff Huber (Chroma) + Jason Liu (567)

Aug 6, 2025 · 19:23

Jeff Huber (Chroma CEO) and Jason Liu argue that AI practitioners should look at their data—both inputs and outputs—with fast evals and conversation analysis to systematically improve retrieval and product decisions. Huber advocates for fast evals using golden datasets of query-document pairs over public benchmarks or expensive LLM judges, showing how synthetic queries aligned to real user behavior can empirically compare embedding models; in a Weights & Biases chatbot case study, Voyage 3 large outperformed text-embedding-3-small and others. Jason Liu focuses on outputs: extract structured metadata from conversations, cluster them to find segments, then compare KPIs like performance across clusters to prioritize what to fix, build, or ignore—for example, if 40% of conversations involve data visualization and the agent performs poorly, invest in better plotting tools. This population-level analysis enables impact-weighted decisions, turning vague metrics like 0.5 factuality into actionable insights by segment. The episode emphasizes that retrieval improvements are foundational, and once you have users, looking at conversation structure drives a data-driven product roadmap.

Continuous Profiling for GPUs — Matthias Loibl, Polar Signals

Jul 22, 2025 · 11:31

Matthias Loibl of Polar Signals explains how continuous profiling for GPUs maximizes GPU efficiency using low-overhead, always-on sampling via eBPF. He contrasts tracing (high cost) with sampled profiling (e.g., 100 Hz, <1% overhead) and details GPU metrics collected from NVIDIA NVMe, including utilization, memory, clock speed, power, temperature, and PCIe throughput. The platform correlates these with CPU stack traces to identify bottlenecks, such as Python and CUDA functions underutilizing the GPU. A new GPU time profiling feature records the duration of CUDA kernel executions, showing actual time spent by functions on the GPU. Deployment runs on Linux with a binary, Docker, or Kubernetes DaemonSet; early adopters like TurboPuffer use it to optimize their vector engine.

The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

Jul 2, 2025 · 12:50

Jeremy Silva (Freeplay) and Chris Hernandez (Chime) argue that the biggest challenge in generative AI isn't building prototypes but crossing the 'quality chasm' from v1 to reliable v2 through operational iteration. They explain how the lower barrier to entry and faster iteration speed in Gen AI accentuate the need for high-quality ops, where product quality becomes a direct function of how fast teams move through monitoring, experimentation, and evaluation loops. Chris emphasizes that human-in-the-loop isn't just a safeguard but a feedback engine, and that existing QA and CX teams in operations are already equipped to become 'model shapers'—labeling data, testing prompts, and defining what good looks like. Jeremy introduces the emerging role of the 'AI quality lead,' a systems thinker who can run experiments and evaluations without writing production code. They conclude that scaling Gen AI is an operational and people challenge, not just a technical one, and that embedding quality and human feedback early is the key to building faster and better.

[Evals Workshop] Mastering AI Evaluation: From Playground to Production

Jul 1, 2025 · 1:25:08

In this workshop, Braintrust Solutions Engineers Carlos Esteban and Doug guide participants through the complete AI evaluation lifecycle, from offline testing in the playground to production monitoring. They explain the three core ingredients of an eval—task, data set, and score—and demonstrate how to run evals both via the Braintrust UI and the SDK. The session covers LLM-as-judge vs. deterministic code scores, the importance of starting small with synthetic data, and how to use online scoring and logging to capture real user feedback. Human-in-the-loop review is highlighted as a way to establish ground truth and close the feedback loop. The presenters also address audience questions on bootstrapping data sets, non-determinism in LLM judges, and integrating evals into existing projects.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.

OpenLLMetry is all you need

Feb 22, 2025 · 9:12

Nir, CEO of Trace Loop, introduces OpenLLMetry, an open-source project extending OpenTelemetry for tracing and monitoring GenAI applications. OpenTelemetry, maintained by CNCF, standardizes logging, metrics, and traces across cloud environments, supported by platforms like Datadog, New Relic, and Grafana. OpenLLMetry provides over 40 automatic instrumentations for foundation models (OpenAI, Anthropic, Cohere), vector databases (Pinecone, Chroma), and frameworks (LangChain, LlamaIndex, CrewAI). These instrumentations emit logs, metrics, and traces in OpenTelemetry format, allowing users to send data to any supported observability backend with a configuration change, avoiding vendor lock-in.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.

What It Actually Takes to Deploy GenAI Applications to Enterprises: Arjun Bansal and Trey Doig

Nov 4, 2024 · 21:30

Trey Doig of Echo AI and Arjun Bansal of Log10 recount Echo AI's journey deploying a GenAI-native conversational intelligence platform for billion-dollar retail brands, focusing on the centrality of accuracy. Echo AI ingests all customer conversations, uses LLMs to surface insights at 100% coverage, but must overcome enterprise trust issues by achieving 95% accuracy within seven days. The platform relies on Log10's auto feedback system, which uses AI-based review to match human accuracy with model speed, yielding a 20 F1 point improvement in one use case. The episode details how Echo AI's solution engineers use Log10 to grade summarizations, catch hallucinations, and track model drift, turning human feedback into curated datasets for fine-tuning. Ultimately, the partnership demonstrates a path to self-improving LLM applications through iterative accuracy measurement and improvement.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.