Grafana

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Aug 1, 2026 · 21:15

Rayan Garg and Gurveer of Theta Software argue that progress on long-horizon AI agents depends on environment and verifier design, not benchmark headlines. They define long horizon via METR-style human time thresholds, such as 50% success on tasks taking humans 16 hours, alongside noisy model units like tokens and steps, and say real difficulty comes from sequential complexity—where a bad early query cascades—rather than artificially chained subtasks. They explain why soft verifiable work needs judge models with detailed rubrics, and why judges must be agents themselves: able to inspect final environment state (e.g. GitHub/CloudWatch logs) with read-only permissions, query long trajectories through sub-agents, and avoid collapsing valid solution spaces. They critique finance benchmarks GDPVal, ToolBench, and Apex agents for average human hours far below METR thresholds, saturation, narrow breadth, and weak reward signals. Theta's own finance tasks average 15 human hours across a 50-task sample, take models long trajectories, and still require mean@5 evaluation.

How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

May 28, 2026 · 20:43

Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

Production software keeps breaking and it will only get worse — Anish Agarwal, Traversal.ai

Jul 10, 2025 · 18:13

Anish Agarwal and Matthew Schoenbauer of Traversal.ai argue that as AI writes more code, production troubleshooting will become vastly harder, requiring a new approach combining causal machine learning, reasoning models, and agentic swarms to autonomously resolve incidents in minutes. They explain that traditional AI ops generates too many false positives, LLMs can't handle petabyte-scale data, and simple agents depend on deprecated runbooks. Their Traversal AI orchestrates thousands of parallel agentic tool calls to sift through trillions of logs and metrics, identifying root causes and citing observability data. A case study with DigitalOcean shows a 40% reduction in mean time to resolution (MTTR), with the system delivering findings in about five minutes. The episode details how this approach turns frantic incident Slack channels into autonomous, cited root-cause analysis, freeing engineers to focus on system design.

Building Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic

Mar 1, 2025 · 1:44:12

Mahesh Murag of Anthropic presents the Model Context Protocol (MCP) as an open standard that replaces fragmented integrations with a single protocol for connecting AI systems to data sources, enabling context-rich AI applications and agentic experiences. He explains MCP's philosophy, inspired by APIs and LSP, and its three interfaces: tools (model-controlled), resources (application-controlled), and prompts (user-controlled). Murag highlights adoption with over 1,100 community-built servers and official integrations from companies like Cloudflare and Stripe. He demonstrates building agents with MCP using the MCP-Agent framework, showing how agents can use tools dynamically and composably across hierarchical systems. Future plans include remote server support with OAuth 2.0, a centralized registry for discovery and verification, and enabling agents to self-evolve by dynamically finding new capabilities via registry search.