Raindrop

Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

Aug 12, 2026 · 19:46

Ben Hylak, CTO and co-founder of Raindrop, argues that most eval advice is stuck in the chatbot era and that agents have effectively infinite issues, so the real question is which ones matter—measured by when an issue started and what share of users it hits. He reframes agent quality around raising the floor (the worst thing an agent can do, like recommending a competitor or deleting data) rather than the ceiling, and says evals belong in your repo as code tests, not prompt playgrounds, because the harness is the product. He offers three tactical lessons from Raindrop: clusters are not issues because boundaries drift and you don't control them; code mode scales to traces, letting you write classifiers and run them in a sandbox at production volume; and agents are poor at anomaly detection but good at investigating anomalies you surface deterministically, like keyword spikes. He also notes that continual learning is rare in the real world, and that your approach should depend on user…

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

Building AI Products That Actually Work — Ben Hylak (Raindrop), Sid Bendre (Oleve)

Jul 24, 2025 · 18:42

Ben Hylak (Raindrop) and Sid Bendre (Oleve) argue that building reliable AI products requires iterative real-world signals over traditional evals. Ben debunks eval myths—evals don't measure product quality, LLM-as-judge fails, and production evals are costly—and stresses tracking explicit signals (thumbs up/down, copy rate) and implicit signals (refusals, frustration) to identify issues. Sid introduces Trellis, a framework for scaling viral AI apps that uses discretization—breaking infinite output into intent buckets—prioritization by volume times negative sentiment times achievable delta, and recursive refinement. Starting with an MVP, teams classify user intents, convert them into semi-deterministic workflows, then repeatedly drill into sub-intents to engineer repeatable, attributable magic. Oleve's approach, powered by Raindrop, has scaled six viral products to $6M ARR profitably with four people.