Trino

Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest

Jul 20, 2026 · 11:21

Drasko Profirovic, a Staff Engineer at Pinterest, presents Medic for Apache Spark, an agentic diagnostics tool that automatically troubleshoots Spark job failures by ingesting logs, correlating context, and producing human-quality diagnoses and actionable recommendations in minutes. The talk covers the journey from a single React agent with unsustainable prompt tuning to a multi-agent architecture built on LangGraph's deep agent library, where specialized agents handle triage, research, and healing. Key improvements include an exception classifier pipeline that filters benign exceptions from logs, converting raw time-series metrics into annotated graphs for token efficiency, and an end-to-end test harness that snapshots production state for offline evaluations. Profirovic shares lessons on handling ambiguity, reducing hallucinations, and balancing automation with human oversight, plus unexpected failure modes that informed iterations. The system now extends to optimizing Spark SQL…

The Hierarchy of Needs for Training Dataset Development: Chang She and Noah Shpak

Oct 15, 2024 · 16:32

Chang She (CEO of LanceDB) and Noah Shpak (AI data platform lead at Character AI) argue that data infrastructure is the critical bottleneck for LLM training, and LanceDB's columnar format solves the 'new cap theorem' for AI: needing fast scans, random access, and handling large multimodal blobs simultaneously. Noah explains how Character AI structures pre-training around wide domain coverage and post-training around granular analytics like token counts and difficulty scores, using synthetic data, quality scoring, and dataset selection to improve models. Chang details how Lance format provides zero-copy schema evolution, time travel, and indexing extensions for vector, scalar, and full-text search, enabling a single table to serve SQL analytics, PyTorch training, and production vector search. The episode emphasizes that speed and iterative dataset management are key to accelerating AI research, with LanceDB facilitating cheap random access and low-infra billion-scale vector search.