Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest
Jul 20, 2026 · 11:21
Drasko Profirovic, a Staff Engineer at Pinterest, presents Medic for Apache Spark, an agentic diagnostics tool that automatically troubleshoots Spark job failures by ingesting logs, correlating context, and producing human-quality diagnoses and actionable recommendations in minutes. The talk covers the journey from a single React agent with unsustainable prompt tuning to a multi-agent architecture built on LangGraph's deep agent library, where specialized agents handle triage, research, and healing. Key improvements include an exception classifier pipeline that filters benign exceptions from logs, converting raw time-series metrics into annotated graphs for token efficiency, and an end-to-end test harness that snapshots production state for offline evaluations. Profirovic shares lessons on handling ambiguity, reducing hallucinations, and balancing automation with human oversight, plus unexpected failure modes that informed iterations. The system now extends to optimizing Spark SQL…
