Nubank

We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank

Jul 29, 2026 · 16:24

Lucas Palma, product security manager at Nubank, explains how his team built Skill Vector to vet AI skills as supply chain risks inside a regulated bank, scanning over 2,000 skills before they reached developers. The tool uses a hybrid approach: deterministic checks catch destructive shell commands and credential requests, then an LLM reviews behavioral context missed by patterns. Across 2,000 skills, the system identified more than 1,500 risks, with 1,000 remediated immediately and a few blocked entirely from the internal marketplace. Key lessons include treating skills like any dependency, running local scans alongside CI enforcement, and requiring proper human-in-the-loop approval rather than AI self-confirmation. Palma also applies the same gates to MCP servers, rules, and third-party plugins, pushing for a trusted canonical marketplace where every entry is scanned before distribution.

SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)

Jul 29, 2026 · 16:29

Shreya Rajpal (CEO of Snowglobe) and Aman (Principal ML Engineer at Nubank) argue that generating evaluation data in simulation instead of waiting on production data lets Nubank ship AI agents 20× faster. Nubank serves 135 million customers and has five agents in production, with TNPS approaching human quality. Snowglobe points at the agent, generates thousands of grounded multi-turn conversations (e.g., persona Maria Souza ordering a credit card), and pipes results into evals. Human review found simulated conversations comparable to real ones 80% of the time, enabling the team to catch regressions before production and improve one agent's self-service rate by 4%. The tight ship-observe-simulate-repeat loop also lets them test open-source models against frontier models in days instead of weeks, because the eval bottleneck is gone.