All episodes

Lets Build An Agent from Scratch — Kam Lasater

Feb 22, 2025 · 19:29

Kam Lasater demonstrates that an AI agent is simply an LLM paired with memory, planning, tools, and a while loop, and shows how to build one from scratch in incremental steps. He starts with a basic LLM call (step 0), adds a condition using an LLM as judge to check if an answer is complete (step 1), then incorporates a tool for web search via SERP AI (step 2). After refactoring for parallel tool calls (step 3), he introduces a to-do list for planning and tracking progress, along with a browse-web tool (step 4). The agent uses a to-do list to plan, execute tasks like searching and browsing, and then checks completion via LLM as judge. Lasater illustrates with examples: finding the average wind speed of a swallow, buying a hoodie near Times Square, and planning a date in Ardmore. He emphasizes that the simplest agent emerges naturally from combining these components, and encourages running and breaking the provided code.

Building Multi agent Systems with Finite State Machines

Feb 22, 2025 · 17:11

Adam Terlson argues that finite state machines (state charts) combined with the Actor model provide a structured, reliable foundation for building AI agents, addressing the predictability, observability, and control that LLMs lack. He introduces five agentic patterns—tool use, human-in-the-loop, feedback, collaboration, orchestration, and chartering—each demonstrated with concrete examples like a recipe-writing agent that checks inventory, a two-agent feedback loop for nutritional critique, and a multi-agent team delivering a meal. For orchestration, an LLM directs state transitions via event emitters, while chartering lets the LLM generate the entire state chart from a privacy-removal process. Terlson uses XState library for implementations and points to his demo 'Red versus Blue' for further exploration.

Reverse Conway's law and GenAI: How agents will take over the organisation - Patrick Debois

Feb 22, 2025 · 28:12

Patrick Debois argues that generative AI will reverse Conway's law, reshaping organizations around agents instead of human teams. He traces a progression from AI as a copilot to a team member, then a peer, and eventually a manager, with each stage unbundling human tasks and shrinking team sizes. Debois cites Amazon's AI pricing glitch as a reminder that humans remain needed for failure cleanup, and notes that LLMs mimic human collaboration behaviors, as shown in a multi-agent simulation. He warns of agent toxic behavior and the need for guardrails akin to human codes of conduct, while speculating that companies may replace SaaS with internally built AI services, making performance reviews and ROI calculations for agents inevitable. Debois advises engineers to focus on building the AI that builds their current work, not the work itself.

Your AI Agent Isn't an Engineer: The Art of Thoughtful Anthropomorphism

Feb 22, 2025 · 20:20

Rizel Scarlet, a staff developer advocate at Block, argues that marketing AI agents as "software engineers" creates unrealistic expectations and alienates developers. She explains that such anthropomorphism—attributing human traits to AI—stems from lazy marketing to executives, but it backfires by making developers feel threatened and discrediting products when AI fails to meet human-level performance. Citing a Wired article, she notes 52% of game companies have adopted generative AI, yet 30% of surveyed developers expressed negative sentiment. To build trust, she proposes a framework: understand how AI works (e.g., the agentic loop), use non-human names like "copilot" (pioneered by GitHub), focus on augmentation over replacement, be transparent, give developers control, show real demos (including live troubleshooting), provide clear documentation, and foster open collaboration (e.g., Cursor.Directory). The episode reframes AI as a tool to work in parallel, not a replacement.

Where AI is superhuman: The right jobs to automate with LLMs

Feb 22, 2025 · 11:53

Andy Treadman, partner at Theory Ventures, argues that LLMs are already superhuman in high-volume, low-complexity jobs, making them the prime targets for full automation. He breaks down LLM capabilities into transformation, synthesis, and reasoning, and maps jobs on a spectrum of volume and complexity. At the high-volume end, he says LLMs can be 10x better than humans because scale is the challenge, not reliability. He highlights two portfolio companies: Dropzone AI automates security alert investigations, performing better than rules-based systems and covering 24/7, while Amp personalizes customer engagement at scale, discovering new cohorts like late-night snackers. Treadman predicts that organizations will shrink and invert from pyramids to diamonds, with fewer entry-level roles. He advises founders to consider technology-problem fit and other factors when building AI workflow automation.

How Coding Agents change Software Development Forever - Hailong Zhang

Feb 22, 2025 · 8:50

Hailong Zhang presents how coding agents, particularly the unit test agent Gru, transform software development by automating routine tasks while humans focus on creative work. In the future workflow, synchronous agents like GitHub Copilot and Cursor assist in real-time, while asynchronous agents like Gru autonomously handle tasks such as writing tests, fixing bugs, and submitting pull requests. Gru, built to boost unit tests, detects code changes from pull requests, generates and runs tests, and submits a new PR with a summary and coverage improvement. In production, over 50% of Gru's PRs are accepted by humans, and it handles 80% of unit tests in its own repo, making it the top contributor. To build such agents, Zhang emphasizes defining a clear problem (e.g., unit tests, not generic software engineering), creating datasets and evaluation harnesses, selecting and fine-tuning LLMs per stage, building task-specific context from environment data, and developing an agent operating system to reuse common infrastructure across different tasks.

The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Feb 22, 2025 · 26:19

Almog Baku introduces the LLM Triangle Principles for building production-ready AI applications, arguing that LLMs should be treated like interns guided by Standard Operating Procedures (SOPs). He breaks down the triangle into three components: the foundation model, engineering techniques, and contextual data, all directed by SOPs derived from expert interviews. Baku contrasts autonomous agents—creative but unpredictable—with handcrafted LLM-native architectures that offer sustainable quality, recommending scoped autonomy. He advises starting with large models to collect data and optimize incrementally, rather than fine-tuning from day one. Data is paramount: 'show, don't tell' via few-shot examples, and balance context to avoid the 'needles in a haystack' problem. The framework aims to turn demos into robust production systems.

Cohere: Building enterprise LLM agents that work (Shaan Desai)

Feb 22, 2025 · 18:29

Shaan Desai, a machine learning engineer at Cohere, presents key strategies for building enterprise LLM agents that are scalable, safe, and seamless. He recommends using native or LangGraph frameworks for high observability in large-scale agents, while CrewAI or AutoGen suit quick proofs of concept. Core insights include starting with a single LLM and a handful of tools, simplifying tool specifications with clear descriptions and sharp examples, and caching chat history to prevent hallucinations beyond 20 turns. For multi-agent setups, the router needs clear routing instructions for edge cases, and sub-agents should be constrained to independent tasks. Safety is paramount, with human-in-the-loop triggered before or after tool calls based on codified rules. Evaluation uses a golden set of ground truth queries, expected tool calls, and outputs. Failure mitigation ranges from prompt engineering for low-severity issues to targeted annotation datasets for 10–20% failure rates and synthetic data fine-tuning for high failure rates. Cohere packages these learnings into NORTH, a single container deployment with RAG, vector DBs, and connectivity to Gmail, Outlook, Drive, and Slack,…

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Patrick Dougherty: How to Build AI Agents that Actually Work

Feb 22, 2025 · 17:44

Patrick Dougherty, co-founder and CTO of Rosco (acquired by Klarity in 2024), argues that building effective AI agents requires prioritizing reasoning over inherent model knowledge, iterating on the Agent-Computer Interface (ACI) rather than fine-tuning or relying on frameworks. He defines an agent as a system that takes directions, calls tools, and autonomously reasons—not a prompt chain. His team abandoned RAG-style knowledge injection for discrete tool calls (e.g., SEARCH TABLES, GET TABLE DETAIL), finding that GPT-4o often hallucinates impossible SQL queries while o1 correctly identifies missing data. Switching tool-response formats—Markdown to JSON for GPT-4o, JSON to XML for Claude—dramatically improved accuracy. For production, he warns against over-dependence on abstractions like LangGraph due to security needs (e.g., OAuth credential cascading). In multi-agent systems, a manager agent should delegate to 5–8 specialized worker agents, incentivized by the overall goal, to avoid loops. The true moat, he concludes, is the ecosystem—UX, connections, and security—not system prompts.

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

Keynote: Why people think "agent" is a buzzword but it isn't

Feb 22, 2025 · 28:07

Chip Huyen argues that agents are not a buzzword but a practical yet hard technology, facing three core challenges: the curve of complexity, tool use translation, and context management. Even simple queries require multiple steps, and models' success rates drop rapidly past 5 steps—newer reasoning models like DeepSeek R1 are pushing the boundary, but most still fail after 10. Tool use requires translating ambiguous natural language to precise API calls, worsened by poor documentation; she advises narrow functions and asking for clarifications. Context is another bottleneck: agents must juggle instructions, tool docs, and outputs, often exceeding a model's efficient context (many hallucinate beyond 30K tokens), necessitating external memory like RAG. Her benchmark shows planning-specialized models struggle with long context and vice versa, and she recommends breaking tasks into subtasks and using test-time compute scaling.

Personality Driven Development: Exploring the Frontier of Agents with Attitude

Feb 17, 2025 · 18:00

Ben, from Perpetual, argues that anthropomorphizing AI agents—giving them personalities, forms, and preferences—transforms user adoption and engagement, despite significant challenges. He presents examples like Tech Lead, an artichoke recruiter, and hamster teams, showing how form factors create instant understanding and enable price anchoring (e.g., 1/20th the cost of a junior employee). Benefits include easy problem decomposition (specialized agents outperform generalized), branding, and fun. However, downsides include reinforcing stereotypes (100% of generated software engineers appear male), raised performance expectations, difficulty rebranding, distraction from core value, and stark reminders of job replacement—illustrated by a CEO introducing an IC as the one who 'knows what they're doing.' Ben highlights personality-driven development where every instance is bespoke, with prompt-driven preferences (e.g., 'I like Ivy League schools'), making debugging uniquely challenging.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

Claude plays Minecraft!

Feb 15, 2025 · 18:16

Derek from AWS demonstrates building Rocky, an autonomous Minecraft agent using Amazon Bedrock, Claude 3 Haiku, and serverless AWS services, proving agentic workflows enable AI to reason and act beyond chatbots. He explains the architecture: Minecraft server on ECS with Mineflare bot framework, agents for Bedrock with return of control, and prompt engineering for tasks like digging and building. In a live demo, Rocky jumps, finds players, digs a hole, and builds a double-decker couch from a chat command, inferring parameters from natural language. The code is open-source and built with CDK/CloudFormation, emphasizing that agentic AI can operate in complex 3D environments through tool orchestration and managed prompt design.

The Adversarial Path to the Personal Assistant: Sumit Agarwal

Feb 15, 2025 · 18:46

Sumit Agarwal, founder of Ario AI, discusses building a personal AI assistant that uses adversarial ETL to extract users' data from services like Google, Amazon, and Doordash, enabling immediate personalization without manual input. He announces $16M in funding and demonstrates how the assistant generates data portraits, recommends travel and routines based on personal history, and manages schedules by detecting conflicts. Agarwal shares RAG insights: avoid LLMs for trivial math, use search and pre-processed profiles, and present the right data at the right time. Ario Boost, a browser extension, lets users download their data locally in developer mode without creating an account.

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

RAG at scale: production ready GenAI apps with Azure AI Search

Feb 13, 2025 · 21:53

Pablo, a Distinguished Engineer at Microsoft, explains how Azure AI Search enables production-grade RAG applications at scale by addressing key scaling challenges: data volume, query load, workflow complexity, and diverse data types. He demonstrates the platform's hybrid retrieval combining vector search (HNSW, exhaustive), keyword search (BM25), and filters, with a two-stage system using cross-encoder re-ranking for improved quality, achieving 100ms re-ranking latency. Azure AI Search supports multi-billion vector indexes through 10–12x storage limit increases and quantization (int8, single-bit) preserving 90–95% precision, with oversampling for full-precision re-ranking. Integrated vectorization automates ingestion from Azure sources (blob, Cosmos DB, OneLake) with change tracking and format handling (PDFs, Office documents, images). OpenAI uses Azure AI Search to back ChatGPT file uploads and the Assistants API, scaling user limits 500x after Microsoft's infrastructure upgrades.

Accelerating Mixture of Experts Training With Rail Optimized InfiniBand Networking in Crusoe Cloud

Feb 12, 2025 · 17:45

Ievgen Bakulenko, product manager at Crusoe Cloud, explains how their rail-optimized InfiniBand networking accelerates training for sparse mixture of experts models. By leveraging NVIDIA's PXN feature, which allows GPUs to communicate across different rails using the internal NVSwitch in a single hop, Crusoe achieves a 50% improvement in synthetic benchmark latency and bandwidth for both small and large messages. In a real-world test fine-tuning the Mixtral model (8 feed-forward blocks, 7 billion parameters) on 240 H100 GPUs, this topology reduced training time by 14%, directly lowering cost and time-to-train. Bakulenko also outlines Crusoe's AI cloud platform, its climate-aligned mission using stranded energy, and its focus on easy-to-use infrastructure for AI engineers.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

AI Music Generation, From Prompt to Production: Phlo Young

Feb 11, 2025 · 54:34

Phlo Young, an AI engineer and musician, demystifies AI music generation by distinguishing text-to-music, audio-to-music, and style transfer, and demonstrates hands-on use of Suno and Udio to turn prompts into songs. He showcases examples like a Kanye West voice conversion, a Randy Travis AI comeback, and a BBL Drizzy meme song generated entirely from text. Phlo explains prompt engineering techniques, provides live access to paid accounts for attendees to generate unlimited music, and covers tools like Wave Tool for extracting stems. He also touches on ethical considerations and the RIAA lawsuits against Suno and Udio. The workshop equips beginners with practical skills to create and refine AI-generated music.

Giving a Voice to AI Agents: Scott Stephenson, CEO, Deepgram

Feb 10, 2025 · 13:08

Scott Stephenson, CEO of Deepgram, outlines the evolution of voice AI from slow, domain-specific systems to fast, open-ended conversational agents powered by LLMs, arguing that speed and accuracy are now solved and the key differentiator is contextual understanding. He explains that state-of-the-art speech-to-text and text-to-speech can achieve round-trip latencies under 500 milliseconds—matching human turn-taking—but notes that current systems fail to pass context between components, causing 10-20% of interactions to feel unnatural. Stephenson introduces Deepgram's 'contextual AI' vision, where models are prompted with the full conversation history, including tone, pace, and background audio, enabling the LLM to generate responses and instruct the TTS on delivery. He cautions against monolithic speech-to-speech models for enterprise use due to controllability and cost concerns, advocating for a modular stack that lets businesses optimize each component (e.g., small LLMs for simple tasks like password resets). Deepgram’s upcoming voice AI agent API integrates all components for low latency and offers $250 in free credits for experimentation.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

Fine tune 20 Llama Models in 5 Minutes: Santosh Radha

Feb 9, 2025 · 6:26

Santosh Radha, Head of Product/Research at Agnostiq, demonstrates Covalent, an open-source platform that lets users fine-tune and deploy hundreds of Llama models directly from Python without Kubernetes or Docker. By adding a single decorator to Python functions, users specify GPU requirements (e.g., H100 with 48 GB, 18-hour limit) and run them on remote compute, paying only for actual usage (e.g., 87 cents for 6 minutes on an L14, 11 cents on a V100). Covalent supports job submission, inference endpoints with custom autoscaling (e.g., scale to 10 GPUs at 9 AM daily), and automated workflows for training, evaluation, and deployment. Radha shows a workflow that iterates over 20 models, fine-tunes each, evaluates accuracy, sorts, and deploys the best—all from a Jupyter notebook with a single dispatch call. The talk, recorded at the AI Engineer World's Fair, emphasizes eliminating infrastructure overhead for accelerated compute.

The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt

Feb 9, 2025 · 18:03

Kyle Corbitt, CEO of OpenPipe, argues that most teams don't yet need fine-tuning and should start with prompted models like GPT-4. He presents a GenAI maturity curve where the trigger to fine-tune is when you hit constraints on cost, latency, or quality consistency—for example, if GPT-4 is 80-90% correct but inconsistent on the last 10-20%. Fine-tuning shifts the paradigm frontier outward, enabling models like fine-tuned LLaMA 38B to outperform GPT-4 at 1/25th the cost. The process has four steps: capture production logs to know your input distribution, prepare high-quality data (using GPT-4 outputs or iterative labeling), train with one-click tools, and evaluate with inner-loop (LLM-as-judge) and outer-loop (business metrics) evals. OpenPipe and other providers make deployment trivial via OpenAI-compatible APIs. The talk delivers a concrete decision framework and a walkthrough so any engineer can fine-tune in under an hour.

Building an AI assistant that makes phone calls [Convex Workshop]

Feb 9, 2025 · 50:53

Tom Redmond, head of DX at Convex, builds Floyd, an AI phone-calling assistant using Convex, GPT-4, Google Cloud speech-to-text, and Twilio. Floyd lets users make voice requests (e.g., calling a school to report a sick child) via a web app; transcription, context retrieval, and real-time conversation with a human on the other end happen through a reactive database architecture. Redmond explains how Convex's WebSocket-based infrastructure solves serverless latency issues, enabling live transcript streaming and status updates without polling. He demonstrates two calls (school absence and flower order), noting that latency (2–5 seconds) comes from OpenAI text-to-speech and growing prompts, with plans to use OpenAI Threads and faster TTS services. The prototype explicitly discloses it's an AI to avoid misrepresentation, and Redmond envisions AI agents efficiently handling service bookings.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

GitHub's AI Powered Security Platform: Sarah Khalife

Feb 8, 2025 · 23:45

Sarah Khalife, Principal Solutions Engineer at GitHub, details how the company is weaving generative AI into GitHub Advanced Security (GHAS), focusing on code scanning auto fix (in public beta, with a 70% success rate on vulnerability fixes within pull requests), secret scanning enhancements such as AI-generated custom regex patterns and unstructured password detection (reducing false positives), and supply chain security via Dependabot. She demonstrates how AI enables faster remediation—detecting cross-site scripting and suggesting fixes before merge—and shows the security dashboard’s coverage view, which reveals that secret scanning covers 99% of repositories while code scanning only 57%. The talk emphasizes that AI shifts security from reactive to proactive, bridging the gap between AppSec teams and developers through shared responsibility and community driven improvements.

Agentic Workflows on Vertex AI: Rukma Sen

Feb 8, 2025 · 18:06

Google Cloud's Rukma Sen argues that AI agents are the essential bridge between generative models and users, and Vertex AI provides a platform to build and deploy them with enterprise-grade safety and flexibility. She defines agents as systems with a model (brain), tools (hands), and orchestration (nervous system), covering deterministic, generative, and hybrid types. For production reliability, she advocates multi-agent architectures, giving the example of a customer service system with dispatcher, expert, and supervisor agents—and a personal anecdote where a supervisor agent kept rejecting outputs. Use cases span customer support, employee HR, knowledge agents, and voice agents for drive-throughs. Vertex AI offers 150+ models (Google, Anthropic, LLaMA) and Agent Builder from no-code to full-code, with enterprise security and data privacy.

GitHub Next Explorations: Rahul Pandita

Feb 8, 2025 · 18:18

Rahul Pandita, a researcher at GitHub Next, presents two explorations: Copilot Next Edit Suggestions and Copilot Workspace. Next Edit Suggestions extends GhostX by suggesting multi-location edits beyond the cursor, such as updating method definitions and documentation when a parameter is added. Copilot Workspace addresses developers' pain points of getting started on tasks, trusting AI output, and retaining control by offering a one-click proposal, built-in runtime verification, and an iterative, collaborative environment. Pandita demonstrates updating a MonoSpace website to add Rust syntax highlighting via a GitHub issue, showing specification generation, planning, and terminal command synthesis. He emphasizes that GitHub Next explores uncertain futures rapidly, learning from experiments like Copilot (which started as a Next project) to shape AI-augmented software engineering without waiting decades.

RAG and the MongoDB Document Model: Ben Flast

Feb 8, 2025 · 13:13

Ben Flast, Director of Product at MongoDB, explains how the MongoDB document model and Atlas Vector Search enable more efficient and scalable Retrieval Augmented Generation (RAG). He contrasts MongoDB's JSON-based documents with relational databases, highlighting that documents store application objects directly without stitching tables. Flast details MongoDB's HNSW-based vector search, which stores embeddings alongside transactional data in the same document, supporting up to 4,096 dimensions. He introduces Search Nodes, which decouple vector search infrastructure from transactional databases for independent scaling. Flast covers AI integrations with LangChain, LlamaIndex, and others, enabling features like semantic caching and chat history memory within a single database. He cites the startup 4149, which uses MongoDB to store user data, meeting notes, and vector embeddings for an AI teammate that assists with tasks by combining transactional and semantic search.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

[Full Workshop] How to add secure code interpreting in your AI app: Vasek Mlejnsky

Feb 6, 2025 · 1:48:16

Vasek Mlejnsky, CEO of E2B, demonstrates how to add secure AI code execution to any app using E2B’s Code Interpreter SDK alongside Anthropic's Sonnet 3.5 and Vercel’s AI SDK. The workshop builds an open-source version of Claude’s artifacts UI, running Python code in isolated Firecracker VMs that start in ~900ms and support custom environments via Dockerfiles. Vasek explains how E2B’s Jupyter-server-based sandbox returns stdout, stderr, runtime errors, and rich outputs like PNG charts, which are streamed to the front end using Vercel’s stream data. He covers security architecture—full root access in a VM that self-destructs on escape—and plans for AWS/GCP self-hosting and snapshot-based agent branching. The episode also addresses customizing sandboxes with private packages, mounting cloud storage (S3, GCS), and interpreting table previews as HTML or images.

GitHub Copilot: The World's Most Widely Adopted AI Developer Tool

Feb 6, 2025 · 29:49

GitHub Senior DevOps Advocate Dave Bernason demonstrates how GitHub Copilot has evolved from an AI code generator to a comprehensive developer assistant with chat, enterprise knowledge bases, pull request summaries, and third-party extensions, emphasizing that Copilot keeps developers in flow and requires human oversight. He shows Copilot Chat explaining code, refactoring, debugging, and generating unit tests. Copilot Enterprise adds Bing search, knowledge bases from Markdown files, and repo indexing for accurate answers, exemplified by updating a hard-coded sales tax function to use the Avalara API. Copilot Extensions integrate third-party tools like Octopus Deploy for deployment dashboards within chat. Bernason also covers prompt engineering tips—specificity improves suggestions—and highlights Copilot's support for any language, even COBOL, for modernization.

Substrate Launch: the API for modular AI

Feb 6, 2025 · 5:20

Rob Cheung, co-founder of Substrate, presents the platform as a new approach to building AI applications using modular intelligence rather than monolithic models. Substrate combines a developer SDK for describing computation graphs over multiple AI nodes with an inference engine optimized for running these graphs efficiently, reducing latency between nodes to microseconds—10,000 times faster than typical API dispatching. Cheung argues that modular systems are more legible, debuggable, and extensible, enabling higher quality outcomes through explicit decision trees and easier evaluation. The platform offers best-in-class JSON mode for multi-inference runs and supports modalities like image generation, speech transcription, text, embeddings, and code execution. Substrate is designed for developers to run 10-20 AI modules on a single user request at scale, solving the infrastructure problem that currently limits multi-step AI workflows.

Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI: Cedric Vidal and David Smith

Feb 6, 2025 · 1:57:58

David Smith, Cedric Vidal, and Miguel Martinez lead a hands-on workshop on building a production-level RAG-based retail copilot using Azure AI. They demonstrate how to build a chatbot backend that retrieves product information from Azure AI Search via vector embeddings and customer history from Cosmos DB, then augments the LLM prompt to generate grounded answers. The session covers using Azure AI Studio and Prompt flow to orchestrate the RAG workflow, deploying the flow as a managed endpoint, and evaluating quality with GPT-4 as a judge on metrics like relevance and groundedness. The speakers also explain the LLM Ops lifecycle for iterative improvement and compare Prompt flow with Semantic Kernel and AutoGen.

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

Which Jobs Can Be Replaced Today: Fryderyk Wiatrowski and Peter Albert

Feb 6, 2025 · 19:59

Fryderyk Wiatrowski and Peter Albert, co-founders of Zeta Labs, argue that autonomous browser agents will first replace reactive jobs—like customer support and scheduling—by automating low-leverage tasks while preserving human focus on high-leverage activities. They propose a trigger-pool system where agents react to emails, Slack, or events, requiring only approval for actions. Peter details building reliable agents: start with prompting optimized for model distribution, then add cognitive architectures (e.g., planning, scratchpads) to split tasks, and finally fine-tune with synthetic data or reinforcement learning. He advises minimizing noise in prompts, preferring text-based reasoning over images, and using language model judges to filter training data. The founders see continuous model improvement enabling agents to handle increasingly complex, proactive roles, moving toward full job replacement.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.

Ionic Launch: Opening the economy to AI agents

Feb 6, 2025 · 5:30

Justin, CEO and co-founder of Ionic Commerce, presents his company's mission to enable AI agents to participate in the economy, starting with ecommerce. He argues that the current web is built for ads, not actions, leaving agents unable to access dynamic product data like stock or shipping. Ionic solves this by partnering with hundreds of merchants, enriching their product feeds via an agentic workflow, and exposing the data through an API that agents can query in their preferred format (relational, vector, graph). The platform also adds a transaction layer so agents can complete purchases in one step using payment tokens. Justin emphasizes that merchants will pay for this service because it provides better attribution and customer relationships than Amazon's cut. Ionic's catalog already includes millions of AI-native SKUs, and developers can start today at docs.ioniccommerce.com.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.

EyeLevel Launch: Your RAG is Tripping, Here's the Real Reason Why

Feb 6, 2025 · 6:13

Benjamin Over, co-founder of EyeLevel.ai, argues that 95% of RAG hallucinations stem from content ingestion problems, not LLMs or prompts, and introduces his platform's approach to solve this. EyeLevel avoids vector databases entirely, instead creating semantic objects with auto-generated metadata and rewriting text for search and completion. Its pipeline uses a fine-tuned vision model to extract tables, figures, and text, then applies nine fine-tuned models for ingestion and search, including a re-ranking LLM. Air France uses EyeLevel to build a call-center co-pilot, achieving over 95% accuracy on hundreds of thousands of complex documents. In a study, EyeLevel reached 98% accuracy on real-world documents, outperforming popular solutions by up to 120%.

The Rise of the AI Software Engineer: Jesse Han

Feb 5, 2025 · 3:48

Jesse Han, CEO of Morph Labs, announces the Morph Code Index and RiffCoder 7B model as milestones toward the personal AI software engineer. The Code Index is a neurosymbolic code database that enables semantic search over codebases, makes entire Git history visible to coding assistants, and transmutes code into training data. It uses static analysis, vector embeddings, and graph algorithms for state-of-the-art code search, with a query language guaranteeing precise results. RiffCoder 7B, trained on data from the Code Index, is the only open-source code editing model that runs on laptops and IDEs. Morph Labs is building a platform for managing personal AI software engineers, available via waitlist. The future of software should belong to everyone, and the tools are open-source and free.