Prompt Engineering

How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

Jul 24, 2026 · 19:29

Preetika Bhateja, Chris Souza, and Daniel Bump from Google share lessons from building evals for a seed-asset agent that turns messy YouTube ad creatives into clean assets. They argue that reliable agent behavior emerges from a loop of prompts, evals, iteration, and feedback — not just prompting alone. The team recommends starting small with intuition-based 'vibing' to understand failure patterns before scaling to human raters or LLM-as-judge. Using clear rubrics, obtaining rater explanations, and analyzing agent trace logs help uncover why failures occur, such as the agent removing disclaimers despite explicit instructions. They emphasize focusing on patterns across multiple examples rather than isolated failures, and investing in online evals with production data to keep evaluations representative. The takeaway: a good eval system evolves with the product, requires curated golden sets, and needs clear launch criteria to distinguish acceptable trade-offs from critical regressions.

The Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest & Isaac Miller

Jul 23, 2026 · 17:11

Maxime Rivest and Isaac Miller, core contributors to the open-source DSPy framework, argue that AI programs should separate task specification from model implementation using signatures—fixed input/output contracts. They detail three components of a task: instructions (what should happen), code (constraints), and evals (what good looks like), enabling automatic optimization. Enterprise case studies like Shopify achieving 550x cost reduction by swapping expensive models for cheap ones while keeping evals constant demonstrate practical gains. DSPy 3.5 and 4.0 introduce new techniques including Recursive Language Models (RLMs) for long-context tasks, DSPy.flex for learning harnesses that generate code, and Qualitative Learning for converting production feedback into evals. The speakers emphasize that even with AGI, models will need to learn business-specific context through this last-mile learning approach.

Claude Fable, Claude Tag, and Anthropic's Culture — Cat Wu & Thariq Shihipar ft Simon Willison

Jul 15, 2026 · 51:30

This episode features Anthropic's Cat Wu and Thariq Shihipar discussing Claude Code, Claude Tag, and Claude Fable, arguing that these tools have shifted engineering from slow spec-driven processes to rapid, ambitious building. Thariq notes that with each model generation, delegation increased, and Claude Fable now enables one-shot features. Cat says engineers now need product taste over execution, as timelines shrink from six months to a week. Claude Tag, a proactive multiplayer agent, lands 65% of product engineering PRs internally by monitoring channels and remembering preferences. The team reduced Claude Code's system prompt by 80% for frontier models by removing examples and hard constraints, relying on model judgment. Auto mode, used internally since January, mitigates prompt injection through thousands of evals and Sonnet classifiers. Cultural hacks include default-public channels and a 'don't negotiate against ourselves' mindset, leading to ambitious builds like Thariq's Claude-powered video editing and a Street Fighter game.

Field Guide to Fable — Thariq Shihipar, Anthropic

Jul 6, 2026 · 19:28

Thariq Shihipar of Anthropic discusses Fable and argues models improve in "spiky" ways: a chat model fails to list Pokémon ending in "aw" (Croconaw and Dreadnaw), but Claude Code fetches and filters the list in seconds – a gap he calls "capability overhang." He explains that to unlock Fable, Claude Code cut 80% of its system prompt because heavy instructions now constrain a more imaginative model, and the "ask user question" tool evolved from barely working under Opus 4 to generating embedded HTML questionnaires. He shares techniques like blind-spot passes and interviews to surface unknown unknowns, and reflects on the grief of moving from hand-coded programming to agentic workflows. Shihipar urges engineers to reject trade-offs – "good, fast, cheap: pick three" – and instead demand all three, citing a four-hour keynote deck built with Fable as proof that agents can deliver ambitious work faster.

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

Stop Writing Tone Instructions. Layer Them. - Isadora Martin-Dye, Isadora & Co

Jun 26, 2026 · 20:57

Isadora Martin-Dye, founder of Isadora & Co and builder of AI for a wedding venue, a personal companion, and a missing persons tool, argues brand voice requires a four-layer architecture instead of a single system prompt. The layers are immutable identity (hard rules like never claiming a body), situational mode (adjusting to user's context like grief), example-anchored voice (tone guide), and a post-generation veto (deterministic check catching errors like offering a booked date). Instructions are probabilistic; the veto is deterministic. A prompt will eventually fail; the only question is whether it's caught before reaching a customer. Examples include a wedding AI that must disclose it is AI and not say "I'd love to show you around," and a missing persons tool that must never use "confirmed" or "matched." She notes that multi-tenant voice leaks require identity to never have a default. The veto layer is the cheapest and most critical safeguard.

Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

May 25, 2026 · 18:54

Phil Hetzel of Braintrust argues that generative AI development should not be isolated to data scientists or ML engineers, but instead requires a diverse team including product engineers, systems engineers, and non-technical domain experts. Because models are already built by OpenAI and Anthropic, the remaining work is prompt and context engineering, distributed systems, human annotation, and functional evaluation—not traditional training pipelines. Data scientists add value through rigorous testing and LLM-as-judge evaluation, but must move beyond precision/recall toward broader functional metrics. Traditional enterprises often mistakenly hand GenAI to ML platform teams, while AI natives use small cross-functional groups with closer problem proximity. The ideal mix combines technical roles for implementation and system design with non-technical experts for prompt engineering and human annotation, keeping the agent relevant through continuous feedback.

Personalization in the Era of LLMs - Shivam Verma, Spotify

May 19, 2026 · 20:12

Shivam Verma of Spotify's AI foundation explains how Spotify personalizes recommendations using open-weight LLMs, user embeddings from 750 million accounts, and Semantic IDs that compress each of 100 million tracks into six hierarchical tokens, enabling autoregressive next-item prediction. Three components: transformer-based user embeddings, Semantic IDs (e.g., Ariana Grande and Bruno Mars share first two tokens as pop artists), and a soft tokenization layer projecting user vectors into the LLM's token space for personalization without fine-tuning. The system is productionized for podcast next-episode recommendations. Verma also introduces Taste Profile, allowing users to edit their taste profile via natural language for steerable recommendations. This moves from traditional multi-stage pipelines to a unified generative model.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Jan 6, 2026 · 52:08

SallyAnn DeLucia and Fuad Ali from Arize present prompt learning, a method to continuously improve AI prompts by using feedback from evals and human annotations. They argue that agent failures often stem from weak instructions rather than weak models, and that adding rules to system prompts can yield 15% improvements on SWE-Bench without fine-tuning or architecture changes. The talk compares prompt learning to GEBA, noting it achieves better results in fewer loops by leveraging text explanations. A case study shows Kline's performance improved from 30% to 45% on SWE-Bench-Lite after optimizing prompts. The workshop demonstrates building an optimization loop that ingests input-output pairs, evaluates them, and iteratively refines the system prompt using OpenAI.

From Arc to Dia: Lessons learned building AI Browsers – Samir Mody, The Browser Company of New York

Dec 19, 2025 · 17:48

Samir Mody, Head of AI Engineering at The Browser Company, explains how the team built Dia, an AI-native browser, after learning from their earlier browser Arc. He details three core lessons: optimizing tools for faster iteration by embedding prompt editors into the product itself; treating model behavior as a craft through a dedicated team formed after a strategy and ops employee rewrote all prompts in a weekend; and designing AI security as an emergent property, using confirmation steps for features like autofill to mitigate prompt injections despite the 'lethal trifecta' of private data, untrusted content, and external communication. The talk emphasizes that building an AI product required not just a technology shift, but a company-wide transformation in hiring, training, and collaboration.

How BlackRock Builds Custom Knowledge Apps at Scale — Vaibhav Page & Infant Vasanth, BlackRock

Aug 23, 2025 · 18:47

Infant Vasanth and Vaibhav Page of BlackRock explain how the world's largest asset manager scales custom AI knowledge apps for investment operations using a modular, Kubernetes-native framework. They detail challenges like prompt engineering for complex financial instruments, choosing LLM strategies (RAG, chain-of-thought), and deploying apps with cost control and compliance. Their sandbox lets domain experts rapidly iterate on extraction templates with inter-field dependencies and validations, while the app factory turns definitions into production apps. Key takeaways: invest in prompt engineering for domain experts, educate on LLM strategies, evaluate ROI, and design for human-in-the-loop in regulated environments. The framework compressed app development from eight months to days for use cases like setting up securities after IPOs or stock splits.

On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

Aug 6, 2025 · 19:12

Omar Khattab, research scientist at Databricks and creator of DSPy, argues that engineering AI systems that endure the rapid pace of model and technique changes requires decoupling high-level task definitions from low-level, swappable components. He reinterprets the Bitter Lesson—that scaling search and learning beats hand-coded domain knowledge—as a warning against premature optimization, which he equates to hard-coding at lower abstraction levels than justified. Khattab criticizes prompts as a poor programming abstraction that entangles task specs, formatting, and model-specific tricks, violating separation of concerns. Instead, he advocates for investing in three orthogonal concerns: natural language specs for things that can't be otherwise expressed, evals to capture true objectives, and code for reliable structure and control flow. DSPy realizes this philosophy through declarative signatures that define what an AI system should do, independent of which LLM, inference strategy (e.g., chain-of-thought, agent), or optimizer (e.g., reinforcement learning, prompt optimization) is used. The takeaway: avoid hand engineering at lower levels than current abstractions allow, and ride…

POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent

Jul 23, 2025 · 19:16

Randall Hunt from Caylent shares hard lessons from over 200 enterprise GenAI deployments, arguing that evals, embeddings, and prompt engineering matter far more than fine-tuning. He emphasizes that speed and UX are critical; a slow inference kills adoption, while techniques like generative UI and caching can mitigate latency. Hunt details real-world examples: using audio amplitude spectrographs for sports highlight reels, pooling multimodal embeddings for nature footage search, and noting that nurses prefer chat over voice bots in noisy hospitals. He reports zero regressions moving from Claude 3.7 to 4, and advises optimizing context and economics, such as leveraging Amazon Bedrock batch for 50% cost reduction. The talk underscores that knowing your end customer and minimizing irrelevant context are key to production success.

Top Ten Challenges to Reach AGI — Stephen Chin, Andreas Kollegger

Jul 22, 2025 · 4:23

Stephen Chin and Andreas Kollegger, curators of the GraphRAG track at AI Engineer World's Fair, use ten sci-fi memes to expose top challenges to AGI—from Memento's short-term memory as prompt engineering to Skynet's unintended consequences, the Matrix's agent simulation, Hal's trust issues, emotions as bug/feature, Frankenstein's creator obligations, Terminator's time travel, Star Wars' language nuance, the Borg's assimilation, and Deep Thought's right questions. They argue graphs and graph technology can solve some of these challenges, inviting the audience to the GraphRAG track to learn which ones. The episode delivers a playful yet pointed framework connecting pop culture to core AGI obstacles.

Prompt Engineering and AI Red Teaming — Sander Schulhoff, HackAPrompt/LearnPrompting

Jul 14, 2025 · 2:01:05

Sander Schulhoff, creator of Learn Prompting and HackAPrompt, argues prompt engineering remains vital despite claims of its demise, drawing on his systematic review of 1,500+ papers for 'The Prompt Report.' He covers advanced techniques including chain-of-thought, decomposition, ensembling, and few-shot prompting, noting that role prompting is ineffective for accuracy-based tasks and that example ordering can swing performance by 50%. He then explains AI red teaming, distinguishing jailbreaking from prompt injection, and warns that system prompts and guardrails cannot prevent attacks—even simple obfuscation like base64 or typos still works. He highlights the critical unsolved problem of agentic security, where agents with real-world actions are easily tricked, and introduces a live competition at the conference to gather more attack data. His key takeaway: AI security is fundamentally harder than classical cybersecurity because 'you cannot patch a brain.'

Unlocking AI Powered DevOps Within Your Organization — Jon Peck, GitHub

Jun 27, 2025 · 22:13

Jon Peck, a developer advocate at GitHub, explains how organizations can unlock AI-powered DevOps by integrating tools like GitHub Copilot into their workflows, achieving up to 1.5x feature throughput and 30% average improvement in successful builds. He emphasizes starting with brownfield (existing code) rather than greenfield, using agent mode for iterative planning and narrowed context, and codifying team best practices via copilot-instructions.md files. Peck also covers governance: safety, privacy, org-wide policies, and excluding sensitive files. Autonomous AI use cases include auto-generating PR descriptions, code review, and assigning issues to Copilot, always keeping a human in the loop with isolated branches. Finally, MCP servers allow agents to interact with GitHub and other systems, automating commits and PR creation.

Prompt Engineering is Dead — Nir Gazit, Traceloop

Jun 27, 2025 · 14:19

Nir Gazit, CEO of Traceloop, argues that manual prompt engineering is ineffective and demonstrates an automated, test-driven alternative. He improves a RAG-based documentation chatbot by 5x without hand-tuning prompts, using an LLM-as-a-judge evaluator with 20 example questions (each with 3 expected facts) and a CrewAI agent that iteratively optimizes the prompt based on evaluation scores. The initial score of 0.4 jumps to 0.9 after just two iterations. Gazit warns of overfitting to the small dataset and suggests splitting into train/test sets. The approach replaces manual tweaking with a 'gradient ascent' on evaluators, effectively vibe-engineering prompts.

The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Jun 3, 2025 · 20:47

Travis Frisinger argues that large language models are not intelligent but coherent, introducing 'coherence reconstruction' as a mental model for understanding their utility. He explains that hallucinations are a feature, not a bug, as models fill gaps predictably to maintain pattern completion. Prompts act as force vectors navigating latent space, and the key to reliable outputs is a 'frame, generate, judge, iterate' loop. Frisinger presents a three-layer model: latent space, execution layer (tools and RAG), and conversational interface. He advises engineers to design for emergence rather than control, using dense context as anchors to steer generation, and to watch for breakdowns in tone as early signs of lost coherence.

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Jun 3, 2025 · 25:09

Andrew Thompson, CTO of Orbital, introduces the concept of Prompt-Tax: the hidden cost of migrating prompts when upgrading AI models. He shares how his agentic product automates real estate due diligence, growing from <1B to 20B monthly tokens and zero to multiple seven-figure ARR over 18 months. Key tactics include optimizing for prompting over fine-tuning, using domain experts (ex-lawyers) to write prompts, and relying on vibes over formal evals. Thompson advocates 'betting on the model'—shipping new frontier models immediately and fixing regressions on the fly using progressive rollouts and rapid feedback loops. A clip from Demis Hassabis underscores the unique challenge of evolving tech stacks. The episode concludes with questions on whether evals or progressive delivery scale to manage Prompt-Tax.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

Lets Build An Agent from Scratch — Kam Lasater

Feb 22, 2025 · 19:29

Kam Lasater demonstrates that an AI agent is simply an LLM paired with memory, planning, tools, and a while loop, and shows how to build one from scratch in incremental steps. He starts with a basic LLM call (step 0), adds a condition using an LLM as judge to check if an answer is complete (step 1), then incorporates a tool for web search via SERP AI (step 2). After refactoring for parallel tool calls (step 3), he introduces a to-do list for planning and tracking progress, along with a browse-web tool (step 4). The agent uses a to-do list to plan, execute tasks like searching and browsing, and then checks completion via LLM as judge. Lasater illustrates with examples: finding the average wind speed of a swallow, buying a hoodie near Times Square, and planning a date in Ardmore. He emphasizes that the simplest agent emerges naturally from combining these components, and encourages running and breaking the provided code.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Claude plays Minecraft!

Feb 15, 2025 · 18:16

Derek from AWS demonstrates building Rocky, an autonomous Minecraft agent using Amazon Bedrock, Claude 3 Haiku, and serverless AWS services, proving agentic workflows enable AI to reason and act beyond chatbots. He explains the architecture: Minecraft server on ECS with Mineflare bot framework, agents for Bedrock with return of control, and prompt engineering for tasks like digging and building. In a live demo, Rocky jumps, finds players, digs a hole, and builds a double-decker couch from a chat command, inferring parameters from natural language. The code is open-source and built with CDK/CloudFormation, emphasizing that agentic AI can operate in complex 3D environments through tool orchestration and managed prompt design.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

GitHub Copilot: The World's Most Widely Adopted AI Developer Tool

Feb 6, 2025 · 29:49

GitHub Senior DevOps Advocate Dave Bernason demonstrates how GitHub Copilot has evolved from an AI code generator to a comprehensive developer assistant with chat, enterprise knowledge bases, pull request summaries, and third-party extensions, emphasizing that Copilot keeps developers in flow and requires human oversight. He shows Copilot Chat explaining code, refactoring, debugging, and generating unit tests. Copilot Enterprise adds Bing search, knowledge bases from Markdown files, and repo indexing for accurate answers, exemplified by updating a hard-coded sales tax function to use the Avalara API. Copilot Extensions integrate third-party tools like Octopus Deploy for deployment dashboards within chat. Bernason also covers prompt engineering tips—specificity improves suggestions—and highlights Copilot's support for any language, even COBOL, for modernization.

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Feb 5, 2025 · 9:33

Niklas Nielsen, CTO and co-founder of Log10, introduces llmeval, a command-line tool that enables teams to ship reliable LLM applications by evaluating and testing prompts and configurations. The tool initializes with four lines of code, uses Meta's Hydra for configurable test structures, and runs multiple samples (default 5) per test to assess stability. Nielsen demonstrates prompt engineering for a math problem, showing how stripping spaces or adding instructions like 'only return the answer' affects strict pass/fail results across Claude, GPT-4, and GPT-3.5. Advanced use cases include testing tools for Python code generation and model-based evaluation, where a larger model grades outputs on criteria like mermaid diagram quality. He notes pitfalls: models favor their own output and struggle with point scores. Log10 addresses this by bridging model-based and human feedback—collecting prior human ratings to train an 'auto-John' that pre-fills reviews for new completions.

Prompt Engineering Tactics: Dan Cleary

Feb 5, 2025 · 5:12

Dan Cleary, co-founder of PromptHub, presents three research-backed prompt engineering tactics to improve LLM output consistency and reduce hallucinations. Multipersona prompting, from University of Illinois, calls on multiple AI agents to collaborate on complex tasks like writing a book. The "according to" method, from Johns Hopkins, grounds prompts to a specific source (e.g., "according to Wikipedia") and can reduce hallucinations by up to 20%. The motion prompt, from Microsoft, adds emotional stimuli at the end of prompts, improving output accuracy by 8% to 115% depending on the task. These tactics are available as templates on PromptHub. Cleary emphasizes that even small changes in prompts can have outsized effects, crucial for maintaining user trust in AI-integrated products.

Scaling AI in Education: A Khanmigo case study: Shawn Jansepar

Feb 5, 2025 · 22:39

Shawn Jansepar, Khan Academy's Director of Engineering, details building Khanmigo, an AI tutor and teacher assistant on GPT-4 with OpenAI, arguing generative AI can democratize one-on-one tutoring. He describes a rapid prototyping culture that launched Khanmigo in three months via a company-wide hackathon, replacing traditional agile with a prototype-to-beta-to-launch framework. Technical challenges include math accuracy through a math agent and chain-of-thought prompting, refactoring prompts into a component architecture for testing, and managing scale using multiple models and dedicated Azure compute. Jansepar highlights ethical design: a Socratic tutor that avoids answers, teacher moderation, and a writing coach with revision history. Khanmigo now has 200,000 paid users, half in school districts, with teacher tools sponsored by Microsoft free to US teachers. Plans include releasing a math tutoring benchmark and evaluating smaller models like Phi-3.

Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald

Oct 23, 2024 · 22:36

Lukas Biewald, founder of Weights and Biases, shares lessons from productionizing GenAI models, emphasizing that while AI is easy to demo, it is hard to productionize. He illustrates this with a personal project building a custom Alexa-like device using LLaMA and Whisper, where accuracy improved from 0% to 98% through prompt engineering, switching to Mistral, and fine-tuning with QLoRA. Biewald argues that tracking experiments (including failures) is critical for reproducibility and collaboration, and that a robust evaluation framework—beyond 'testing by vibes'—is essential for iterating and shipping v2. He notes that 70% of the audience had LLM apps in production, yet many lack solid evaluations, and recommends starting with lightweight prototypes and incorporating end-user feedback.

No more bad outputs with structured generation: Remi Louf

Oct 14, 2024 · 15:32

Rémi Louf, CEO of .txt and co-maintainer of Outlines, argues that structured generation—guiding LLMs to produce valid regex, JSON, or context-free grammar outputs—eliminates parsing errors and hallucinations while adding negligible overhead. Outlines masks tokens that violate the target structure, enabling Mistral 7B to achieve 99.9% valid JSON (vs. 17% without). It also accelerates inference: a chatty 50-token ChatGPT answer shrinks to 8 tokens, and structured generation boosts open models beyond GPT-4—Phi-3 Medium hits 96.5% on Berkeley function calling (GPT-4 gets 93.5%). With one shot matching eight shots in accuracy, Louf contends that most text is structured and that structured generation should be the default for non-chatbot workflows.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.

Building with Anthropic Claude: Prompt Workshop with Zack Witten

Aug 17, 2024 · 1:34:56

In this live prompt engineering workshop, Anthropic's Zack Witten and Jamie Neuwirth demonstrate techniques for improving Claude prompts in real time, using the Anthropic console. They show that XML tags clearly separate prompt sections, that placing instructions after information improves adherence, and that prefilling assistant responses with opening JSON or tags reliably forces JSON output without preamble. Witten advises using stop sequences and code to handle formatting rather than over-prompting, and emphasizes that few-shot examples—especially contrastive pairs with reasoning—drive more improvement than any other technique. The workshop covers role-playing multiple personas by routing via code, mitigating hallucinations by extracting quotes before summarization, and grading translations with chain-of-thought and fine-grained examples.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD

Jan 23, 2024 · 7:30

Lars Grammel presents Storyteller, a kid's story generator built with TypeScript and his ModelFusion library, which orchestrates multiple AI models to turn voice input into narrated audio stories with images. The app transcribes speech via OpenAI Whisper in 1.5 seconds, generates a story outline with GPT-3 TurboInstruct in 4 seconds, and then runs title, image, and audio generation in parallel. For consistency, it uses GPT-4 with a low temperature to produce structured story passages, and streams partial results so narration can begin before the full story is ready—addressing the 1.5-minute full generation time. Voices are selected by embedding speaker descriptions and retrieving matching ElevenLabs voices filtered by gender. Image prompts are extracted from the story via GPT-4 and rendered with Stability AI's Stable Diffusion XL. The result is a responsive, multi-modal app that lets users listen while the server continues processing.

Pragmatic AI with TypeChat: Daniel Rosenwasser

Nov 14, 2023 · 18:34

Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use.

The AI Evolution: Mario Rodriguez, GitHub

Nov 7, 2023 · 19:32

Mario Rodriguez, VP of Product at GitHub, recounts the history and future of GitHub Copilot, the first at-scale AI programmer. Copilot now serves over 20,000 organizations and 1 million-plus developers, generating over $100M in ARR, with 46% of code written via completions. He shares insider lessons: ghost text, low latency under 100 milliseconds, and prompt engineering were key to success. He warns that 'syntax is not software' and that global presence and offline/online scorecards are essential at scale. Looking ahead, Rodriguez envisions moving from procedures to goals and constraints, enabling AI to reason on code, and designing immersive UIs for human-AI collaboration. He concludes that GitHub has evolved from a version control system into an end-to-end platform infused with AI.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Pydantic is all you need: Jason Liu

Nov 1, 2023 · 17:55

Jason Liu argues that Pydantic is the key to building reliable LLM applications by using structured prompting with OpenAI function calling. He introduces Instructor, a library that patches OpenAI's API to return Pydantic objects instead of raw JSON, ensuring type safety and validation. Liu demonstrates how Pydantic's 70 million downloads make it a trusted tool for defining data models with type hints, field validators, and even LLM-powered validators that catch errors like "don't say mean things" and retry via max retries. He shows concrete examples: decomposing user queries into structured search objects for RAG, generating a query plan DAG with parallel dependencies, extracting knowledge graphs for visualization, and verifying facts by requiring substring quotes from source text. Liu emphasizes that structuring prompts as code moves development from string manipulation to domain modeling, enabling cleaner, more maintainable systems that integrate easily with existing software.

Principles for Prompt Engineering - Karina Nguyen (Claude Instant @ Anthropic)

Oct 20, 2023 · 55:31

Karina Nguyen, an engineer on Anthropic's Claude, presents principles for effective prompt engineering, treating it as a creative writing process requiring iteration. She advises using XML tags, placing instructions at the end of long prompts (improving accuracy), and decomposing questions to improve faithfulness over chain-of-thought. She covers reducing hallucinations by asking Claude to hedge or quote sources, and using self-consistency and contrastive examples for labeling. Nguyen explains generating evaluation datasets with Claude by splitting documents into multiple-choice questions, and notes that for long-document QA, asking at the end outperforms the beginning. She distinguishes Claude (larger, smarter) from Claude Instant (cheaper, faster, better at math and code), and predicts prompt engineering will remain essential for complex tasks.