Speech Recognition

While my guitar gently speaks — Todd Fisher, Philo Ventures

Aug 18, 2026 · 18:35

Todd Fisher demonstrates how he built a speaking and singing guitar, walking through the audio engineering and AI stack behind it. He traces it from a Halloween Stranger Things garage door display to a JUCE plugin in Logic Pro that plays text-to-speech clips when he picks notes. Slicing words proved hard: energy gap segmentation fails on continuous speech, so he combined it with a sonority peak syllabifier and still hand-edited boundaries. For singing, he uses YIN pitch detection, synthesizes the note, and routes it through a vocoder. Because pitch-shifting samples from World and VocalSet is too heavy to run live, he pre-bakes vowel samples per fret. He also adds a microphone, Whisper, and a local LLM, letting the guitar answer audience questions like 'What is reality?'

Reachy Mini: the $300 open source robot you can actually hack — Andres Marafioti, Hugging Face

May 29, 2026 · 21:16

Andres Marafioti from Hugging Face presents Reachy Mini, a $300 open source robot shipped to 7,500 people unassembled, and explains that its most popular app is voice conversation. To make that app responsive, his team optimized Qwen3-TTS from a below-real-time 0.8x factor to 5.8x real time by adding streaming, switching to a static KV cache, and enabling CUDA graph captures, cutting time to first audio to under 200 milliseconds. The full voice pipeline runs Parakeet transcription every 150 milliseconds with partial results, feeds Qwen 3.5 27B for the LLM, and uses the optimized TTS, while infrastructure round trips are handled by a load balancer that separates LLM endpoints from conversation nodes. Marafioti argues that expensive humanoid robots limit creativity and access, whereas Reachy Mini's hackable, repairable design invites hackers, students, and dreamers to build new interactions. He demonstrates the robot's ability to take photos, show emotions, and even be pet, and emphasizes that all models and software are open source.

From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

May 20, 2026 · 21:01

Google's Cormac Brick explains how developers can build on-device AI agents using either system-level Gemini Nano via AI Core or app-level custom models via LiteRT-LM. He demonstrates a skill harness built on Gemma 4 that enables agentic tasks like restaurant roulette, running fully on-device with JavaScript UI. For fine-tuned tiny models, Function Gemma at 270M parameters improved from 46% to over 90% accuracy on eight of ten app-intent functions after synthetic data fine-tuning. The talk also covers the Eloquent transcription app, which chains two Gemma 3-based models (ASR and text polishing) under a few hundred million parameters for offline use. Key trade-offs are latency, privacy, and customization versus system integration effort.

[Full Workshop] Building Conversational AI Agents - Thor Schaeff, ElevenLabs

Jul 31, 2025 · 1:01:42

Thor Schaeff of ElevenLabs demonstrates how to build multilingual conversational AI agents using ElevenLabs' platform, which combines speech-to-text (ASR) with 99-language support, a voice library of over 5,000 voices, and language detection system tools that automatically switch between 31 languages (with plans to expand). The agent pipeline transcribes user speech, feeds it to any LLM (like GPT-4 or Gemini), and streams the response back as speech for low-latency conversations. Schaeff shows real-time language switching in Mandarin, Hindi, and English, and explains how to assign per-language voices (e.g., Chennai-accented Tamil). He addresses safety tooling—voice watermarking, live moderation, and consent verification—and discusses cost (per-minute pricing), latency mitigation via Flash models and RAG, and handling multi-language mixing within a single utterance, though accuracy degrades with more than two languages intermixed.