Jetson Nano

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Jul 25, 2026 · 21:45

Cormac Brick of Google AI Edge argues that the real constraint on edge AI is DRAM cost, not compute, making tiny models essential for widespread deployment. Small models (1–4 billion parameters) require 4–8 GB of RAM, but their quantized Gemma 2B model—at just 2.9 bits per weight—runs on a Raspberry Pi at 7.6 tokens per second and on a Qualcomm NPU at 31 tokens per second. For even lower-end devices, tiny models (50–500 million parameters) need under 2 GB of RAM and can be fine-tuned for specific tasks like voice-to-function calling, reaching over 86% reliability on ten actions. A shipped example is an offline voice dictation app that uses two fine-tuned sub-billion Gemma models to clean up ums and ahs without a subscription. The talk covers the Gemini-optimized toolchain (Lighter TLM, MediaPipe), synthetic data generation for fine-tuning, and the trade-off between zero-shot prompting in small models versus fine-tuning tiny models for reach and speed.

Your Personal Open-Source Humanoid Robot for $8,999 — JX Mo, K-Scale Labs

Jul 2, 2025 · 19:26

Jingxiang Mo, founding engineer at K-Scale Labs, introduces their open-source humanoid robots: the 5-foot K-Bot (pre-order for $8,999, delivered by October) and the 1.5-foot Z-Bot, both fully open-source from hardware to ML models. The K-Bot uses MIT CHiTA actuators, provides up to 250 TOPS compute, runs an RL-based whole-body controller trained with MJX in 1-2 hours, and offers a Python/Rust SDK (pip install kos) with a digital twin simulator for rapid development. Mo emphasizes modularity—swappable end-effectors and upgradable heads—and positions K-Scale as the first US consumer humanoid robotics company, targeting developers and households. The robots are significantly cheaper than competitors (Tesla Optimus at ~60K, Unitree at 40K) and include VR teleoperation, OTA software updates, and a bimonthly hackathon community.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.