Lead with the operations gap, not the model
Small work stays on the chip; big work goes through a gate. The runtime is table stakes; signed release, staged rollout, policy, audit, and rollback are the product.
Field note 01 · September 2026
Edge AI and on-device AI trends, 2026.
What changed in the last eighteen months, what practitioners are actually running, and how the narrative reads on X, LinkedIn, GitHub, and Reddit. Written to keep our own claims honest.
1 · Five shifts
Why deployable edge AI is a 2026 problem, not a 2028 one.
| Shift | Evidence | What it means for ByteHub |
|---|---|---|
| 1–9B small language models are production-usable at INT4 | Calibration-aware 4-bit keeps roughly 95% of full-precision benchmark quality on 8B models; a 4B model fits in about 2.5 GB at 4-bit (GeniusTechLab, Jun 2026; IoT Digital Twin PLM). Liquid AI's LFM2.5 quantization-aware-distilled Q4_0 checkpoints reach about 97% of BF16 averages (Liquid AI). | The model is no longer the blocker. Packaging, signing, compatibility, and rollback are. |
| NPUs are a baseline, not a feature | Phone NPUs at tens to 100 TOPS; Copilot+ PCs require at least 40 TOPS; the embedded tier is covered by Arm Ethos, NXP Neutron (i.MX95), and Jetson Orin/Thor; AMD Ryzen AI Embedded X100 began sampling in June 2026 (Derek Molloy, 2026). | Per-target execution profiles matter more than one universal runtime. |
| Vendor runtimes ship OpenAI-compatible LLM servers on the device | NVIDIA TensorRT Edge-LLM: C++ LLM/VLM runtime for Jetson, DRIVE, and DGX Spark with a chat-completions endpoint (NVIDIA). NXP eIQ GenAI Flow with Neutron NPU acceleration for INT4 LLMs on i.MX95 — Gemma3-1B, Llama-3.2-1B, Danube-500M via the ONNX Runtime Neutron execution provider (NXP). | ByteHub builds on these and adds the lifecycle they do not ship: signed artifacts, staged rollout, policy gate, audit, rollback. |
| Industrial buyers already deploy SLMs on the floor | Rockwell's FactoryTalk Design Studio Copilot runs Nemotron-Nano-9B-v2 on HMI panels and air-gapped edge appliances; Siemens' Erlangen plant targets fully adaptive manufacturing with NVIDIA; the envelope is 20–35 W in a DIN-rail enclosure (industry report, 2026). | The initial customer profile — industrial OEMs and fleet operators on Linux devices — is buying now. |
| "Physical AI" and on-device agents are the dominant frame | Qualcomm, Arduino, and Forgis demonstrated a voice-controlled robot arm with all inference local at about 20 ms (Qualcomm, Jul 2026). Liquid AI's LFM2.5-2.6B agentic model runs under 2.5 GB at about 30 tokens per second on phones (LinkedIn). | Agents that act on equipment raise the bar for policy, approval, and audit — exactly the layer we build. |
2 · Tooling
What practitioners actually run, mid-2026.
Rule of thumb from the Jetson community: model file size plus about 1.5 GB of KV cache must fit unified RAM. Orin Nano 8 GB carries 1–3B at Q4; Orin NX 16 GB carries 7–8B; AGX Orin 64 GB carries 70B at Q4 (ProventusNova, 2026).
| Tier | Runtime | Where it wins | Where it hurts |
|---|---|---|---|
| Jetson Orin / Thor | llama.cpp (GGUF, CUDA offload) | Single-user latency and smallest footprint. On Thor, about 80 tokens per second on Gemma 4 E2B Q4_0 at 2.45 GB, versus a CPU-only LiteRT-LM run (NVIDIA forum). | One request at a time. |
| Jetson | vLLM (jetson-containers) | Multi-client OpenAI API with continuous batching; 231 tokens per second at concurrency 8 on AGX Orin 64 GB (NVIDIA forum). | 8 GB SKUs need memory tuning to avoid out-of-memory (LearnOpenCV). |
| Jetson | TensorRT-LLM / TensorRT Edge-LLM | Highest raw speed; vendor-supported for automotive and robotics. | Per-model engine build step (2026 benchmark). |
| i.MX95 / i.MX9 | eIQ GenAI Flow + ONNX Runtime Neutron EP | Wake word → speech-to-text → RAG → SLM → text-to-speech pipeline; NPU offload for INT4 LLMs; v3.0 (Mar 2026) adds a VLM demonstrator (NXP demonstrator). | Small supported model set; Neutron LLM acceleration is i.MX95 only. |
| Cross-platform | ExecuTorch, LiteRT / LiteRT-LM, Core ML, GenieX (formerly NexaSDK) | Portability across mobile and embedded NPUs. | LiteRT-LM ran CPU-only on Thor in the forum test; the ExecuTorch Neutron backend targets microcontrollers today. |
| Serving pattern | Ollama, LM Studio, OpenAI-compatible servers | Developer ergonomics. | No fleet lifecycle, no signing, no policy. |
3 · Channels
What X, LinkedIn, GitHub, and Reddit are saying.
| Channel | Signal, Q2–Q3 2026 | Read-through |
|---|---|---|
| X | "Physical AI" and "edge agents" dominate. Vendor demos (Qualcomm / Arduino / Forgis), Jetson Thor benchmark threads, LFM2.5 and Gemma 4 E2B local-agent threads, the Nexa AI acquisition by Qualcomm. | The market has accepted that models run on devices; the conversation is moving to operating them. |
| Liquid AI QAD releases; Rockwell and Siemens factory deployments; Qualcomm's "agentic AI across the compute continuum"; IoT agent platforms standardizing on 2–3B models at the edge. | Enterprise buyers talk about deployments and governance, not benchmarks. | |
| GitHub | NVIDIA/TensorRT-Edge-LLM, NXP's eIQ GenAI Flow demonstrator, dusty-nv/jetson-containers (vLLM and llama.cpp images), GGUF Q4_0 as the interchange format, NexaSDK at #1 trending before its acquisition. | Runtimes are commoditizing; the release pipeline around them is not. |
| Reddit and forums (r/LocalLLaMA, r/embedded, NVIDIA Jetson forums) | Recurring pain: JetPack / CUDA version drift, out-of-memory on 8 GB boards, garbage output from unsupported CUDA builds, "which engine for which SKU", power and thermal caps. | "It boots, it is not a product" — now for models. This is the gap ByteHub closes. |
4 · Implications
What this changes in how we describe ByteHub.
01
02
The silicon ladder is current
i.MX95 (eIQ + Neutron), Jetson Orin, and Jetson Thor on the bench and validating; an Android device runtime planned for the handhelds and operator terminals beside the fleet.
03
Provider-aware cloud plane
Buyers ask for Vertex AI, Amazon Bedrock, or Microsoft Foundry by name. The policy gate is the same; the provider is the customer's choice.
04
Use the market's frame, not its numbers
Physical AI, on-device agents, governed fleets. Never a benchmark ByteHub has not measured. The EU Cyber Resilience Act is a tailwind for traceable model releases on covered products.
Corrections welcome
Send a better source.
These notes are secondary syntheses. If a primary source contradicts a line here, we would rather fix it than defend it.