Board bring-up
NXP i.MX95 bring-up, Linux kernel, U-Boot, Device Tree, BSP development.
Deep-tech architecture
One artifact. Per-SoC profiles. A gate the model cannot open.
The model does not decide whether it may act. ByteHub policy decides what can run locally, what data may leave the device, when a cloud model may be called, and which actions require approval.
Silicon ladder
Constrained hardware profiles.
One runtime, one signed artifact format, per-SoC execution profiles. Status reflects what is on our bench today.
| Target | Status | Execution profile | Stack we build on |
|---|---|---|---|
| i.MX95-class | in hand · validating | imx95-npu-int4: Neutron NPU for supported operators, Cortex-A CPU fallback, offline-first behaviour. Low-power industrial and automotive-grade envelope. | NXP eIQ / eIQ GenAI Flow, ONNX Runtime Neutron execution provider, Yocto BSP, RAUC / SWUpdate |
| Jetson Orin | in hand · validating | orin-trt-int4: GPU-accelerated vision, multimodal, and language workloads within a defined power and thermal envelope. | JetPack, CUDA, TensorRT / TensorRT-LLM / TensorRT Edge-LLM, JetPack OTA |
| Jetson Thor | in hand · validating | thor-trt-fp8: larger local multimodal and language-model capacity for physical AI, robotics, and multi-camera inspection. Same signed artifact; a higher-capability execution profile. | JetPack 7, CUDA, TensorRT Edge-LLM, llama.cpp / vLLM evaluated |
| Android devices | planned | android-npu-int4: device runtime for rugged handhelds, operator terminals, and Android-based HMIs beside the fleet. Same artifact format, same policy gate, NPU-accelerated where the SoC allows, CPU fallback otherwise. | Android NNAPI / LiteRT, Qualcomm AI Engine Direct (Hexagon NPU), ExecuTorch |
1signed artifact formatdesign
per-SoCexecution profiles, CPU fallback always defineddesign
default-offcloud path; metadata only unless policy opens itdesign
n−1verified rollback target kept on devicedesign
Pipeline
Release engineering flow.
Device update systems are mature for OS and container artifacts. ByteHub extends release discipline to models: versioned, signed, target-compatible, staged, observable, and reversible.
Distill
A frontier teacher generates and grades task data; an open base model under ~1B parameters is fine-tuned to the device’s one skill. How specialized models are made.
Build / quantize
INT8 / INT4 packaging per target with calibration data recorded in the artifact metadata.
Evaluate
Measured on the real target: judge score on the golden set across seeds, latency, memory, power, failure recovery.
Sign
Artifact and metadata signed: compatibility rules, execution profile, policy version, evaluation result.
Verify / run
Signature checked on device; staged rollout by percentage; previous artifact retained and verified for rollback.
Observe / re-distill
Judge failures and escalations are logged; misses feed the next training set and the loop runs again.
Signed artifact · contents
- Model weights and tokenizer (quantized per profile)
- Execution profile id and compatibility rules (SoC, memory floor, runtime version)
- Policy version: what may run, what may leave, what needs approval
- Evaluation record: task set id, measured results, bench identity
- Provenance chain: build inputs, signer, timestamp
Policy gate · audit record
- Device id, tenant, artifact, profile, policy version
- Decision: local / escalate / refuse — with the reason string
- Context minimization applied; hashes only, never prompt bodies in shared logs
- Cloud provider and model id when escalated; cost estimate
- Approval id for consequential device actions
Hybrid execution
Local by default. Cloud by policy.
Edge plane
Small scope, on the silicon in the product
- Local perception, short prompts, control loops, offline continuity
- Purpose-trained specialist models with an on-device confidence and judge check; a failed check is what opens the gate
- Per-target profile picks NPU, GPU, or CPU; the artifact does not change
- Predictable behaviour when the link drops: queue, degrade, never silently fail
⏻
policy gate
default-off
minimized context
append-only audit
default-off
minimized context
append-only audit
Cloud plane
Large scope, provider-aware
- Long-context reasoning, multimodal analysis, multi-step work the device cannot hold
- Initial evaluations across Vertex AI, Amazon Bedrock, and Microsoft Foundry; ByteHub remains provider-aware for residency and procurement
- Per-tenant isolation: separate context, keys, retention; nothing crosses tenants
Evidence
Quantization evidence, measured — not quoted.
Quantization trade-offs vary by model, target hardware, runtime, and workload. ByteHub measures accuracy, latency, memory, and power across supported execution profiles and publishes the resulting compatibility matrix. Until a number is measured on our bench and validated on a customer workflow, it does not appear on this site.
| Dimension | How it is measured | Published as |
|---|---|---|
| Accuracy | Customer task set on the target, full-precision vs quantized | Delta per profile, per model |
| Latency | Time to first token and tokens/s under sustained thermal load | p50 / p95 per profile |
| Memory | Resident set incl. KV cache at the rated context length | Peak MB per profile |
| Power | Board-level draw during sustained inference | W per profile, J per token |
| Failure recovery | Rollback time and success rate under staged-rollout fault injection | Seconds, % |
2026 tooling landscape
What the field runs today, and where ByteHub fits.
Vendor and open runtimes have made on-device LLM and VLM inference routine. None of them owns the release lifecycle across a fleet. That is the layer ByteHub adds.
| Runtime | Where it is used | ByteHub relationship |
|---|---|---|
| NVIDIA TensorRT Edge-LLM | C++ LLM/VLM runtime for Jetson, DRIVE, DGX Spark; OpenAI-compatible server | Orin / Thor execution profiles build on it |
| NXP eIQ GenAI Flow + Neutron NPU | Wake word → STT → RAG → SLM → TTS on i.MX95 / i.MX9; INT4 LLMs via ONNX Runtime Neutron EP | i.MX95 execution profile builds on it |
| llama.cpp (GGUF) | Single-user, smallest footprint, CUDA offload on Jetson | Evaluated per target; packaged when measured |
| vLLM (jetson-containers) | Multi-client OpenAI API on Jetson with continuous batching | Gateway-class devices; evaluated |
| ExecuTorch, LiteRT / LiteRT-LM | Portable PyTorch / Google runtimes for mobile and embedded NPUs | Tracked for future profiles |
| JetPack OTA, RAUC, SWUpdate | OS / container update systems | Extended with signed model artifacts, compatibility rules, staged rollout, rollback |
Engineering foundation
Built by people who have operated the layers AI products usually ignore.
Embedded Linux
Yocto / BitBake layers, reproducible images, multi-SKU trees.
Secure updates
RAUC firmware updates, partitioning, signed release workflows, rollback, device debugging.
Distributed systems
DDS, messaging, fleet and device communications; edge AI integration and hardware-aware optimization.
Technical brief
Request the technical brief.
Artifact format, execution-profile plan, policy gate and audit design, and the evaluation method we use before publishing any number.