Quality
Judge score on the golden set: student vs untuned base vs teacher. Stability across at least three training seeds; a one-seed result is not shipped.
Field note 04 · September 2026
On-device distillation: how a specialist model is made.
A frontier model teaches, a small open model learns one skill, the device runs it, and a policy gate handles what the small model should not answer. This note records the method, why it works, why it fits a company of ByteHub’s size, and the public research it rests on. Product framing lives on the Models page.
1 · Method
Six steps and a loop.
Nothing is trained before the skill, a golden set, and a machine judge exist. The teacher is a frontier model behind an API; the student is an open base model under about one billion parameters; the artifact that ships is the same signed artifact EdgeRuntime already verifies.
Define the skill
One narrow, high-volume task as a task card; a golden set with known-good outputs; a rubric the judge can score.
Teacher generates
The frontier model writes answers, rationales, and scores for many inputs. Cost is tokens; it runs for hours per iteration.
Student learns
Supervised fine-tuning, knowledge distillation on the teacher’s output distribution, rationale distillation where it helps. One small GPU suffices at this size.
Quantize and sign
INT4 weights in the target’s format; packaged with profile, compatibility rules, policy version, and the evaluation record; signed.
Run with a judge
Every answer passes a confidence and judge check on the device. Pass: used locally. Fail: escalated through the policy gate with minimized context.
Flywheel
Misses are logged, corrected by teacher or reviewer, added to the set; the student is re-distilled, re-evaluated across seeds, re-signed, rolled out in stages.
2 · Roles
Each part has a job it must not exceed.
| Part | What | Where | Cost | Must never |
|---|---|---|---|---|
| Teacher | Frontier LLM via API; writes and scores trajectories | Cloud, at training time and as fallback | Tokens per iteration | Ship to the device; see raw device data outside policy |
| Student | Open base model under ~1B, tuned to one skill | On the device, inside EdgeRuntime | One small GPU to train; device compute to run | Answer outside its task card |
| Judge | Rubric plus golden set | Pipeline; light form on the device | Engineering time | Be skipped; share the student’s training seed |
| Fallback | Policy-gated escalation to a frontier model | Through the gate; provider per tenant | Tokens on misses only | Open by default; over-share context |
| Flywheel | Miss log → fix → re-distill → re-evaluate → re-sign | ModelOps control plane | Periodic training runs | Deploy without a golden-set pass and a signature |
3 · Why a narrow model is strong
Capacity on one distribution.
Mechanisms
- All parameters serve one input distribution instead of every possible one; small students can match or beat much larger general models on that task.
- Teacher rationales as supervision transfer the task with far less data than answer-only labels.
- Distillation objectives match the teacher’s output distribution; reverse-KL variants suit generative students.
- Progressive interpolation between student and teacher keeps training stable across large size gaps.
- The task lives in the weights, so the runtime prompt shrinks; long instructions can be compressed further into learned tokens when the weights are white-box.
- Evaluation sits inside the loop: golden set and judge gate every release; multi-seed checks catch distillation collapse before a model ships.
References
- Hsieh et al., Distilling Step-by-Step, ACL Findings 2023. arXiv 2305.02301
- Gu et al., MiniLLM: Knowledge Distillation of Large Language Models, 2023. arXiv 2306.08543
- Wang et al., Self-Instruct, 2022. arXiv 2212.10560
- Shing et al. (Sakana AI), TAID, ICLR 2025. arXiv 2501.16937
- Mu et al., Learning to Compress Prompts with Gist Tokens, NeurIPS 2023. arXiv 2304.08467
- Shopify Engineering, Gisting and Sidekick’s continual learning loop — public production write-ups; cited as pattern inspiration, not as ByteHub results.
- NXP, eIQ GenAI Flow product documentation and UG10166 (INT4 LLM path on the Neutron NPU).
4 · Why it fits a small company
Rent the expensive model by the token; ship the cheap one.
| Assumption | Reality |
|---|---|
| You need a large company’s proprietary specialist weights | No. Students start from public open base models and train on data generated for the customer’s task. |
| You need a hundred-GPU cluster | No. That scale describes cloud serving of a specialist at tens of millions of requests a day. Edge metrics are tokens per second, resident memory, and judge score. |
| The teacher is a GPU farm you operate | No. The teacher is a frontier API, paid in tokens, idle between iterations. |
| Training must happen in the cloud | Optional. Sub-1B students train on one local GPU; cloud GPUs for bursts. |
| A small model means a worse product | On its one task a distilled student is routinely competitive with its teacher, and it is faster, private, and offline. The gate covers the rest. |
Planning bands used internally (not measured, not promised): first useful specialist on lab boards two to four weeks after the skill and data exist; a first untuned smoke test on a CPU-only lab board within days of hardware arriving. None of these appear as claims on the product pages.
5 · Packaging per target
Same student, different container.
| Target | Status | Packaging | Runtime |
|---|---|---|---|
| i.MX95-class | in hand · validating | INT4 weights in the NPU vendor’s ONNX format; CPU fallback | NXP eIQ GenAI Flow, ONNX Runtime Neutron execution provider |
| Jetson Orin / Thor | in hand · validating | INT4 / FP8 engines per profile; multimodal students for vision tasks | TensorRT Edge-LLM; llama.cpp with CUDA offload evaluated |
| Raspberry Pi 5-class CPU | lab target | INT4 GGUF, CPU only | llama.cpp |
| Android devices | planned | NPU-accelerated INT4 where the SoC allows; CPU fallback | LiteRT / LiteRT-LM, Qualcomm AI Engine Direct, ExecuTorch |
Public order-of-magnitude figures exist for sub-1B INT4 students on these classes of hardware. They are not reproduced here: ByteHub publishes only what it has measured on its own boards, with the method.
6 · Proof standard
What is measured before anything is published.
Device
Tokens per second and time to first token under sustained thermal load; resident memory including the KV cache at the rated context length.
Operation
Escalation rate after deployment and its trend per flywheel cycle; rollback exercised on the staged rollout path.
Related