Models — Specialized on-device models

How on-device distillation works

A frontier model teaches. A small model learns one skill. The device runs it.

Distillation is the transfer of a narrow capability from a large model to a small one. The large model never ships. What ships is a compact student that has seen thousands of worked examples of exactly the task the device performs, and a policy that decides when the student is not enough.

Training in progress. The teacher writes and scores worked examples for one skill; the student, an open base model under about one billion parameters, is fine-tuned on them while a judge scores each checkpoint against the golden set. Passing checkpoints are quantized and signed for the device; misses are corrected and become the next training set.

Define the skill

One narrow, high-volume task, written as a task card. A golden set of inputs with known-good outputs and a machine judge that scores answers against a rubric. Nothing is trained before this exists.

Teacher generates

A frontier LLM, called through an API, produces answers, rationales, and scores for many task inputs. Tokens are the cost. No GPU farm is involved.

Student learns

An open base model, under about one billion parameters, is fine-tuned on the teacher’s trajectories: supervised fine-tuning, knowledge distillation on the teacher’s output distribution, and rationale distillation where it helps. One small GPU, or a workstation, is enough at this size.

Quantize and package

The student is quantized for the target (INT4 weights, NPU-specific formats) and packaged as the same signed artifact EdgeRuntime already verifies: profile, compatibility rules, policy version, evaluation record.

Run with a check

On the device, every answer passes a confidence and judge check. Pass: the answer is used locally. Fail: the request is escalated through the policy gate to a frontier model, with minimized context and a logged crossing.

Flywheel

Misses are logged, corrected by the teacher or a reviewer, added to the training set, and the student is re-distilled, re-evaluated on the golden set, and re-signed. The device gets better at its one job without getting bigger.

Cloud · training time

Teacher

Frontier model behind an API; provider-aware (Vertex AI, Amazon Bedrock, Microsoft Foundry)
Writes and scores training trajectories for the one skill
Cost is tokens; runs for hours per iteration, then stops

⏻ distill
quantize
evaluate
sign

Device · run time

Student

Open base model under ~1B parameters, tuned to one task card
Runs on the NPU, GPU, or CPU already in the product, offline
Escalates to the teacher class only when the judge says so

Roles

Five parts, each with a job it must never exceed.

Part	What it is	Where it runs	What it costs	What it must never do
Teacher	Frontier LLM used to generate and grade training data	Cloud, via API, during training and for fallback	API tokens per iteration	Ship to the device; see raw device data outside policy
Student	Open base model under ~1B parameters, fine-tuned to one skill	On the device, inside EdgeRuntime	One small GPU or workstation to train; device compute to run	Answer outside its task card; pretend to be general
Judge	Rubric plus golden set that scores every candidate answer	Training pipeline and, in light form, on the device	Engineering time to write well	Be skipped; be tuned to the same seed as the student
Fallback	Policy-gated escalation to a frontier model when the judge fails	Through the policy gate; provider chosen per tenant	Tokens only on the misses	Open by default; send more context than approved
Flywheel	Miss log → correction → re-distill → re-evaluate → re-sign	ModelOps control plane	Periodic training runs	Deploy a new student without a golden-set pass and a signature

Why a narrow model is strong

Capacity spent on one distribution beats capacity spread across all of them.

A general model holds billions of parameters so it can answer anything. A device needs one thing answered well, fast, and offline. When training data covers only that distribution, a small model can concentrate all of its capacity there — and the public research record shows that small models trained this way can match or beat much larger models on the narrow task they were trained for.

What makes it work

Teacher rationales as supervision. Training on the teacher’s reasoning, not only its final answer, lets a small model learn the task with far less data.
Distribution matching, not label copying. Knowledge-distillation objectives fit the student to the teacher’s output distribution for the task, which transfers more than hard labels do.
Evaluation inside the loop. A golden set and a machine judge are part of training, so regressions are caught before a model is signed.
A short task card instead of a long prompt. The task is baked into the weights; the runtime prompt shrinks, which cuts latency and memory on the device.
Progressive transfer. Bridging the size gap in stages keeps the student stable when the teacher is orders of magnitude larger.

References

Peer-reviewed methods ByteHub builds on, plus publicly documented production patterns. None of these are ByteHub results; our own numbers appear only after they are measured.

Hsieh et al., Distilling Step-by-Step, ACL Findings 2023 — small models trained on teacher rationales outperform few-shot large models with less data. arXiv 2305.02301
Gu et al., MiniLLM: Knowledge Distillation of Large Language Models, 2023 — reverse-KL distillation for generative students. arXiv 2306.08543
Wang et al., Self-Instruct, 2022 — bootstrapping instruction data from a seed set with the model itself. arXiv 2212.10560
Shing et al. (Sakana AI), TAID: Temporally Adaptive Interpolated Distillation, ICLR 2025 — progressive transfer across large teacher–student gaps. arXiv 2501.16937
Mu et al., Learning to Compress Prompts with Gist Tokens, NeurIPS 2023 — long instructions compressed into a few learned tokens. arXiv 2304.08467
Shopify Engineering, Gisting: compressing LLM agent context and Sidekick’s continual learning loop — public production write-ups of prompt compression and the distill-deploy-correct-retrain loop. Cited as pattern inspiration only. gisting · continual learning

Why this fits a company our size

The expensive model is rented by the token. The cheap model is the one that ships.

The pipeline splits cost the right way. The frontier teacher is an API bill that runs for hours per iteration. A sub-1B student trains on a single small GPU or a workstation. Inference runs on the silicon the customer already pays for. There is no cluster to own and no cloud serving fleet to keep warm.

Assumption people bring	What is actually required
You need a large company’s proprietary specialist weights	No. Students start from public open base models and are trained on data generated for the customer’s task.
You need a hundred-GPU cluster	No. Numbers like that describe serving a specialist to tens of millions of cloud requests a day. On a device the metrics are tokens per second, resident memory, and judge score.
The teacher is a GPU farm you operate	No. The teacher is a frontier model behind an API; it is paid for in tokens and switched off between iterations.
Training has to happen in the cloud	Optional. At under ~1B parameters a local GPU is usually enough; cloud GPUs are used for bursts.
A small model means a worse product	For the one task it was trained on, a well-distilled small model is routinely competitive with its teacher, and it is faster, private, and works offline. For everything else, the policy gate escalates.

tokensteacher cost, per training iterationdesign

1 GPUstudent training at sub-1B scaledesign

on deviceinference on the customer’s own silicondesign

misses onlyfrontier tokens after deploymentdesign

Application types

Narrow jobs where latency and privacy decide the product.

Each of these is one task card, one golden set, one student. The device answers locally; the policy gate decides what may leave.

Vision

Security-camera and site-event understanding

Input: frames and detector output on the camera SoC or gateway. The student describes what happened — a person entered a restricted zone after hours, a vehicle stopped at a gate — rather than listing objects. Video never leaves the site; only the event summary crosses if policy allows. Escalates for ambiguous or novel scenes.

Inspection

On-prem inspection explanation

Input: inspection images and line context. The student classifies a defect and explains the likely cause in the operator’s terms. Stays on the line because product images are confidential and the line cannot wait for a round trip. Escalates when the defect class is outside the trained set.

Operations

Firmware and fleet-operations copilot

Input: device logs, configuration, and locked reference documents. The student answers bring-up and field questions for one product family — boot failures, pin and configuration checks, next diagnostic step. Runs on the technician’s terminal or gateway without connectivity. Escalates for multi-device reasoning.

Policy

Single-SKU device policy and fault triage

Input: device state and event stream. The student decides the next safe action for one device type — degrade, retry, hold, alert — and produces a structured triage record. Deterministic where possible; the student handles the unstructured part. Escalates when an action needs approval.

Interaction

Command understanding for appliances, kiosks, and HMIs

Input: short voice or text commands in the product’s domain. The student maps them to the device’s functions and asks one clarifying question when needed. Works offline, responds in the time a physical control would. Escalates open-ended requests that are outside the product.

Maintenance

Predictive-maintenance log summarisation

Input: sensor and event logs on an industrial gateway. The student turns a day of logs into a short, structured status with anomalies ranked and the evidence lines cited. Data stays in the plant; the summary is what travels. Escalates for cross-site correlation.

Hardware targets

Same student, packaged per execution profile.

The student is quantized and packaged for each target on the silicon ladder. The artifact format, policy gate, and rollback path are the ones described on the Technology page. Status reflects what is on our bench.

Target	Status	Student packaging	Runtime
i.MX95-class	in hand · validating	INT4 weights in the NPU vendor’s ONNX format; Cortex-A CPU fallback	NXP eIQ GenAI Flow, ONNX Runtime Neutron execution provider
Jetson Orin / Thor	in hand · validating	INT4 / FP8 engines per profile; multimodal students for vision tasks	TensorRT Edge-LLM, llama.cpp with CUDA offload evaluated
Raspberry Pi 5-class CPU	lab target	INT4 GGUF, CPU only; the low-cost lab and field-laptop path	llama.cpp
Android devices	planned	NPU-accelerated INT4 where the SoC allows, CPU fallback otherwise	LiteRT / LiteRT-LM, Qualcomm AI Engine Direct, ExecuTorch

What we build with you

A device, a named skill model, and the policy that governs it — shipped as one signed artifact.

You get

A task card and golden set for the skill, written with your domain experts
A student model trained for that skill from an open base model, quantized for your target
A fallback policy: what the student may answer, what escalates, what needs approval
The signed artifact, its evaluation record, and a rollback target on the device
A flywheel schedule: how misses become the next training set

Proof standard · measured before published

Judge score on the golden set: student vs untuned base vs teacher
Stability across training seeds; a student that only works for one seed is not shipped
Tokens per second and time to first token on the target board under sustained load
Resident memory including the KV cache at the rated context length
Escalation rate after deployment and how it moves with each flywheel cycle

FAQ

Questions we get about specialized models.

Do you need a large company’s proprietary model weights to do this?

No. Students start from publicly available open base models. The specialization comes from training data generated for your task by a frontier teacher, plus your golden set. Nothing proprietary from a third party is required.

Do you need a hundred GPUs?

No. Cluster-scale numbers you may have seen describe serving a specialist to tens of millions of cloud requests a day. Training a sub-1B student fits on one small GPU or a workstation, and inference runs on the device. The teacher is rented by the token.

Is training done locally or in the cloud?

Either. At this model size a local GPU is usually enough and keeps the training data on your side. Cloud GPUs are used for bursts or when the customer prefers it. The teacher is always a cloud API call, made with the minimized, approved data the policy allows.

Which base model do you use?

An open base model with a permissive licence, under about one billion parameters, chosen per target for its runtime support and quantization behaviour. We do not tie the product to one model family; the pipeline, the golden set, and the artifact format are what stay constant.

Does the student replace the frontier model?

For the one task it was trained on, yes: it answers locally, offline, and fast. For everything outside its task card, the judge fails and the policy gate escalates to a frontier model with a logged, minimized crossing. The two are designed to work together, not to compete.

How does the device get better over time?

Through the flywheel. Misses are logged, corrected by the teacher or a reviewer, and added to the training set. The student is re-distilled, re-evaluated on the golden set, re-signed, and rolled out in stages with the previous artifact kept for rollback. The model improves at its job without growing.

Is this the bring-up work ByteHub also does?

Bring-up is the other lane: getting a customer’s board, BSP, and update path to production. Specialized models are a ByteHub product built on top of that foundation. A firmware and fleet-operations copilot is one example skill, not the only one.

Start with one skill

Tell us the one job your device should do without the cloud.

Send the device, the task, and how you would judge a good answer. We reply with the skill definition, the golden-set plan, the target profile, and what we will measure before anything is published.

Contact us Read the research note

One device. One job. One small model trained for it.