Running vision inference on-device — a phone, a browser, a small board — avoids the latency and privacy cost of a cloud round trip, but the model has to actually fit and run fast enough on far weaker hardware.
The two levers
- Quantization: converting model weights from 32-bit to 8-bit (or lower) shrinks size and speeds up inference, usually with a small, measurable accuracy cost.
- ONNX: an intermediate model format that runs across many devices and runtimes without depending on the original training framework, which is what actually makes 'runs on this specific phone' practical.
A realistic workflow
Train and validate normally, export to ONNX, quantize the ONNX model, then re-run your evaluation set on the quantized version before shipping — the accuracy drop from quantization is usually acceptable but should be measured, not assumed.
See ONNX Runtime Web: running ML in the browser for the specific case of shipping this to a browser rather than a native app.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.