Step 8
Step 8 — Browser inference boundaries with ONNX Runtime Web
2 views
Table of contents
Small classification, embedding, and OCR-assistance models can run in the browser instead of on a server. This can reduce server calls and user-data transfer, but introduces model download, memory, battery, and device-performance costs.
Select an execution provider by capability
Do not assume WebGPU. Detect support and model operators at startup: prefer WebGPU when compatible, fall back to WebAssembly, then use a server API or disable the optional feature. Do not preload the model on the first screen; load it after an action that actually needs inference.
Inference choice
Feature request → WebGPU capability → WASM capability → server fallback
Model fetch → verify size and hash → create session → bounded inference
Treat download and memory as budgets
Measure model size, uncompressed memory, first-inference latency, and repeated latency on a low-end mobile device. Key Cache Storage by version and hash, and retire an old model only after validating the new one. Be able to release the session when the tab is hidden or memory is constrained.
Completion evidence
- The core interface remains usable in an unsupported browser.
- The model is not downloaded before use, and size plus integrity are verified.
- WebGPU, WASM, and server fallback results stay within an accepted tolerance.
- The feature remains disabled by default when transfer or inference exceeds its budget.
Related term: ONNX Runtime Web
Terms in this content
🎉 You finished Local LLM · pgvector · building a RAG chatbot
What's next? Pick another course below.