TL;DR: Run computer vision and other AI models on the device or a nearby gateway when you need low latency, must keep working without a network, cannot afford to stream raw video, or should not send images off site for privacy reasons. Run them in the cloud when models are large, change often, or need data from many sites at once. Most real systems are hybrid. Edge hardware ranges from microcontrollers that run tiny models in kilobytes of memory to GPU modules for multi-camera pipelines. Getting a model to fit relies on optimisation: quantisation (for example, Google's LiteRT documentation lists post-training integer quantisation at up to 75 percent smaller models with a small accuracy loss), pruning and choosing a smaller architecture. Always re-measure accuracy on your own data after optimising.
Edge vs cloud: how to decide
"Edge AI" simply means running inference close to where data is produced: on a camera, a phone, an industrial PC or a gateway on the factory floor, rather than on a remote server. Neither side is better in general. The decision is driven by a handful of questions.
| Factor | Favours edge | Favours cloud |
| Latency | Decisions in milliseconds, such as stopping a line or triggering a reject gate | Seconds or minutes are acceptable |
| Connectivity | Unreliable, expensive or absent | Reliable and cheap |
| Bandwidth | Many high-resolution video streams | Small payloads, occasional images |
| Privacy | Images of people or confidential processes should not leave the site | Data is already centralised and permitted |
| Model size and change rate | Model is stable and fits the hardware | Large models, frequent retraining, rapid experiments |
| Fleet management | You can update and monitor devices remotely | You want one place to deploy and monitor |
A common hybrid: run a compact detection model at the edge, send only events, crops or metadata to the cloud, and use the cloud for heavier analysis, dashboards, retraining and cross-site reporting. The device layer and messaging behind that are covered in our guide to industrial IoT architecture.
Hardware classes
Edge hardware spans several orders of magnitude in capability. Rather than chase specific products, think in classes.
Microcontrollers (TinyML)
Very small models for keyword spotting, simple anomaly detection on vibration or current, or very low-resolution vision. Google's LiteRT for Microcontrollers is designed to run machine learning models on microcontrollers and other devices with only a few kilobytes of memory; its documentation states that the core runtime fits in 16 KB on an Arm Cortex M3. Power consumption is tiny, so these can run on batteries.
Single-board computers and phones
Linux boards and mobile devices with CPUs and, increasingly, built-in neural processing units (NPUs). Suitable for single-camera classification and detection with optimised models, and for mobile apps that process images on the phone.
Accelerator modules and AI PCs
Dedicated GPU or NPU modules designed for embedded AI. NVIDIA's Jetson range runs from the Orin Nano, which NVIDIA positions as the entry-level option, up to the AGX Thor series, which it describes as its platform for physical AI and robotics. Intel's OpenVINO toolkit targets CPUs, GPUs and NPUs from a common workflow. This class handles multi-camera pipelines and larger models.
Edge servers
Rack or industrial servers with data-centre GPUs, placed on site. They serve many cameras, run larger models and act as a local cloud when data must stay on premises.
Choose hardware after you know the model, the frame rate and the number of streams you need, and test on the real device. Benchmarks from a different model, resolution or software version rarely transfer.
Making models fit: optimisation
Quantisation
Models are usually trained with 32-bit floating-point numbers. Quantisation stores weights, and often activations, at lower precision such as 8-bit integers. Google's LiteRT model optimisation guide summarises the main options:
- Post-training float16: no data needed, up to 50 percent smaller, insignificant accuracy loss.
- Post-training dynamic range: no data needed, up to 75 percent smaller, smallest accuracy loss.
- Post-training full integer: needs an unlabelled representative sample, up to 75 percent smaller, small accuracy loss, and compatible with integer-only accelerators.
- Quantisation-aware training: needs labelled training data, up to 75 percent smaller, smallest accuracy loss.
The same guide notes that quantisation can reduce latency by simplifying inference calculations, and that the accuracy change depends on the model and is hard to predict in advance. ONNX Runtime's documentation is equally direct: quantisation "is not a loss-less transformation" and may affect accuracy. It distinguishes dynamic quantisation, where activation ranges are computed at inference time (recommended there for transformer and recurrent models), from static quantisation, where ranges are computed ahead of time from calibration data (recommended for convolutional networks). If neither post-training approach meets your accuracy target, quantisation-aware training in the original framework is the next step.
Pruning
Pruning removes weights that contribute little, making the model sparse. PyTorch's tutorial distinguishes unstructured pruning, which removes individual connections, from structured pruning, which removes entire channels or units. The practical catch: a sparse model is not automatically faster. LiteRT's guide states that pruned models are the same size on disk and have the same runtime latency, but compress more effectively, so the main benefit there is a smaller download. Speed-ups need structured pruning or a runtime that exploits sparsity.
Smaller architectures and distillation
Often the biggest win is choosing a model family designed for mobile and edge, or training a small "student" model to imitate a larger one. Combine this with a sensible input resolution: many inspection tasks need a crop of the region of interest, not the full frame.
ONNX is an open format built to represent machine learning models, so a model trained in one framework can run on different runtimes and hardware. Exporting to a standard format keeps your options open when you change hardware vendors.
Privacy by design
Edge processing can be a strong privacy measure, because raw images need never leave the device. The EU GDPR requires personal data to be "adequate, relevant and limited to what is necessary" (data minimisation, Article 5(1)(c)) and requires data protection by design and by default (Article 25). Practical patterns:
- Send events and counts rather than images ("person entered zone B at 10:42").
- Blur or crop faces and number plates on the device when images must be kept.
- Keep retention short and documented, with access controls on any stored footage.
- Assess whether the use case is lawful and proportionate before deploying; workplace and public-space monitoring is regulated, and biometric identification especially so.
Edge processing reduces risk, but it does not by itself make a use case lawful. Get advice for anything that involves identifying people.
Use cases that work
Quality inspection
Cameras on a line detect defects such as scratches, missing components, wrong labels or fill levels, and trigger a reject or alert within the line's cycle time. This is a classic edge case: latency matters, the line must keep running if the network drops, and video volumes are high. Success depends less on the model than on consistent lighting, fixed camera positions and a good set of labelled defect examples, including rare ones.
Safety monitoring
Detecting people in exclusion zones near machinery or vehicles, or checking for required protective equipment at an entry point. Treat these as aids to existing safety measures, not replacements for certified safety systems, and involve your safety and HR functions in the design.
Retail and venue analytics
Counting footfall, measuring queue length, detecting empty shelves. Privacy-preserving designs that output only counts and events are both easier to justify and cheaper to run.
Predictive maintenance signals
Small models on vibration, acoustic or current data running on a gateway or even a microcontroller, flagging anomalies for a technician. These pair naturally with a digital twin of the asset.
Running edge AI in production
- Model updates. Ship models like firmware: versioned, signed, rolled out in stages, with rollback.
- Monitoring. Track inference latency, confidence distributions and error reports per device. A change in lighting or a new product variant can silently degrade accuracy.
- Feedback loop. Capture uncertain or disputed cases, label them and use them to retrain.
- Hardware lifecycle. Check the vendor's long-term availability and software support for the modules you choose.
How ApexlyTech can help
We build edge AI and computer vision systems from proof of concept to production: model selection and optimisation, on-device and gateway deployment, the dashboards and alerts people use, and the cloud side for retraining and reporting. See our edge AI and computer vision and AI and automation services, or contact us to discuss a use case.
Frequently asked questions
What is edge AI?
Running AI inference close to where data is produced, on a device, camera, phone or local gateway, instead of on a remote cloud server. It lowers latency, reduces bandwidth and can keep sensitive data on site.
Does quantisation reduce accuracy?
Usually a little, and sometimes not measurably. ONNX Runtime notes quantisation is not loss-less, and LiteRT lists small or insignificant accuracy loss depending on the method. Always re-test on your own validation data after quantising.
Does pruning make a model faster?
Not automatically. Unstructured pruning makes a model sparse, which helps compression, but LiteRT notes pruned models have the same runtime latency there. Structured pruning or a runtime that exploits sparsity is needed for speed gains.
Can I run computer vision on a microcontroller?
For very small models and low-resolution images, yes. LiteRT for Microcontrollers targets devices with only a few kilobytes of memory. Detection on high-resolution video needs a more capable class of hardware.
Is processing video at the edge automatically GDPR-compliant?
No. It supports data minimisation and privacy by design, but you still need a lawful basis, a proportionate purpose, transparency, and appropriate retention and security. Seek advice for any use case involving identifiable people.
Sources