← writing

Edge AI and the case for running models locally

2026.05.28Edge AI · local AI · inference · model deployment · edge computing · privacy

For most of the last decade, cloud data centres have been the default place to train and run machine learning models. Centralising computation made sense while models were large, data pipelines were centrally managed, and the primary constraint was raw compute capacity. That default is shifting. Edge AI moves the inference stage of machine learning workloads — and, increasingly, limited on-device training — onto devices and servers positioned close to where data is generated.

This is not an argument against the cloud. Edge computing extends it, distributing work along a continuum from constrained end devices, through local gateways and on-premises edge servers, to regional data centres and centralised cloud infrastructure. The practical question is no longer whether intelligence belongs at the edge, but which workloads belong there, and how to run them reliably on hardware that is constrained, heterogeneous, and often physically exposed.

This post distils a research review on edge AI deployment — the architectures, use cases, challenges, and solutions — into a practical case for running models locally.

What edge AI actually means

Edge AI applies machine learning at the network edge: on end-user devices, industrial gateways, or small local servers, rather than routing every byte of sensor data to a distant data centre. Production systems are typically organised in layers:

  • Data generation — sensors, cameras, and end devices that capture raw input.
  • Edge nodes — local processors that run inference on live data.
  • Infrastructure optimisation — software that places and coordinates workloads across available nodes.
  • Application layer — domain-specific services (manufacturing, healthcare, smart cities) that consume the results.

In most production deployments, the heavy stages of the AI lifecycle — data preparation and model training — still happen in the data centre. The trained model is then deployed to edge devices to perform real-time inference close to its source. Limited on-device fine-tuning is becoming more common when a device needs to adapt to local operating conditions, but full model training at the edge remains the exception rather than the rule.

The latency argument

Some applications simply cannot tolerate a cloud round trip. Autonomous vehicles and mobile robots must decide about braking, steering, and obstacle avoidance in milliseconds. Industrial safety systems on high-speed production lines need inspection results at line speed. Sending sensor data to a remote server and waiting for a response is operationally unacceptable.

Processing data locally can reduce response latency from hundreds of milliseconds to single-digit milliseconds — the difference between a useful real-time decision and a late one. These systems also tend to run several edge AI workloads simultaneously: perception, path planning, and safety monitoring, each with different latency and accuracy requirements. Local inference is what makes that combination feasible.

The privacy and bandwidth argument

Latency is not the only driver. Keeping raw data on the device rather than transmitting it to the cloud reduces exposure to network-based privacy and security risks and can simplify compliance with data-residency regulations.

Healthcare devices that monitor vital signs or analyse diagnostic images increasingly perform analysis on-device, allowing patient data to remain local while still enabling real-time alerts. In retail and public-safety deployments, surveillance systems that process video locally can transmit only relevant events to the cloud, substantially reducing bandwidth and storage costs.

Local processing also lowers the volume of data that must be transmitted and stored centrally. In remote or bandwidth-constrained environments — underground mines, maritime settings, agricultural operations — that reduction is not a nice-to-have. It is a hard operational requirement.

Where local inference shows up in production

Edge AI has moved from research prototypes to production deployment across sectors, each driven by a slightly different combination of latency, privacy, and resilience requirements.

Autonomous vehicles and robotics

Perception, path planning, and safety monitoring must all run without depending on continuous cloud access. Any reliance on a remote round trip is a single point of failure in a safety-critical system.

Healthcare and remote monitoring

On-device analysis keeps sensitive patient data local and supports real-time clinical alerts. Federated learning — where hospitals contribute model updates without centralising raw records — extends this privacy benefit into collaborative model improvement.

Industrial automation and manufacturing

Computer-vision models trained centrally are commonly deployed to edge devices for real-time quality control and defect detection on factory floors. Edge deployment also keeps safety-monitoring systems running in facilities where network connectivity is unreliable.

Smart cities and smart retail

Traffic management, public-safety monitoring, and infrastructure analytics often coordinate models across large numbers of distributed edge nodes such as traffic cameras. In retail, edge inference supports personalised in-store experiences and inventory monitoring without streaming every camera feed to the cloud.

What still needs the cloud

Running models locally does not mean abandoning centralised infrastructure. The cloud remains the right place for:

  • Model training — the computationally heavy stages of data preparation and training still favour data-centre hardware.
  • Fleet orchestration — coordinating updates, policy enforcement, and observability across thousands of geographically dispersed devices.
  • Aggregation — federated learning and other collaborative techniques still need a central point to combine model updates.

The productive mental model is a hybrid architecture: train centrally, infer locally, and use the cloud for coordination rather than for every real-time decision.

A practical starting point for local inference on a workstation or edge device looks like this:

ollama run llama3.2 "Summarise this document"

Tools like Ollama make it straightforward to run quantised models locally. The harder engineering work begins when you need consistent behaviour across a heterogeneous fleet of devices in production.

The hard parts of edge deployment

Moving intelligence to the edge trades abundant, homogeneous cloud resources for a landscape of constrained, heterogeneous, and often physically exposed devices. The literature converges on a similar set of recurring challenges.

Resource constraints and hardware heterogeneity

Edge devices operate with limited processing power, memory, and — in battery-powered scenarios — energy budgets. Models originally designed for cloud-scale hardware do not scale down gracefully. Compounding this, the edge landscape is highly heterogeneous: devices differ in operating systems, accelerators, and software environments, which makes consistent model behaviour difficult to guarantee across a deployment.

Connectivity and environmental conditions

Many edge deployments occur where network connectivity is intermittent or entirely absent. Edge AI systems must function reliably under these conditions rather than assuming continuous cloud access. Physical operating environments add further difficulty — industrial and outdoor hardware must withstand dust, vibration, humidity, and temperature extremes.

Privacy, security, and trust

Distributing intelligence across many physically accessible endpoints expands the attack surface relative to a centralised deployment. Edge devices are more exposed to physical tampering, and securing a fleet of distributed devices while maintaining consistent policy enforcement is harder than securing a single data centre. Decentralising data processing can improve privacy, but that benefit depends on the trustworthiness of the edge device itself.

Energy efficiency

For battery-powered and power-constrained devices, energy efficiency is a hard operational limit. Models must be optimised not only for accuracy but for the energy they consume per inference — a consideration that becomes more pressing as generative and other computationally intensive models are pushed toward edge deployment.

Fleet management and integration

Once deployed, edge AI systems must be monitored, updated, and maintained across thousands of geographically dispersed devices. Coordinating software updates, tracking model performance, and enforcing consistent configuration is substantially more complex than managing centralised infrastructure. Edge solutions also frequently need to interoperate with legacy industrial equipment, existing point-of-sale systems, and pre-existing security infrastructure.

Making it work in practice

No single technique resolves the edge deployment problem. Successful deployments combine model-level optimisation, architectural choices, and operational tooling.

Model optimisation

The most direct response to hardware constraints is to shrink the model itself. Compression, quantisation, and pruning reduce the memory and compute footprint of a trained model, allowing it to run within the limits of edge hardware while preserving as much predictive accuracy as possible. Frameworks that automate optimisation for a target device are increasingly used to make this process repeatable across large and varied device fleets.

Federated and privacy-preserving training

Federated learning allows multiple edge devices — or institutions such as hospitals — to contribute to a shared model without exchanging raw data. Each participant trains locally and shares only model updates, which are aggregated centrally. This addresses the tension between decentralisation benefits and the trust limitations of individual edge devices.

Hardware-agnostic, modular design

To manage hardware heterogeneity, practitioners increasingly favour modular deployment frameworks that are not tied to a specific chipset or vendor, together with APIs that allow AI components to interoperate with existing systems. Designing for offline operation — processing data directly on the device rather than assuming continuous connectivity — further reduces dependence on network availability.

Fleet management and observability

At the operational level, container-orchestration and fleet-management platforms have been adapted for constrained edge environments. Lightweight orchestration tools, combined with image-based atomic update mechanisms that transfer only changed portions of a system, allow organisations to roll out updates and roll back failures across large device fleets with less manual intervention. Integrating telemetry and tracing into edge deployments lets teams track inference latency, model performance, and system health across distributed locations from a single view.

The takeaway

Edge AI offers substantial benefits in latency, privacy, resilience, and bandwidth efficiency, and it is already supporting production use cases in autonomous transport, healthcare, industrial automation, and smart-city infrastructure. Realising those benefits, however, requires deliberately addressing the resource constraints, hardware heterogeneity, security exposure, and operational complexity that distinguish edge deployment from centralised cloud computing.

The most effective approaches combine model-level techniques such as compression and federated learning with modular, hardware-agnostic software design and platform-based fleet management. As edge systems are asked to support increasingly capable models — including generative workloads that were not the primary design target of most existing optimisation techniques — the central design tension remains the same: balancing model capability against the resource, energy, and connectivity constraints of the edge.

Successful edge AI deployment depends less on any single technology than on the deliberate alignment of model design, infrastructure, and operational practice with the constraints of the specific edge environment. Train in the cloud. Infer locally. Engineer for the fleet.

References