Knowledge Vault

Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models

2 min read

Overview

This survey explores the transition of AI models from cloud-based infrastructure to edge and terminal devices to meet real-time processing and privacy needs. It details the technical challenges of deploying large-scale models on resource-constrained hardware and outlines optimization strategies like model compression and hardware acceleration.

Abstract

This survey (ACM Computing Surveys, 2025) comprehensively explores on-device AI models — those designed to perform local data processing and inference on edge and terminal devices. As IoT proliferates and real-time processing demands increase, the shift from cloud-based to on-device inference addresses latency, bandwidth, and privacy concerns. Gartner projects 75% of enterprise data will be produced outside traditional data centers by 2025.

Methodology

The survey is structured around four research questions:

  1. What are the applications of on-device AI? (smartphones, smart home, autonomous vehicles, medical devices)
  2. What are the main deployment challenges? (computational constraints, energy management, data privacy)
  3. What are effective optimization methods? (model compression, hardware acceleration, data preprocessing)
  4. What are the future trends? (edge computing convergence, foundation model adaptation)

Key Contributions

  • On-device AI defined by three characteristics: real-time performance, resource constraints, and enhanced data privacy through local processing
  • Optimization strategies cataloged: quantization, pruning, knowledge distillation, neural architecture search, and hardware-specific acceleration
  • Foundation model adaptation identified as the key emerging challenge — adapting large models (GPT-3’s 175B parameters, 800GB storage) for constrained devices
  • Edge computing convergence — the boundary between cloud, edge, and device is blurring, requiring new architectural paradigms

Significance

The survey provides a structured overview bridging the gap between AI model development (traditionally cloud-focused) and the growing demand for edge deployment. Particularly relevant for IoT, Industry 4.0, and privacy-sensitive applications where cloud latency or data sovereignty constraints make on-device inference necessary.

Graph View

Backlinks