Overview
Consolidated from three papers covering the same domain: Patil et al. (ICRAES-2K25), Ubale (IJSRCSEIT 2025), and Ooko et al. (University of Rwanda). Collectively survey the principles, optimization techniques, and applications of running AI inference locally on edge devices without cloud connectivity.
Core Motivation
Cloud-based AI introduces: privacy exposure, latency, connectivity dependency, limited user data agency. On-device/offline AI addresses all four by running inference locally.
Model Compression Techniques
| Technique | Method | Effect |
|---|---|---|
| Pruning | Remove near-zero-magnitude weights | Leaner network, lower memory, faster inference |
| Quantization | FP32 → INT8 (or lower) | Up to 4x compression with <2% accuracy loss |
| Knowledge Distillation | Teacher→student model transfer | 40% size reduction maintaining 95% performance |
| Neural Architecture Search | Automated architecture optimization | Hardware-aware optimal model design |
Key Architectures
- MobileNetV1/V2/V3: Depthwise separable convolutions, inverted residuals, 8-9x computational cost reduction
- TinyML Models: Run on microcontrollers with 256KB RAM, no OS required
- Efficient Transformers: MobileViT, Linformer — transformer variants optimized for local NLP/vision
Hardware Ecosystem
- NVIDIA Jetson Nano: GPU acceleration for edge inference
- Google Coral (Edge TPU): 4 TOPS at <2W power consumption
- NPUs in mobile SoCs: Dedicated AI accelerators in modern smartphone chipsets
- Raspberry Pi 4: General-purpose edge compute
Market Context (Ubale)
Global mobile AI market: 41.6B (2028), 21.4% CAGR. GDPR/CCPA driving privacy-preserving local processing.
Edge AI Process (Ooko et al.)
- Collect/acquire training data
- Train with ML framework producing TinyML model
- Package as WebAssembly library
- Deploy to mobile device for local inference
- No connectivity required for prediction
Proof of concept: Offline Braille interpreter using Edge AI on mobile phone.
Applications
- Healthcare: Diabetic retinopathy diagnosis on smartphones (Medios)
- Autonomous vehicles: Real-time fault detection
- IoT: Smart agriculture, energy management
- Accessibility: Braille interpretation, assistive technology
- Security: On-device biometric processing
Related
- empowering-edge — Broader edge computing architecture (P1 integration)
- tiny-machine-learning — TinyML foundations (P1 integration)
- deepseek-ocr-context-compression — Related optical compression for on-device context management