Awesome-On-Device-AI-Systems
Awesome On-Device AI Systems is a curated repository focused on efficient on-device AI inference for mobile and edge devices. It serves as a bridge between academic systems research and practical engineering deployment, covering optimization techniques for machine learning models such as large language models, vision-language models, and vision transformers running on resource-constrained hardware. The repository is organized into two main sections. The first covers inference engines and runtimes, including general purpose frameworks like LiteRT, ExecuTorch, ONNX Runtime, MNN, and NCNN, along with vendor specific SDKs for platforms such as Qualcomm Snapdragon NPUs, Apple Core ML, NVIDIA TensorRT, Intel OpenVINO, and MediaTek NeuroPilot. It also features LLM and generative AI specialized engines like llama.cpp, MLC LLM, TensorRT-LLM, mllm, and MLX. The second section compiles research papers grouped into topics including LLM inference on mobile SoCs, processor characterization and optimization, compiler-based