Home
Softono
LightCL

LightCL

Open source Python
GitHub
15
Stars
0
Forks
0
Issues
1
Watchers
1 year
Last Commit
Edge Computing ML Frameworks

About LightCL

LightCL is a compact continual learning algorithm designed for edge devices with limited memory. It addresses the challenge of learning new tasks without forgetting previous ones by analyzing the generalizability of neural network layers, which varies significantly across different layers. LightCL maintains generalizability by freezing well-generalized parts of the network, avoiding costly retraining, and memorizes feature patterns by stabilizing feature extraction of prior tasks for less-generalized parts using a small memory buffer. This approach reduces memory footprint by up to 6.16x compared to state-of-the-art continual learning methods while maintaining or improving performance. The algorithm supports CIFAR10 and TinyImageNet datasets, works with PyTorch, and offers both standard and sparse training modes. Configurable parameters include learning rate, regulation loss weight, memory buffer size, vital feature map ratio, random seed, pretrained model usage, dataset selection, and sparsity. A pretrained

Platforms

Web Self-hosted

Languages

Python

Links

Source Code
i
Published by
inv-wzq
Visit View Profile

README.md

View on GitHub

LightCL

1 Introduction

LightCL is a compact algorithm for continual learning. Specifically, we consider two factors of generalizability, learning plasticity and memory stability, and design metrics of both to quantitatively assess generalizability of neural networks during CL. This evaluation shows that generalizability of different layers in a neural network exhibits a significant variation. Thus, as shown in the following figure, we Maintain Generalizability by freezing generalized parts without the resource-intensive training process and Memorize Feature Patterns by stabilizing feature extracting of previous tasks to enhance generalizability for less-generalized parts with a little extra memory, which is far less than the reduction by freezing. The following Experiment illustrate that LightCL outperforms other state-of-the-art methods and reduces at most 6.16× memory footprint. We also verify the effectiveness of LightCL on the edge device.

2 Implementation

Requirements

  • Python==3.11.4
  • numpy==1.24.3
  • torch==2.1.2
  • torchvision==0.16.2
  • tqdm==4.65.0, etc(which can be easily installed with pip)

Usage

Run LightCL.py with the commands like below:

  • Default
    python LightCL.py
  • With sparse
    python LightCL.py --Sparse

Here the major parameters are:

  • lr: learning rate (default=0.01)
  • Beta: hyperparameter for regulation loss (default=0.0002)
  • BufferNum: number of Memory Buffer (default=15)
  • Ratio: selecting vital feature map with Ratio (default=0.15)
  • Seed: the random seed (default=0)
  • pretrain: whether use the pre-trained model (default=True)
  • Dataset: dataset (default: CIFAR10; Other: TinyImageNet)
  • Sparse: whether sparse (default=False)

Note that we already have the pre-trained model (ResNet18_for_LightCL.pth) in the directory. If you want to pre-train the model yourself, you can download dataset ImageNet32x32 under data/ and run code get_parameter.py.

3 Cite

@inproceedings{wang2025lightCL,
author = {Wang, Zeqing and Cheng, Fei and Ji, Kangye and Huang, Bohu},
title = {LightCL: Compact Continual Learning with Low Memory Footprint For Edge Device},
booktitle = {Proceedings of the 30th Asia and South Pacific Design Automation Conference},
year = {2025},
}