LUMI AI guide

This guide is designed to assist users in migrating their machine learning applications from smaller-scale computing environments to the LUMI supercomputer. We will walk you through a detailed example of training an image classification model using PyTorch's Vision Transformer (VIT) on the ImageNet dataset.

All Python and bash scripts referenced in this guide are accessible in this GitHub repository. We start with a basic python script, visiontransformer.py, that could run on your local machine and modify it over the next chapters to run it efficiently on LUMI. This guide also covers the transition from training to production with a chapter on LLM Inference (Chapter 10), focusing on memory-efficient hosting with vLLM.

Even though this guide uses PyTorch, most of the covered topics are independent of the used machine learning framework. We therefore believe this guide is helpful for all new ML users on LUMI while also providing a concrete example that runs on LUMI.

Requirements

Before proceeding, please ensure you meet the following prerequisites:

• A basic understanding of machine learning concepts and Python programming. This guide will focus primarily on aspects specific to training models on LUMI.
• An active user account on LUMI and familiarity with its basic operations.
• If you wish to run the included examples, you need to be part of a project with GPU hours on LUMI.

Table of contents

The guide is structured into the following sections:

• 01. QuickStart
• 02. Setting up your own environment
• 03. File formats for training data
• 04. Data Storage Options
• 05. Multi-GPU and Multi-Node Training
• 06. Monitoring and Profiling jobs
• 07. TensorBoard visualization
• 08. MLflow visualization
• 09. Wandb visualization
• 10. LLM Inference

Further reading

• LUMI Documentation
• LUMI AI Factory Services
• LUMI AI software environment, PyTorch
• LUMI software library, TensorFlow
• LUMI software library, Jax
• Workshop material - Moving your AI training jobs to LUMI

Primary contributors

The primary contributors to this repository are the LUMI User Support Team (LUST) and the LUMI AI Factory.