Generative AI Project Template

Template for a new AI Cloud project.

Click on Use this template to start your own project!

This project is a generative ai template. It contains the following features: LLMs, information extraction, chat, rag & evaluation. It uses LLMs(local or cloud), NiceGUI (frontend) & FastAPI (backend) & Promptfoo as an evaluation and redteam framework for your AI system.

Test LLM

Engineering tools:

• Use UV to manage packages in a workspace (frontend and backend).
• pre-commit hooks: use ruff to ensure the code quality & detect-secrets to scan the secrets in the code.
• Logging using loguru (with colors)
• Pytest for unit tests
• Dockerized project (Dockerfile & docker-compose).
• NiceGUI (frontend) & FastAPI (backend)
• Make commands to handle everything for you: install, run, test

AI tools:

• LLM running locally with Ollama or in the cloud with any LLM provider (LiteLLM)
• Information extraction and Question answering from documents
• Chat to test the AI system
• Efficient async code using asyncio.
• AI Evaluation framework: using Promptfoo, Ragas & more...

CI/CD & Maintenance tools:

• CI/CD pipelines: .github/workflows for GitHub (Testing the AI system, local models with Ollama and the dockerized app)
• Local CI/CD pipelines: GitHub Actions using github act
• GitHub Actions for deploying to GitHub Pages with mkdocs gh-deploy
• Dependabot .github/dependabot.yml for automatic dependency and security updates

Documentation tools:

• Wiki creation and setup of documentation website using Mkdocs
• GitHub Pages deployment using mkdocs gh-deploy plugin

Upcoming features:

• add RAG again
• optimize caching in CI/CD
• Pull requests templates
• Additional MLOps templates: https://github.com/fmind/mlops-python-package
• Add MLFlow
• add Langfuse

1. Getting started

This project is a monorepo containing two main packages:

• frontend: A NiceGUI application.
• backend: A FastAPI application that serves the AI models and business logic.

The project uses uv as a package manager and is configured as a workspace, so dependencies for both packages can be installed with a single command.

Project Structure

• frontend/: The NiceGUI frontend application.
  • pyproject.toml: Frontend-specific dependencies.
  • src/: Source code for the frontend.
• backend/: The FastAPI backend application.
  • pyproject.toml: Backend-specific dependencies.
  • src/: Source code for the backend.
• .github/: GitHub-specific files.
  • workflows/: GitHub Actions CI/CD pipelines.
  • dependabot.yml: Dependabot configuration for dependency updates.
• assets/: Static assets like images.
• scripts/: Utility scripts.
• tests/: Pytest unit and integration tests.
• .env.example: Example environment variables. Create a .env from this.
• Dockerfile: To build the application container.
• docker-compose.yml: To run services like frontend, backend, and ollama.
• .gitlab-ci.yml: GitLab CI configuration file.
• Makefile: Main entry point that includes modular makefiles.
• makefiles/: Modular makefile includes organized by purpose.
  • common.mk: Shared variables (colors, UV path, Ollama models).
  • install.mk: Installation targets (uv, dependencies, Ollama).
  • dev.mk: Development targets (run frontend/backend, pre-commit).
  • test.mk: Testing targets (pytest, Ollama tests).
  • build.mk: Build and Docker targets.
  • ci.mk: CI/CD targets (GitHub Actions locally).
  • clean.mk: Cleanup targets (cache management).
• pyproject.toml: Defines uv workspace and shared dependencies.
• uv.lock: Lock file for uv package manager.
• .pre-commit-config.yaml: Configuration for pre-commit hooks.
• mkdocs.yml: Configuration for the documentation site.
• README.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md, LICENSE: Project documentation.

1.1. Local Prerequisites

• Ubuntu 22.04 or MacOS
• git clone the repository
• UV & Python 3.12 (will be installed by the Makefile)
• Create a .env file (take a look at the .env.example file)

1.2 ⚙️ Steps for Installation

This project uses a Makefile to simplify the installation and execution process.

Local Installation

1. For CPU-based environment (or MacOS) To install all dependencies for both frontend and backend for a CPU environment, run:

   make install-dev
   

2. For NVIDIA GPU (CUDA) environment If you have an NVIDIA GPU and want to use CUDA for acceleration, run:

   make install-dev-cuda
   

   This will install the CUDA-enabled version of PyTorch.

Using Docker

The project can be fully containerized using Docker. This is the recommended way to run the application as it handles all services and networks.

• The docker-compose.yml and docker-compose-cuda.yml files define the services.
• To run the entire application stack using Docker Compose with CPU support:

  make docker-compose
  

• To run with CUDA support:

  make docker-compose-cuda
  

• To rebuild and run (useful after code changes):

  # CPU
  make docker-compose-rebuild
  # CUDA
  make docker-compose-cuda-rebuild
  

Using Local vs. Cloud LLMs

• Local model (Ollama):
  • Install Ollama: make install-ollama
  • Ensure Ollama is running (make run-ollama can help).
  • Set your .env file to point to the local Ollama endpoint (copy and paste from the .env.example file).
  • Download a model: make download-ollama-models
  • Test the connection: make test-ollama
  • Test the connection: make test-inference-llm
• Cloud model (OpenAI, Anthropic, etc.):
  • Update your .env file with the correct API keys and model names, following the LiteLLM naming convention.
  • Test the connection: make test-inference-llm

Running the Application

Once installed (either locally or via Docker), you can run the services.

• Run Everything: The make run-app command is the easiest way to start all services, including the frontend, backend, database, and Ollama.

• Run Services Individually:

  • Run Backend: make run-backend
  • Run Frontend: make run-frontend

You can then access:

• Frontend (NiceGUI): http://localhost:8080 (or the configured port)
• Backend (FastAPI): http://localhost:8000 (or the configured port). Docs http://localhost:8000/docs

1.3 ⚙️ Steps for Installation (Contributors and maintainers)

Check the CONTRIBUTING.md file for more information.

2. Makefile Commands

The project uses a modular Makefile structure. Run make help to see all available commands.

Installation Commands

Development Commands

Testing Commands

Build & Docker Commands

CI/CD Commands

Cleanup Commands

3. Contributing

Check the CONTRIBUTING.md file for more information.