mini-rag
This is a minimal implementation of the RAG model for question answering.
The Course
This is an educational project where all of the codes where explained (step by step) via a set of Arabic youtube videos. Please check the list:
| # | Title | Link | Codes |
|---|---|---|---|
| 1 | About the Course ماذا ولمـــاذا | Video | NA |
| 2 | What will we build ماذا سنبنى في المشروع | Video | NA |
| 3 | Setup your tools الأدوات الأساسية | Video | NA |
| 4 | Project Architecture | Video | branch |
| 5 | Welcome to FastAPI | Video | branch |
| 6 | Nested Routes + Env Values | Video | branch |
| 7 | Uploading a File | Video | branch |
| 8 | File Processing | Video | branch |
| 9 | Docker - MongoDB - Motor | Video | branch |
| 10 | Mongo Schemes and Models | Video | branch |
| 11 | Mongo Indexing | Video | branch |
| 12 | Data Pipeline Enhancements | Video | branch |
| 13 | Checkpoint-1 | Video | branch |
| 14 | LLM Factory | Video | branch |
| 15 | Vector DB Factory | Video | branch |
| 16 | Semantic Search | Video | branch |
| 17 | Augmented Answers | Video | branch |
| 18 | Checkpoint-1 + Fix Issues | Video | branch |
| 19 | Ollama Local LLM Server | Video | branch |
| 20 | From Mongo to Postgres + SQLAlchemy & Alembic | Video | branch |
| 21 | The way to PgVector | Video | branch |
| 22 | App Deployments 1/2 | Video | branch |
| 22 | App Deployments 2/2 | Video | branch |
| 24 | Celery Workers 1/2 | Video | branch |
| 25 | Celery Workers 2/2 | Video | branch |
Requirements
- Python 3.10
Install Dependencies
sudo apt update
sudo apt install libpq-dev gcc python3-dev
Install Python using MiniConda
- Download and install MiniConda from here
- Create a new environment using the following command:
$ conda create -n mini-rag python=3.10
- Activate the environment:
$ conda activate mini-rag
(Optional) Setup you command line interface for better readability
export PS1="\[\033[01;32m\]\u@\h:\w\n\[\033[00m\]\$ "
(Optional) Run Ollama Local LLM Server using Colab + Ngrok
Installation
Install the required packages
$ pip install -r requirements.txt
Setup the environment variables
$ cp .env.example .env
Run Alembic Migration
$ alembic upgrade head
Set your environment variables in the .env file. Like OPENAI_API_KEY value.
Run Docker Compose Services
$ cd docker
$ cp .env.example .env
- update
.envwith your credentials
$ cd docker
$ sudo docker compose up -d
Access Services
- FastAPI: http://localhost:8000
- Flower Dashboard: http://localhost:5555 (admin/password from env)
- Grafana: http://localhost:3000
- Prometheus: http://localhost:9090
Run the FastAPI server (Development Mode)
$ uvicorn main:app --reload --host 0.0.0.0 --port 5000
Celery (Development Mode)
For development, you can run Celery services manually instead of using Docker:
To Run the Celery worker, you need to run the following command in a separate terminal:
$ python -m celery -A celery_app worker --queues=default,file_processing,data_indexing --loglevel=info
To run the Beat scheduler, you can run the following command in a separate terminal:
$ python -m celery -A celery_app beat --loglevel=info
To Run Flower Dashboard, you can run the following command in a separate terminal:
$ python -m celery -A celery_app flower --conf=flowerconfig.py
open your browser and go to http://localhost:5555 to see the dashboard.
POSTMAN Collection
Download the POSTMAN collection from /assets/mini-rag-app.postman_collection.json