Data & Machine Learning - The Boring Way

This tutorial walks you through setting up and building a Data Engineering & Machine Learning Platform. The tutorial is designed to explore many different technologies for the similar problems without any bias.

This is not a Production Ready Setup

Target Audience

Data Engineers, Machine Learning Engineer, Data Scientist, SRE, Infrastructure Engineer, Data Analysts, Data Analytics Engineer

Expected Technologies & Workflow

Data Engineering & Analytics

• Kubernetes Kind Installation link
• MinIO Integrate Object Storage on top of Kubernetes and use minio interface for simulating the s3 link
• Apache Airflow on top of Kubernetes & Running an end to end Airflow Workflow using Kubernetes Executor link
• Apache Spark Deploy Apache Spark on Kubernetes and run an example link
• Prefect Setup & Running an end to end Workflow
• Dagster Setup & Running an end to end Workflow
• Set up an ETL job running end-2-end on apache airflow. This job contains Spark & Python Operator
• Apache Hive Setting up Hive & Hive Metastore
• Deploy Trino & Open Source Presto and run dana Analytics queries.
• Integrate Superset & Metabase to run visualization. Integrate Presto with the visualization system.
• Open Table Format using Delta
• Open Table Format using Apache Iceberg
• Open Table Format using Apache Hudi
• Metadata Management using Amundsen
• Metadata Management using Datahub
• Setting up Apache Kafka distributed event streaming platform
• Using Spark Structered Streaming to run an end-2-end pipeline over any realtime data sources
• Using Apache Flink to run an end-2-end pipeline over any realtime data sources
• Redpanda, streaming data platform to run similar workflow
• Airbyte Data Integration platform
• Talend UI based Data Integration
• DBT DBT Sql Pipeline to compare with Spark and other tech
• Debezium Change Data Capture using Debezium to sync multiple databases

Monitoring & Observability

• Grafana Setting Up Grafana for Monitoring components. Start with Monitoring Pods
• FluentD logging metrics from pods & interact the same with Monitoring layer
• Setting up a full Monitoring and Alerting Platform & integrate minitoring across other technologies
• Setting up an Observability system

Machine Learning

• Setup Ray for Data Transformations
• Use Scikit-learn for an example ML training
• Setup Argo Pipeline for deploying ML Jobs
• Setup Flyte Orchestrator for pythonic Deployment
• Use Pytorch Lightening for runing ML training
• Use Tensorflow for running ML training
• Setup ML End-2-End Workflow on Flyte
• Deploy MLFlow for ML Model Tracking & Experimentation
• Deploy BentoML For deploying ML Model
• Deploy Sendon Core for ML Model Management
• Integrate MLflow with Seldon Core

Prerequisites

• 🐳 Docker Installed
• kubectl Installed, The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters
• Lens Installed, UI for Kubernetes.
  This is optional, kubectl is enough for getting all relevant stats from kubernetes cluster
• Helm The package manager for Kubernetes

Lab Basic Setup

• Setting Up Kind
• Deleting older Pods PodCleaner