Home
Softono
r

rasbt

Professional software vendor delivering innovative solutions on the Softono platform. Specialized in both open-source and proprietary software development.

Visit Website
Total Products
8

Software by rasbt

LLMs-from-scratch
Open Source

LLMs-from-scratch

# Build a Large Language Model (From Scratch) This repository contains the code for developing, pretraining, and finetuning a GPT-like LLM and is the official code repository for the book [Build a Large Language Model (From Scratch)](https://amzn.to/4fqvn0D). <br> <br> <a href="https://amzn.to/4fqvn0D"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/cover.jpg?123" width="250px"></a> <br> In [*Build a Large Language Model (From Scratch)*](http://mng.bz/orYv), you'll learn and understand how large language models (LLMs) work from the inside out by coding them from the ground up, step by step. In this book, I'll guide you through creating your own LLM, explaining each stage with clear text, diagrams, and examples. The method described in this book for training and developing your own small-but-functional model for educational purposes mirrors the approach used in creating large-scale foundational models such as those behind ChatGPT. In addition, this book includes code for loading the weights of larger pretrained models for finetuning. - Link to the official [source code repository](https://github.com/rasbt/LLMs-from-scratch) - [Link to the book at Manning (the publisher's website)](http://mng.bz/orYv) - [Link to the book page on Amazon.com](https://www.amazon.com/gp/product/1633437167) - ISBN 9781633437166 <a href="http://mng.bz/orYv#reviews"><img src="https://sebastianraschka.com//images/LLMs-from-scratch-images/other/reviews.png" width="220px"></a> <br> <br> To download a copy of this repository, click on the [Download ZIP](https://github.com/rasbt/LLMs-from-scratch/archive/refs/heads/main.zip) button or execute the following command in your terminal: ```bash git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git ``` <br> (If you downloaded the code bundle from the Manning website, please consider visiting the official code repository on GitHub at [https://github.com/rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) for the latest updates.) <br> <br> # Table of Contents Please note that this `README.md` file is a Markdown (`.md`) file. If you have downloaded this code bundle from the Manning website and are viewing it on your local computer, I recommend using a Markdown editor or previewer for proper viewing. If you haven't installed a Markdown editor yet, [Ghostwriter](https://ghostwriter.kde.org) is a good free option. You can alternatively view this and other files on GitHub at [https://github.com/rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) in your browser, which renders Markdown automatically. <br> <br> > **Tip:** > If you're seeking guidance on installing Python and Python packages and setting up your code environment, I suggest reading the [README.md](setup/README.md) file located in the [setup](setup) directory. <br> <br> [![Code tests Linux](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-linux-uv.yml/badge.svg)](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-linux-uv.yml) [![Code tests Windows](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-windows-uv-pip.yml/badge.svg)](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-windows-uv-pip.yml) [![Code tests macOS](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-macos-uv.yml/badge.svg)](https://github.com/rasbt/LLMs-from-scratch/actions/workflows/basic-tests-macos-uv.yml) - [Troubleshooting Guide](./troubleshooting.md) | Chapter Title | Main Code (for Quick Access) | All Code + Supplementary | |------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|-------------------------------| | [Setup recommendations](setup) <br/>[How to best read this book](https://sebastianraschka.com/blog/2025/reading-books.html) | - | - | | Ch 1: Understanding Large Language Models | No code | - | | Ch 2: Working with Text Data | - [ch02.ipynb](ch02/01_main-chapter-code/ch02.ipynb)<br/>- [dataloader.ipynb](ch02/01_main-chapter-code/dataloader.ipynb) (summary)<br/>- [exercise-solutions.ipynb](ch02/01_main-chapter-code/exercise-solutions.ipynb) | [./ch02](./ch02) | | Ch 3: Coding Attention Mechanisms | - [ch03.ipynb](ch03/01_main-chapter-code/ch03.ipynb)<br/>- [multihead-attention.ipynb](ch03/01_main-chapter-code/multihead-attention.ipynb) (summary) <br/>- [exercise-solutions.ipynb](ch03/01_main-chapter-code/exercise-solutions.ipynb)| [./ch03](./ch03) | | Ch 4: Implementing a GPT Model from Scratch | - [ch04.ipynb](ch04/01_main-chapter-code/ch04.ipynb)<br/>- [gpt.py](ch04/01_main-chapter-code/gpt.py) (summary)<br/>- [exercise-solutions.ipynb](ch04/01_main-chapter-code/exercise-solutions.ipynb) | [./ch04](./ch04) | | Ch 5: Pretraining on Unlabeled Data | - [ch05.ipynb](ch05/01_main-chapter-code/ch05.ipynb)<br/>- [gpt_train.py](ch05/01_main-chapter-code/gpt_train.py) (summary) <br/>- [gpt_generate.py](ch05/01_main-chapter-code/gpt_generate.py) (summary) <br/>- [exercise-solutions.ipynb](ch05/01_main-chapter-code/exercise-solutions.ipynb) | [./ch05](./ch05) | | Ch 6: Finetuning for Text Classification | - [ch06.ipynb](ch06/01_main-chapter-code/ch06.ipynb) <br/>- [gpt_class_finetune.py](ch06/01_main-chapter-code/gpt_class_finetune.py) <br/>- [exercise-solutions.ipynb](ch06/01_main-chapter-code/exercise-solutions.ipynb) | [./ch06](./ch06) | | Ch 7: Finetuning to Follow Instructions | - [ch07.ipynb](ch07/01_main-chapter-code/ch07.ipynb)<br/>- [gpt_instruction_finetuning.py](ch07/01_main-chapter-code/gpt_instruction_finetuning.py) (summary)<br/>- [ollama_evaluate.py](ch07/01_main-chapter-code/ollama_evaluate.py) (summary)<br/>- [exercise-solutions.ipynb](ch07/01_main-chapter-code/exercise-solutions.ipynb) | [./ch07](./ch07) | | Appendix A: Introduction to PyTorch | - [code-part1.ipynb](appendix-A/01_main-chapter-code/code-part1.ipynb)<br/>- [code-part2.ipynb](appendix-A/01_main-chapter-code/code-part2.ipynb)<br/>- [DDP-script.py](appendix-A/01_main-chapter-code/DDP-script.py)<br/>- [exercise-solutions.ipynb](appendix-A/01_main-chapter-code/exercise-solutions.ipynb) | [./appendix-A](./appendix-A) | | Appendix B: References and Further Reading | No code | [./appendix-B](./appendix-B) | | Appendix C: Exercise Solutions | - [list of exercise solutions](appendix-C) | [./appendix-C](./appendix-C) | | Appendix D: Adding Bells and Whistles to the Training Loop | - [appendix-D.ipynb](appendix-D/01_main-chapter-code/appendix-D.ipynb) | [./appendix-D](./appendix-D) | | Appendix E: Parameter-efficient Finetuning with LoRA | - [appendix-E.ipynb](appendix-E/01_main-chapter-code/appendix-E.ipynb) | [./appendix-E](./appendix-E) | <br> &nbsp; The mental model below summarizes the contents covered in this book. <img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/mental-model.jpg" width="650px"> <br> &nbsp; ## Prerequisites The most important prerequisite is a strong foundation in Python programming. With this knowledge, you will be well prepared to explore the fascinating world of LLMs and understand the concepts and code examples presented in this book. If you have some experience with deep neural networks, you may find certain concepts more familiar, as LLMs are built upon these architectures. This book uses PyTorch to implement the code from scratch without using any external LLM libraries. While proficiency in PyTorch is not a prerequisite, familiarity with PyTorch basics is certainly useful. If you are new to PyTorch, Appendix A provides a concise introduction to PyTorch. Alternatively, you may find my book, [PyTorch in One Hour: From Tensors to Training Neural Networks on Multiple GPUs](https://sebastianraschka.com/teaching/pytorch-1h/), helpful for learning about the essentials. <br> &nbsp; ## Hardware Requirements The code in the main chapters of this book is designed to run on conventional laptops within a reasonable timeframe and does not require specialized hardware. This approach ensures that a wide audience can engage with the material. Additionally, the code automatically utilizes GPUs if they are available. (Please see the [setup](https://github.com/rasbt/LLMs-from-scratch/blob/main/setup/README.md) doc for additional recommendations.) &nbsp; ## Video Course [A 17-hour and 15-minute companion video course](https://www.manning.com/livevideo/master-and-build-large-language-models) where I code through each chapter of the book. The course is organized into chapters and sections that mirror the book's structure so that it can be used as a standalone alternative to the book or complementary code-along resource. <a href="https://www.manning.com/livevideo/master-and-build-large-language-models"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/video-screenshot.webp?123" width="350px"></a> &nbsp; ## Companion Book / Sequel [*Build A Reasoning Model (From Scratch)*](https://mng.bz/lZ5B), while a standalone book, can be considered as a sequel to *Build A Large Language Model (From Scratch)*. It starts with a pretrained model and implements different reasoning approaches, including inference-time scaling, reinforcement learning, and distillation, to improve the model's reasoning capabilities. Similar to *Build A Large Language Model (From Scratch)*, [*Build A Reasoning Model (From Scratch)*](https://mng.bz/lZ5B) takes a hands-on approach implementing these methods from scratch. <a href="https://mng.bz/lZ5B"><img src="https://sebastianraschka.com/images/reasoning-from-scratch-images/cover.webp?123" width="120px"></a> - Amazon link (TBD) - [Manning link](https://mng.bz/lZ5B) - [GitHub repository](https://github.com/rasbt/reasoning-from-scratch) <br> &nbsp; ## Exercises Each chapter of the book includes several exercises. The solutions are summarized in Appendix C, and the corresponding code notebooks are available in the main chapter folders of this repository (for example, [./ch02/01_main-chapter-code/exercise-solutions.ipynb](./ch02/01_main-chapter-code/exercise-solutions.ipynb). In addition to the code exercises, you can download a free 170-page PDF titled [Test Yourself On Build a Large Language Model (From Scratch)](https://www.manning.com/books/test-yourself-on-build-a-large-language-model-from-scratch) from the Manning website. It contains approximately 30 quiz questions and solutions per chapter to help you test your understanding. <a href="https://www.manning.com/books/test-yourself-on-build-a-large-language-model-from-scratch"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/test-yourself-cover.jpg?123" width="150px"></a> &nbsp; ## Bonus Material Several folders contain optional materials as a bonus for interested readers: - **Setup** - [Python Setup Tips](setup/01_optional-python-setup-preferences) - [Installing Python Packages and Libraries Used in This Book](setup/02_installing-python-libraries) - [Docker Environment Setup Guide](setup/03_optional-docker-environment) - **Chapter 2: Working With Text Data** - [Byte Pair Encoding (BPE) Tokenizer From Scratch](ch02/05_bpe-from-scratch/bpe-from-scratch-simple.ipynb) - [Comparing Various Byte Pair Encoding (BPE) Implementations](ch02/02_bonus_bytepair-encoder) - [Understanding the Difference Between Embedding Layers and Linear Layers](ch02/03_bonus_embedding-vs-matmul) - [Dataloader Intuition With Simple Numbers](ch02/04_bonus_dataloader-intuition) - **Chapter 3: Coding Attention Mechanisms** - [Comparing Efficient Multi-Head Attention Implementations](ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb) - [Understanding PyTorch Buffers](ch03/03_understanding-buffers/understanding-buffers.ipynb) - **Chapter 4: Implementing a GPT Model From Scratch** - [FLOPs Analysis](ch04/02_performance-analysis/flops-analysis.ipynb) - [KV Cache](ch04/03_kv-cache) - [Attention Alternatives](ch04/#attention-alternatives) - [Grouped-Query Attention](ch04/04_gqa) - [Multi-Head Latent Attention](ch04/05_mla) - [Sliding Window Attention](ch04/06_swa) - [Gated DeltaNet](ch04/08_deltanet) - [DeepSeek Sparse Attention (DSA)](ch04/09_dsa) - [Cross-Layer KV Sharing](ch04/10_kv-sharing) - [Mixture-of-Experts (MoE)](ch04/07_moe) - **Chapter 5: Pretraining on Unlabeled Data** - [Alternative Weight Loading Methods](ch05/02_alternative_weight_loading/) - [Pretraining GPT on the Project Gutenberg Dataset](ch05/03_bonus_pretraining_on_gutenberg) - [Adding Bells and Whistles to the Training Loop](ch05/04_learning_rate_schedulers) - [Optimizing Hyperparameters for Pretraining](ch05/05_bonus_hparam_tuning) - [Building a User Interface to Interact With the Pretrained LLM](ch05/06_user_interface) - [Converting GPT to Llama](ch05/07_gpt_to_llama) - [Memory-efficient Model Weight Loading](ch05/08_memory_efficient_weight_loading/memory-efficient-state-dict.ipynb) - [Extending the Tiktoken BPE Tokenizer with New Tokens](ch05/09_extending-tokenizers/extend-tiktoken.ipynb) - [PyTorch Performance Tips for Faster LLM Training](ch05/10_llm-training-speed) - [LLM Architectures](ch05/#llm-architectures-from-scratch) - [Llama 3.2 From Scratch](ch05/07_gpt_to_llama/standalone-llama32.ipynb) - [Qwen3 Dense and Mixture-of-Experts (MoE) From Scratch](ch05/11_qwen3/) - [Gemma 3 From Scratch](ch05/12_gemma3/) - [Olmo 3 From Scratch](ch05/13_olmo3/) - [Tiny Aya From Scratch](ch05/15_tiny-aya/) - [Qwen3.5 From Scratch](ch05/16_qwen3.5/) - [Gemma 4 E2B and E4B From Scratch](ch05/17_gemma4/) - [Chapter 5 with other LLMs as Drop-In Replacement (e.g., Llama 3, Qwen 3)](ch05/14_ch05_with_other_llms/) - **Chapter 6: Finetuning for classification** - [Additional Experiments Finetuning Different Layers and Using Larger Models](ch06/02_bonus_additional-experiments) - [Finetuning Different Models on the 50k IMDb Movie Review Dataset](ch06/03_bonus_imdb-classification) - [Building a User Interface to Interact With the GPT-based Spam Classifier](ch06/04_user_interface) - **Chapter 7: Finetuning to follow instructions** - [Dataset Utilities for Finding Near Duplicates and Creating Passive Voice Entries](ch07/02_dataset-utilities) - [Evaluating Instruction Responses Using the OpenAI API and Ollama](ch07/03_model-evaluation) - [Generating a Dataset for Instruction Finetuning](ch07/05_dataset-generation/llama3-ollama.ipynb) - [Improving a Dataset for Instruction Finetuning](ch07/05_dataset-generation/reflection-gpt4.ipynb) - [Generating a Preference Dataset With Llama 3.1 70B and Ollama](ch07/04_preference-tuning-with-dpo/create-preference-data-ollama.ipynb) - [Direct Preference Optimization (DPO) for LLM Alignment](ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb) - [Building a User Interface to Interact With the Instruction-Finetuned GPT Model](ch07/06_user_interface) More bonus material from the [Reasoning From Scratch](https://github.com/rasbt/reasoning-from-scratch) repository: - **Qwen3 (From Scratch) Basics** - [Qwen3 Source Code Walkthrough](https://github.com/rasbt/reasoning-from-scratch/blob/main/chC/01_main-chapter-code/chC_main.ipynb) - [Optimized Qwen3](https://github.com/rasbt/reasoning-from-scratch/tree/main/ch02/03_optimized-LLM) - **Evaluation** - [Verifier-Based Evaluation (MATH-500)](https://github.com/rasbt/reasoning-from-scratch/tree/main/ch03) - [Multiple-Choice Evaluation (MMLU)](https://github.com/rasbt/reasoning-from-scratch/blob/main/chF/02_mmlu) - [LLM Leaderboard Evaluation](https://github.com/rasbt/reasoning-from-scratch/blob/main/chF/03_leaderboards) - [LLM-as-a-Judge Evaluation](https://github.com/rasbt/reasoning-from-scratch/blob/main/chF/04_llm-judge) - **Inference Scaling** - [Self-Consistency](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch04/01_main-chapter-code/ch04_main.ipynb) - [Self-Refinement](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch05/01_main-chapter-code/ch05_main.ipynb) - **Reinforcement Learning** (RL) - [RLVR with GRPO From Scratch](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch06/01_main-chapter-code/ch06_main.ipynb) <br> &nbsp; ## Questions, Feedback, and Contributing to This Repository I welcome all sorts of feedback, best shared via the [Manning Forum](https://livebook.manning.com/forum?product=raschka&page=1) or [GitHub Discussions](https://github.com/rasbt/LLMs-from-scratch/discussions). Likewise, if you have any questions or just want to bounce ideas off others, please don't hesitate to post these in the forum as well. Please note that since this repository contains the code corresponding to a print book, I currently cannot accept contributions that would extend the contents of the main chapter code, as it would introduce deviations from the physical book. Keeping it consistent helps ensure a smooth experience for everyone. &nbsp; ## Citation If you find this book or code useful for your research, please consider citing it. Chicago-style citation: > Raschka, Sebastian. *Build A Large Language Model (From Scratch)*. Manning, 2024. ISBN: 978-1633437166. BibTeX entry: ``` @book{build-llms-from-scratch-book, author = {Sebastian Raschka}, title = {Build A Large Language Model (From Scratch)}, publisher = {Manning}, year = {2024}, isbn = {978-1633437166}, url = {https://www.manning.com/books/build-a-large-language-model-from-scratch}, github = {https://github.com/rasbt/LLMs-from-scratch} } ```

Education & Learning ML Frameworks
96.9K Github Stars
reasoning-from-scratch
Open Source

reasoning-from-scratch

# Build A Reasoning Model (From Scratch) This repository contains the code for developing an LLM reasoning model and is the official code repository for the book [*Build a Reasoning Model (From Scratch)*](https://mng.bz/lZ5B). <br> <br> <a href="https://mng.bz/lZ5B"><img src="https://sebastianraschka.com/images/reasoning-from-scratch-images/cover.webp?123" width="250px"></a> (Printed in color.) <br> In [*Build a Reasoning Model (From Scratch)*](https://mng.bz/lZ5B), you will learn and understand how a reasoning large language model (LLM) works. Reasoning is one of the most exciting and important recent advances in improving LLMs, but it’s also one of the easiest to misunderstand if you only hear the term reasoning and read about it in theory. This is why this book takes a hands-on approach. We will start with a pre-trained base LLM and then add reasoning capabilities ourselves, step by step in code, so you can see exactly how it works. The methods described in this book walk you through the process of developing your own small-but-functional reasoning model for educational purposes. It mirrors the approaches used in creating large-scale reasoning models such as DeepSeek R1, GPT-5 Thinking, and others. In addition, this book includes code for loading the weights of existing, pretrained models. - Link to the official [source code repository](https://github.com/rasbt/reasoning-from-scratch) - Link to the [book at Manning](https://mng.bz/lZ5B) (the publisher's website) - Link to the book page on Amazon.com (TBD) - ISBN 9781633434677 <br> <br> To download a copy of this repository, click on the [Download ZIP](https://github.com/rasbt/reasoning-from-scratch/archive/refs/heads/main.zip) button or execute the following command in your terminal: ```bash git clone --depth 1 https://github.com/rasbt/reasoning-from-scratch.git ``` <br> > **Tip:** > Chapter 2 provides additional tips on installing Python, managing Python packages, and setting up your coding environment. <br> <br> ## Table of Contents (In Progress) [![Code tests Linux](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-linux.yml/badge.svg)](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-linux.yml) [![Code tests macOS](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-macos.yml/badge.svg)](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-macos.yml) [![Code tests Windows](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-windows.yml/badge.svg)](https://github.com/rasbt/reasoning-from-scratch/actions/workflows/tests-windows.yml) - [Troubleshooting Guide](./troubleshooting.md) | Chapter Title | Main Code | | ----------------------------------------------------------- | ------------------------------------------------------------ | | Ch 1: Understanding reasoning Models | No code | | Ch 2: Generating Text with a Pre-trained LLM | - [ch02_main.ipynb](ch02/01_main-chapter-code/ch02_main.ipynb)<br/>- [ch02_exercise-solutions.ipynb](ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb) | | Ch 3: Evaluating Reasoning Models | - [ch03_main.ipynb](ch03/01_main-chapter-code/ch03_main.ipynb)<br/>- [ch03_exercise-solutions.ipynb](ch03/01_main-chapter-code/ch03_exercise-solutions.ipynb) | | Ch 4: Improving Reasoning with Inference-Time Scaling | - [ch04_main.ipynb](ch04/01_main-chapter-code/ch04_main.ipynb)<br/>- [ch04_exercise-solutions.ipynb](ch04/01_main-chapter-code/ch04_exercise-solutions.ipynb) | | Ch 5: Inference-Time Scaling via Self-Refinement | - [ch05_main.ipynb](ch05/01_main-chapter-code/ch05_main.ipynb)<br/>- [ch05_exercise-solutions.ipynb](ch05/01_main-chapter-code/ch05_exercise-solutions.ipynb) | | Ch 6: Training Reasoning Models with Reinforcement Learning | - [ch06_main.ipynb](ch06/01_main-chapter-code/ch06_main.ipynb)<br/>- [ch06_exercise-solutions.ipynb](ch06/01_main-chapter-code/ch06_exercise-solutions.ipynb) | | Ch 7: Improving GRPO for Reinforcement Learning | - [ch07_main.ipynb](ch07/01_main-chapter-code/ch07_main.ipynb)<br/>- [ch07_exercise-solutions.ipynb](ch07/01_main-chapter-code/ch07_exercise-solutions.ipynb) | | Ch 8: Distilling Reasoning Models for Efficient Reasoning | - [ch08_main.ipynb](ch08/01_main-chapter-code/ch08_main.ipynb)<br/>- [ch08_exercise-solutions.ipynb](ch08/01_main-chapter-code/ch08_exercise-solutions.ipynb) | | Appendix A: References and Further Reading | No code | | Appendix B: Exercise Solutions | Code and solutions are in each chapter's subfolder | | Appendix C: Qwen3 LLM Source Code | - [chC_main.ipynb](chC/01_main-chapter-code/chC_main.ipynb) | | Appendix D: Using larger LLMs | - [chD_main.ipynb](chD/chD_main.ipynb) | | Appendix E: Batching and throughput-oriented execution | - [chE_main.ipynb](chE/chE_main.ipynb) | | Appendix F: Common Approaches to LLM Evaluation | - [chF_main.ipynb](chF/01_main-chapter-code/chF_main.ipynb) | | Appendix G: Building a Chat Interface | - [chG](chG) | <br> &nbsp; The mental model below summarizes the main techniques covered in this book. <img src="https://sebastianraschka.com/images/reasoning-from-scratch-images/mental-model.webp" width="650px"> <br> &nbsp; ## Companion Book Please note that *Build A Reasoning Model (From Scratch)* is a standalone book focused on methods to improve LLM reasoning. In this book, we work with a pre-trained open-source base LLM (Qwen3) on top of which we code apply reasoning methods from scratch. This includes inference-time scaling, reinforcement learning, and distillation. However, if you are interested in understanding how a conventional base LLM is implemented, you may like my previous book, [*Build a Large Language Model (From Scratch)*](https://amzn.to/4fqvn0D). <a href="https://amzn.to/4fqvn0D"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/cover.jpg?123" width="120px"></a> - [Amazon link](https://amzn.to/4fqvn0D) - [Manning link](http://mng.bz/orYv) - [GitHub repository](https://github.com/rasbt/LLMs-from-scratch) <br> &nbsp; ## Hardware Requirements The code in the main chapters of this book is designed to mostly run on consumer hardware within a reasonable timeframe and does not require specialized server hardware. This approach ensures that a wide audience can engage with the material. Additionally, the code automatically utilizes GPUs if they are available. That being said, chapters 2-4 will work well on CPUs and GPUs. For chapters 5 and 6, it is recommended to use a GPU if you want to replicate the results in the chapter. (Please see the [setup_tips](ch02/02_setup-tips/python-instructions.md) doc for additional recommendations.) &nbsp; ## Exercises Each chapter of the book includes several exercises. The solutions are summarized in Appendix B, and the corresponding code notebooks are available in the main chapter folders of this repository (for example, [`ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb`](ch02/01_main-chapter-code/ch02_exercise-solutions.ipynb)). &nbsp; ## Bonus Material Several folders contain optional materials as a bonus for interested readers: - **Chapter 2: Generating Text with a Pre-trained LLM** - [Optional Python Setup and Cloud GPU Recommendations](ch02/02_setup-tips) - [Using a GPU-optimized version of the LLM](ch02/03_optimized-LLM) - [Using `torch.compile()` on Windows](ch02/04_torch-compile-windows) - [Run inference and chat with the model](ch02/05_use_model) - **Chapter 3: Evaluating LLMs** - [MATH-500 Verifier Scripts](ch03/02_math500-verifier-scripts) - [Advanced Parser](ch03/03_advanced-parser) (hybrid LaTeX parser) - **Chapter 4: Improving Reasoning with Inference-Time Scaling** - [Inference Scaling on MATH-500](ch04/02_math500-inference-scaling-scripts) (CoT prompting, self-consistency) - **Chapter 5: Inference-Time Scaling Via Self-Refinement** - [More Inference Scaling on MATH-500](ch05/02_math500-more-inference-scaling-scripts) (Best-of-N, self-refinement) - **Chapter 6: Training Reasoning Models with Reinforcement Learning** - [GRPO scripts](ch06/02_rlvr_grpo_scripts_intro) with a batched mode - **Chapter 7: Improving GRPO for Reinforcement Learning** - [Advanced GRPO scripts](ch07/03_rlvr_grpo_scripts_advanced) (including DeepSeek-V3.2-, Olmo3-, and GDPO-style training) - [Download training checkpoints](ch07/04_download_trainining_checkpoints) (how to download and use the chapter 6 and 7 GRPO checkpoints) - **Chapter 8: Distilling Reasoning Models for Efficient Reasoning** - [Generate distillation data](ch08/02_generate_distillation_data) (teacher-output generation via Ollama or OpenRouter) - [Train with distillation](ch08/04_train_with_distillation) (including single-example and batched distillation scripts) - [Download training checkpoints](ch08/05_download_training_checkpoints) (how to download and use the chapter 8 distillation checkpoints) - [Use Qwen3 with Hugging Face](ch08/06_use_via_huggingface) (how to use the base model and chapter 6-8 checkpoints with `transformers`) - **Appendix F: Common Approaches to LLM Evaluation** - [MMLU Evaluation Methods](chF/02_mmlu) - [LLM leaderboards](chF/03_leaderboards) - [LLM-as-a-judge](chF/04_llm-judge) - **Appendix G: Building a Chat Interface** - [Chat interface code](chG/01_main-chapter-code) &nbsp; ## Questions, Feedback, and Contributing to This Repository For common problems, please see the [Troubleshooting Guide](./troubleshooting.md). I welcome all sorts of feedback, best shared via the [Manning Discussion Forum](https://livebook.manning.com/forum?product=raschka2&page=1) or [GitHub Discussions](https://github.com/rasbt/reasoning-from-scratch/discussions). Likewise, if you have any questions or just want to bounce ideas off others, please don't hesitate to post these in the forum as well. Please note that since this repository contains the code corresponding to a print book, I currently cannot accept contributions that would extend the contents of the main chapter code, as it would introduce deviations from the physical book. Keeping it consistent helps ensure a smooth experience for everyone. &nbsp; ## Citation If you find this book or code useful for your research, please consider citing it. Chicago-style citation: > Raschka, Sebastian. *Build A Reasoning Model (From Scratch)*. Manning, 2025. ISBN: 9781633434677. BibTeX entry: ``` @book{build-llms-from-scratch-book, author = {Sebastian Raschka}, title = {Build A Reasoning Model (From Scratch)}, publisher = {Manning}, year = {2025}, isbn = {9781633434677}, url = {https://mng.bz/lZ5B}, github = {https://github.com/rasbt/reasoning-from-scratch} } ```

Education & Learning ML Frameworks
4.5K Github Stars
python-machine-learning-book
Open Source

python-machine-learning-book

# Python Machine Learning book code repository [![Google Group](https://img.shields.io/badge/-Google%20Group-lightgrey.svg)](https://groups.google.com/forum/#!forum/python-machine-learning-reader-discussion-board) --- #### IMPORTANT NOTE (09/21/2017): This GitHub repository contains the code examples of the **1st Edition** of Python Machine Learning book. If you are looking for the code examples of the **2nd Edition**, please refer to [this](https://github.com/rasbt/python-machine-learning-book-2nd-edition#whats-new-in-the-second-edition-from-the-first-edition) repository instead. --- What you can expect are 400 pages rich in useful material just about everything you need to know to get started with machine learning ... from theory to the actual code that you can directly put into action! This is not yet just another "this is how scikit-learn works" book. I aim to explain all the underlying concepts, tell you everything you need to know in terms of best practices and caveats, and we will put those concepts into action mainly using NumPy, scikit-learn, and Theano. You are not sure if this book is for you? Please checkout the excerpts from the [Foreword](./docs/foreword_ro.pdf) and [Preface](./docs/preface_sr.pdf), or take a look at the [FAQ](#faq) section for further information. --- [![](./images/pymle_cover_double_small.jpg)](https://www.amazon.com/Python-Machine-Learning-Sebastian-Raschka/dp/1783555130/ref=sr_1_1?ie=UTF8&qid=1470882464&sr=8-1&keywords=python+machine+learning) 1st edition, published September 23rd 2015<br> Paperback: 454 pages<br> Publisher: Packt Publishing<br> Language: English<br> ISBN-10: 1783555130<br> ISBN-13: 978-1783555130<br> Kindle ASIN: B00YSILNL0<br> <br> [![](./images/CRBadgeNotableBook.jpg)](http://www.computingreviews.com/recommend/bestof/notableitems.cfm?bestYear=2016) <br> German ISBN-13: 978-3958454224<br> Japanese ISBN-13: 978-4844380603<br> Italian ISBN-13: 978-8850333974<br> Chinese (traditional) ISBN-13: 978-9864341405<br> Chinese (mainland) ISBN-13: 978-7111558804<br> Korean ISBN-13: 979-1187497035<br> Russian ISBN-13: 978-5970604090<br> ## Table of Contents and Code Notebooks Simply click on the `ipynb`/`nbviewer` links next to the chapter headlines to view the code examples (currently, the internal document links are only supported by the NbViewer version). **Please note that these are just the code examples accompanying the book, which I uploaded for your convenience; be aware that these notebooks may not be useful without the formulae and descriptive text.** - Excerpts from the [Foreword](./docs/foreword_ro.pdf) and [Preface](./docs/preface_sr.pdf) - [Instructions for setting up Python and the Jupiter Notebook](./code/ch01/README.md) <br> 1. Machine Learning - Giving Computers the Ability to Learn from Data [[dir](./code/ch01)] [[ipynb](./code/ch01/ch01.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch01/ch01.ipynb)] 2. Training Machine Learning Algorithms for Classification [[dir](./code/ch02)] [[ipynb](./code/ch02/ch02.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch02/ch02.ipynb)] 3. A Tour of Machine Learning Classifiers Using Scikit-Learn [[dir](./code/ch03)] [[ipynb](./code/ch03/ch03.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch03/ch03.ipynb)] 4. Building Good Training Sets – Data Pre-Processing [[dir](./code/ch04)] [[ipynb](./code/ch04/ch04.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch04/ch04.ipynb)] 5. Compressing Data via Dimensionality Reduction [[dir](./code/ch05)] [[ipynb](./code/ch05/ch05.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch05/ch05.ipynb)] 6. Learning Best Practices for Model Evaluation and Hyperparameter Optimization [[dir](./code/ch06)] [[ipynb](./code/ch06/ch06.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch06/ch06.ipynb)] 7. Combining Different Models for Ensemble Learning [[dir](./code/ch07)] [[ipynb](./code/ch07/ch07.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch07/ch07.ipynb)] 8. Applying Machine Learning to Sentiment Analysis [[dir](./code/ch08)] [[ipynb](./code/ch08/ch08.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch08/ch08.ipynb)] 9. Embedding a Machine Learning Model into a Web Application [[dir](./code/ch09)] [[ipynb](./code/ch09/ch09.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch09/ch09.ipynb)] 10. Predicting Continuous Target Variables with Regression Analysis [[dir](./code/ch10)] [[ipynb](./code/ch10/ch10.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch10/ch10.ipynb)] 11. Working with Unlabeled Data – Clustering Analysis [[dir](./code/ch11)] [[ipynb](./code/ch11/ch11.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch11/ch11.ipynb)] 12. Training Artificial Neural Networks for Image Recognition [[dir](./code/ch12)] [[ipynb](./code/ch12/ch12.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb)] 13. Parallelizing Neural Network Training via Theano [[dir](./code/ch13)] [[ipynb](./code/ch13/ch13.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/ch13/ch13.ipynb)] <br> #### Equation Reference <a href="https://github.com/rasbt/python-machine-learning-book/tree/master/docs/equations"><img src="images/equation-ref-logo.png" width="200" height="200" /></a> [[PDF](./docs/equations/pymle-equations.pdf)] [[TEX](./docs/equations/pymle-equations.tex)] #### Slides for Teaching A big thanks to [Dmitriy Dligach](dmitriydligach) for sharing his slides from his machine learning course that is currently offered at [Loyola University Chicago](http://www.luc.edu/cs/). - [https://github.com/dmitriydligach/PyMLSlides](https://github.com/dmitriydligach/PyMLSlides) - #### Additional Math and NumPy Resources Some readers were asking about Math and NumPy primers, since they were not included due to length limitations. However, I recently put together such resources for another book, but I made these *chapters* freely available online in hope that they also serve as helpful background material for this book: - Algebra Basics [[PDF](https://sebastianraschka.com/pdf/books/dlb/appendix_b_algebra.pdf)] [[EPUB](https://sebastianraschka.com/pdf/books/dlb/appendix_b_algebra.epub)] - A Calculus and Differentiation Primer [[PDF](https://sebastianraschka.com/pdf/books/dlb/appendix_d_calculus.pdf)] [[EPUB](https://sebastianraschka.com/pdf/books/dlb/appendix_d_calculus.epub)] - Introduction to NumPy [[PDF](https://sebastianraschka.com/pdf/books/dlb/appendix_f_numpy-intro.pdf)] [[EPUB](https://sebastianraschka.com/pdf/books/dlb/appendix_f_numpy-intro.epub)] [[Code Notebook](https://github.com/rasbt/deep-learning-book/blob/master/code/appendix_f_numpy-intro/appendix_f_numpy-intro.ipynb)] --- #### Citing this Book You are very welcome to re-use the code snippets or other contents from this book in scientific publications and other works; in this case, I would appreciate citations to the original source: **BibTeX**: ``` @Book{raschka2015python, author = {Raschka, Sebastian}, title = {Python Machine Learning}, publisher = {Packt Publishing}, year = {2015}, address = {Birmingham, UK}, isbn = {1783555130} } ``` **MLA**: Raschka, Sebastian. *Python machine learning*. Birmingham, UK: Packt Publishing, 2015. Print. --- ### [Feedback & Reviews](./docs/feedback.md) #### [Short review snippets](./docs/feedback.md) [![](./images/pymle_amzn.png)](https://www.amazon.com/Python-Machine-Learning-Sebastian-Raschka/dp/1783555130/ref=sr_1_1?ie=UTF8&qid=1472342570&sr=8-1&keywords=sebastian+raschka) --- > *Sebastian Raschka’s new book, Python Machine Learning, has just been released. I got a chance to read a review copy and it’s just as I expected - really great! It’s well organized, super easy to follow, and it not only offers a good foundation for smart, non-experts, practitioners will get some ideas and learn new tricks here as well.* – Lon Riesberg at [Data Elixir](http://dataelixir.com/issues/55#start) > *Superb job! Thus far, for me it seems to have hit the right balance of theory and practice…math and code!* – [Brian Thomas](http://sebastianraschka.com/blog/2015/writing-pymle.html#comment-2295668894) > *I've read (virtually) every Machine Learning title based around Scikit-learn and this is hands-down the best one out there.* – [Jason Wolosonovich](https://www.linkedin.com/pulse/python-machine-learning-sebastian-raschka-review-jason-wolosonovich?trk=prof-post) > *The best book I've seen to come out of PACKT Publishing. This is a very well written introduction to machine learning with Python. As others have noted, a perfect mixture of theory and application.* – [Josh D.](https://www.amazon.com/gp/customer-reviews/R27WB1GWTNGIR2/ref=cm_cr_getr_d_rvw_ttl?ie=UTF8&ASIN=1783555130) > *A book with a blend of qualities that is hard to come by: combines the needed mathematics to control the theory with the applied coding in Python. Also great to see it doesn't waste paper in giving a primer on Python as many other books do just to appeal to the greater audience. You can tell it's been written by knowledgeable writers and not just DIY geeks.* – [Amazon Customer](https://www.amazon.com/gp/customer-reviews/RZWY4TF66Z6V0/ref=cm_cr_getr_d_rvw_ttl?ie=UTF8&ASIN=1783555130) > *Sebastian Raschka created an amazing machine learning tutorial which combines theory with practice. The book explains machine learning from a theoretical perspective and has tons of coded examples to show how you would actually use the machine learning technique. It can be read by a beginner or advanced programmer.* - William P. Ross, [7 Must Read Python Books](http://williampross.com/7-must-read-python-books/) #### Longer reviews If you need help to decide whether this book is for you, check out some of the "longer" reviews linked below. (If you wrote a review, please let me know, and I'd be happy to add it to the list). - [Python Machine Learning Review](http://www.bcs.org/content/conWebDoc/55586) by Patrick Hill at the Chartered Institute for IT - [Book Review: Python Machine Learning by Sebastian Raschka](http://whatpixel.com/python-machine-learning-book-review/) by Alex Turner at WhatPixel --- ## Links - ebook and paperback at [Amazon.com](http://www.amazon.com/Python-Machine-Learning-Sebastian-Raschka/dp/1783555130/ref=sr_1_2?ie=UTF8&qid=1437754343&sr=8-2&keywords=python+machine+learning+essentials), [Amazon.co.uk](http://www.amazon.co.uk/Python-Machine-Learning-Sebastian-Raschka/dp/1783555130), [Amazon.de](http://www.amazon.de/s/ref=nb_sb_noss_2?__mk_de_DE=ÅMÅŽÕÑ&url=search-alias%3Daps&field-keywords=python+machine+learning) - [ebook and paperback](https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning) from Packt (the publisher) - at other book stores: [Google Books](https://books.google.com/books?id=GOVOCwAAQBAJ&source=gbs_slider_cls_metadata_7_mylibrary), [O'Reilly](http://shop.oreilly.com/product/9781783555130.do), [Safari](https://www.safaribooksonline.com/library/view/python-machine-learning/9781783555130/), [Barnes & Noble](http://www.barnesandnoble.com/w/python-machine-learning-essentials-sebastian-raschka/1121999969?ean=9781783555130), [Apple iBooks](https://itunes.apple.com/us/book/python-machine-learning/id1028207310?mt=11), ... - social platforms: [Goodreads](https://www.goodreads.com/book/show/25545994-python-machine-learning) #### Translations - [Italian translation](https://www.amazon.it/learning-Costruire-algoritmi-generare-conoscenza/dp/8850333978/) via "Apogeo" - [German translation](https://www.amazon.de/Machine-Learning-Python-mitp-Professional/dp/3958454224/) via "mitp Verlag" - [Japanese translation](http://www.amazon.co.jp/gp/product/4844380605/) via "Impress Top Gear" - [Chinese translation (traditional Chinese)](https://taiwan.kinokuniya.com/bw/9789864341405) - [Chinese translation (simple Chinese)](https://book.douban.com/subject/27000110/) - [Korean translation](http://www.kyobobook.co.kr/product/detailViewKor.laf?mallGb=KOR&ejkGb=KOR&barcode=9791187497035) via "Kyobo" - [Polish translation](https://www.amazon.de/Python-Uczenie-maszynowe-Sebastian-Raschka/dp/8328336138/ref=sr_1_11?ie=UTF8&qid=1513601461&sr=8-11&keywords=sebastian+raschka) via "Helion" --- ### [Literature References & Further Reading Resources](./docs/references.md) ### [Errata](./docs/errata.md) --- ### Bonus Notebooks (not in the book) - Logistic Regression Implementation [[dir](./code/bonus)] [[ipynb](./code/bonus/logistic_regression.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/logistic_regression.ipynb)] - A Basic Pipeline and Grid Search Setup [[dir](./code/bonus)] [[ipynb](./code/bonus/svm_iris_pipeline_and_gridsearch.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/svm_iris_pipeline_and_gridsearch.ipynb)] - An Extended Nested Cross-Validation Example [[dir](./code/bonus)] [[ipynb](./code/bonus/nested_cross_validation.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/nested_cross_validation.ipynb)] - A Simple Barebones Flask Webapp Template [[view directory](./code/bonus/flask_webapp_ex01)][[download as zip-file](https://github.com/rasbt/python-machine-learning-book/raw/master/code/bonus/flask_webapp_ex01/flask_webapp_ex01.zip)] - Reading handwritten digits from MNIST into NumPy arrays [[GitHub ipynb](./code/bonus/reading_mnist.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/reading_mnist.ipynb)] - Scikit-learn Model Persistence using JSON [[GitHub ipynb](./code/bonus/scikit-model-to-json.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/scikit-model-to-json.ipynb)] - Multinomial logistic regression / softmax regression [[GitHub ipynb](./code/bonus/softmax-regression.ipynb)] [[nbviewer](http://nbviewer.ipython.org/github/rasbt/python-machine-learning-book/blob/master/code/bonus/softmax-regression.ipynb)] <hr> **"Related Content" (not in the book)** - [Model evaluation, model selection, and algorithm selection in machine learning - Part I](http://sebastianraschka.com/blog/2016/model-evaluation-selection-part1.html) - [Model evaluation, model selection, and algorithm selection in machine learning - Part II](http://sebastianraschka.com/blog/2016/model-evaluation-selection-part2.html) - [Model evaluation, model selection, and algorithm selection in machine learning - Part III](http://sebastianraschka.com/blog/2016/model-evaluation-selection-part3.html) --- #### SciPy 2016 We had such a great time at [SciPy 2016](http://scipy2016.scipy.org/ehome/index.php?eventid=146062&tabid=332930&) in Austin! It was a real pleasure to meet and chat with so many readers of my book. Thanks so much for all the nice words and feedback! And in case you missed it, Andreas Mueller and I gave an **Introduction to Machine Learning with Scikit-learn**; if you are interested, the video recordings of [Part I](https://www.youtube.com/watch?v=OB1reY6IX-o&index=91&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6) and [Part II](https://www.youtube.com/watch?v=Cte8FYCpylk&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6&index=90) are now online! [![](images/scipy2016.jpg)](https://www.youtube.com/watch?v=OB1reY6IX-o&index=91&list=PLYx7XA2nY5Gf37zYZMw6OqGFRPjB1jCy6) #### PyData Chicago 2016 I attempted the rather challenging task of introducing scikit-learn & machine learning in *just* 90 minutes at PyData Chicago 2016. The slides and tutorial material are available at "[Learning scikit-learn -- An Introduction to Machine Learning in Python](https://github.com/rasbt/pydata-chicago2016-ml-tutorial)." --- **Note** I have set up a separate library, [`mlxtend`](http://rasbt.github.io/mlxtend/), containing additional implementations of machine learning (and general "data science") algorithms. I also added implementations from this book (for example, the decision region plot, the artificial neural network, and sequential feature selection algorithms) with additional functionality. [![](./images/mlxtend_logo.png)](http://rasbt.github.io/mlxtend/) <br> <hr> ### Translations [![](./images/pymle-cover_it.jpg)](https://www.amazon.it/learning-Costruire-algoritmi-generare-conoscenza/dp/8850333978/) [![](./images/pymle-cover_de.jpg)](https://www.amazon.de/Machine-Learning-Python-mitp-Professional/dp/3958454224/) [![](./images/pymle-cover_jp.jpg)](http://www.amazon.co.jp/gp/product/4844380605/) [![](./images/pymle-cover_cn.jpg)](https://taiwan.kinokuniya.com/bw/9789864341405) [![](./images/pymle-cover_cn_mainland.jpg)](https://book.douban.com/subject/27000110/) [![](./images/pymle-cover_kr.jpg)](http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9791187497035&orderClick=LEA&Kc=) [![](./images/pymle-cover_ru.jpg)](http://www.ozon.ru/context/detail/id/140152222/) [![](./images/pymle-cover_pl.jpg)](https://www.amazon.de/Python-Uczenie-maszynowe-Sebastian-Raschka/dp/8328336138/ref=sr_1_11?ie=UTF8&qid=1513601461&sr=8-11&keywords=sebastian+raschka) <hr> --- ***Dear readers***, first of all, I want to thank all of you for the great support! I am really happy about all the great feedback you sent me so far, and I am glad that the book has been so useful to a broad audience. Over the last couple of months, I received hundreds of emails, and I tried to answer as many as possible in the available time I have. To make them useful to other readers as well, I collected many of my answers in the FAQ section (below). In addition, some of you asked me about a platform for readers to discuss the contents of the book. I hope that this would provide an opportunity for you to discuss and share your knowledge with other readers: #### [Google Groups Discussion Board](https://groups.google.com/forum/#!forum/python-machine-learning-reader-discussion-board) (And I will try my best to answer questions myself if time allows! :)) > The only thing to do with good advice is to pass it on. It is never of any use to oneself. — Oscar Wilde --- ## Examples and Applications by Readers Once again, I have to say (big!) THANKS for all the nice feedback about the book. I've received many emails from readers, who put the concepts and examples from this book out into the real world and make good use of them in their projects. In this section, I am starting to gather some of these great applications, and I'd be more than happy to add your project to this list -- just shoot me a quick mail! - [40 scripts on Optical Character Recognition](https://github.com/rrlyman/PythonMachineLearingExamples) by [Richard Lyman](https://github.com/rrlyman) - [Code experiments](https://github.com/jeremyn/python-machine-learning-book) by [Jeremy Nation](https://github.com/jeremyn) - [What I Learned Implementing a Classifier from Scratch in Python](http://www.jeannicholashould.com) by [Jean-Nicholas Hould](http://www.jeannicholashould.com) ## FAQ ### General Questions - [What are machine learning and data science?](./faq/datascience-ml.md) - [Why do you and other people sometimes implement machine learning algorithms from scratch?](./faq/implementing-from-scratch.md) - [What learning path/discipline in data science I should focus on?](./faq/data-science-career.md) - [At what point should one start contributing to open source?](./faq/open-source.md) - [How important do you think having a mentor is to the learning process?](./faq/mentor.md) - [Where are the best online communities centered around data science/machine learning or python?](./faq/ml-python-communities.md) - [How would you explain machine learning to a software engineer?](./faq/ml-to-a-programmer.md) - [How would your curriculum for a machine learning beginner look like?](./faq/ml-curriculum.md) - [What is the Definition of Data Science?](./faq/definition_data-science.md) - [How do Data Scientists perform model selection? Is it different from Kaggle?](./faq/model-selection-in-datascience.md) ### Questions about the Machine Learning Field - [How are Artificial Intelligence and Machine Learning related?](./faq/ai-and-ml.md) - [What are some real-world examples of applications of machine learning in the field?](./faq/ml-examples.md) - [What are the different fields of study in data mining?](./faq/datamining-overview.md) - [What are differences in research nature between the two fields: machine learning & data mining?](./faq/datamining-vs-ml.md) - [How do I know if the problem is solvable through machine learning?](./faq/ml-solvable.md) - [What are the origins of machine learning?](./faq/ml-origins.md) - [How was classification, as a learning machine, developed?](./faq/classifier-history.md) - [Which machine learning algorithms can be considered as among the best?](./faq/best-ml-algo.md) - [What are the broad categories of classifiers?](./faq/classifier-categories.md) - [What is the difference between a classifier and a model?](./faq/difference_classifier_model.md) - [What is the difference between a parametric learning algorithm and a nonparametric learning algorithm?](./faq/parametric_vs_nonparametric.md) - [What is the difference between a cost function and a loss function in machine learning?](./faq/cost-vs-loss.md) ### Questions about ML Concepts and Statistics ##### Cost Functions and Optimization - [Fitting a model via closed-form equations vs. Gradient Descent vs Stochastic Gradient Descent vs Mini-Batch Learning -- what is the difference?](./faq/closed-form-vs-gd.md) - [How do you derive the Gradient Descent rule for Linear Regression and Adaline?](./faq/linear-gradient-derivative.md) ##### Regression Analysis - [What is the difference between Pearson R and Simple Linear Regression?](./faq/pearson-r-vs-linear-regr.md) ##### Tree models - [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](./faq/bagging-boosting-rf.md) - [What are the disadvantages of using classic decision tree algorithm for a large dataset?](./faq/decision-tree-disadvantages.md) - [Why are implementations of decision tree algorithms usually binary, and what are the advantages of the different impurity metrics?](./faq/decision-tree-binary.md) - [Why are we growing decision trees via entropy instead of the classification error?](./faq/decisiontree-error-vs-entropy.md) - [When can a random forest perform terribly?](./faq/random-forest-perform-terribly.md) ##### Model evaluation - [What is overfitting?](./faq/overfitting.md) - [How can I avoid overfitting?](./faq/avoid-overfitting.md) - [Is it always better to have the largest possible number of folds when performing cross validation?](./faq/number-of-kfolds.md) - [When training an SVM classifier, is it better to have a large or small number of support vectors?](./faq/num-support-vectors.md) - [How do I evaluate a model?](./faq/evaluate-a-model.md) - [What is the best validation metric for multi-class classification?](./faq/multiclass-metric.md) - [What factors should I consider when choosing a predictive model technique?](./faq/choosing-technique.md) - [What are the best toy datasets to help visualize and understand classifier behavior?](./faq/clf-behavior-data.md) - [How do I select SVM kernels?](./faq/select_svm_kernels.md) - [Interlude: Comparing and Computing Performance Metrics in Cross-Validation -- Imbalanced Class Problems and 3 Different Ways to Compute the F1 Score](./faq/computing-the-f1-score.md) ##### Logistic Regression - [What is Softmax regression and how is it related to Logistic regression?](./faq/softmax_regression.md) - [Why is logistic regression considered a linear model?](./faq/logistic_regression_linear.md) - [What is the probabilistic interpretation of regularized logistic regression?](./faq/probablistic-logistic-regression.md) - [Does regularization in logistic regression always results in better fit and better generalization?](./faq/regularized-logistic-regression-performance.md) - [What is the major difference between naive Bayes and logistic regression?](./faq/naive-bayes-vs-logistic-regression.md) - [What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning?](./faq/softmax.md) - [What is the relation between Logistic Regression and Neural Networks and when to use which?](./faq/logisticregr-neuralnet.md) - [Logistic Regression: Why sigmoid function?](./faq/logistic-why-sigmoid.md) - [Is there an analytical solution to Logistic Regression similar to the Normal Equation for Linear Regression?](./faq/logistic-analytical.md) ##### Neural Networks and Deep Learning - [What is the difference between deep learning and usual machine learning?](./faq/difference-deep-and-normal-learning.md) - [Can you give a visual explanation for the back propagation algorithm for neural networks?](./faq/visual-backpropagation.md) - [Why did it take so long for deep networks to be invented?](./faq/inventing-deeplearning.md) - [What are some good books/papers for learning deep learning?](./faq/deep-learning-resources.md) - [Why are there so many deep learning libraries?](./faq/many-deeplearning-libs.md) - [Why do some people hate neural networks/deep learning?](./faq/deeplearning-criticism.md) - [How can I know if Deep Learning works better for a specific problem than SVM or random forest?](./faq/deeplearn-vs-svm-randomforest.md) - [What is wrong when my neural network's error increases?](./faq/neuralnet-error.md) - [How do I debug an artificial neural network algorithm?](./faq/nnet-debugging-checklist.md) - [What is the difference between a Perceptron, Adaline, and neural network model?](./faq/diff-perceptron-adaline-neuralnet.md) - [What is the basic idea behind the dropout technique?](./faq/dropout.md) ##### Other Algorithms for Supervised Learning - [Why is Nearest Neighbor a Lazy Algorithm?](./faq/lazy-knn.md) ##### Unsupervised Learning - [What are some of the issues with clustering?](./faq/issues-with-clustering.md) ##### Semi-Supervised Learning - [What are the advantages of semi-supervised learning over supervised and unsupervised learning?](./faq/semi-vs-supervised.md) ##### Ensemble Methods - [Is Combining Classifiers with Stacking Better than Selecting the Best One?](./faq/logistic-boosting.md) ##### Preprocessing, Feature Selection and Extraction - [Why do we need to re-use training parameters to transform test data?](./faq/scale-training-test.md) - [What are the different dimensionality reduction methods in machine learning?](./faq/dimensionality-reduction.md) - [What is the difference between LDA and PCA for dimensionality reduction?](./faq/lda-vs-pca.md) - [When should I apply data normalization/standardization?](./faq/when-to-standardize.md) - [Does mean centering or feature scaling affect a Principal Component Analysis?](./faq/pca-scaling.md) - [How do you attack a machine learning problem with a large number of features?](./faq/large-num-features.md) - [What are some common approaches for dealing with missing data?](./faq/missing-data.md) - [What is the difference between filter, wrapper, and embedded methods for feature selection?](./faq/feature_sele_categories.md) - [Should data preparation/pre-processing step be considered one part of feature engineering? Why or why not?](./faq/dataprep-vs-dataengin.md) - [Is a bag of words feature representation for text classification considered as a sparse matrix?](./faq/bag-of-words-sparsity.md) ##### Naive Bayes - [Why is the Naive Bayes Classifier naive?](./faq/naive-naive-bayes.md) - [What is the decision boundary for Naive Bayes?](./faq/naive-bayes-boundary.md) - [Can I use Naive Bayes classifiers for mixed variable types?](./faq/naive-bayes-vartypes.md) - [Is it possible to mix different variable types in Naive Bayes, for example, binary and continues features?](./naive-bayes-vartypes.md) ##### Other - [What is Euclidean distance in terms of machine learning?](./faq/euclidean-distance.md) - [When should one use median, as opposed to the mean or average?](./faq/median-vs-mean.md) ##### Programming Languages and Libraries for Data Science and Machine Learning - [Is R used extensively today in data science?](./faq/r-in-datascience.md) - [What is the main difference between TensorFlow and scikit-learn?](./faq/tensorflow-vs-scikitlearn.md) <br> ### Questions about the Book - [Can I use paragraphs and images from the book in presentations or my blog?](./faq/copyright.md) - [How is this different from other machine learning books?](./faq/different.md) - [Which version of Python was used in the code examples?](./faq/py2py3.md) - [Which technologies and libraries are being used?](./faq/technologies.md) - [Which book version/format would you recommend?](./faq/version.md) - [Why did you choose Python for machine learning?](./faq/why-python.md) - [Why do you use so many leading and trailing underscores in the code examples?](./faq/underscore-convention.md) - [What is the purpose of the `return self` idioms in your code examples?](./faq/return_self_idiom.md) - [Are there any prerequisites and recommended pre-readings?](./faq/prerequisites.md) - [How can I apply SVM to categorical data?](./faq/svm_for_categorical.md) ## Contact I am happy to answer questions! Just write me an [email](mailto:[email protected]) or consider asking the question on the [Google Groups Email List](https://groups.google.com/forum/#!forum/python-machine-learning-book). If you are interested in keeping in touch, I have quite a lively twitter stream ([@rasbt](https://twitter.com/rasbt)) all about data science and machine learning. I also maintain a [blog](http://sebastianraschka.com/articles.html) where I post all of the things I am particularly excited about.

Education & Learning ML Frameworks
12.6K Github Stars
python-machine-learning-book-2nd-edition
Open Source

python-machine-learning-book-2nd-edition

## Python Machine Learning (2nd Ed.) Code Repository [![Build Status](https://travis-ci.com/rasbt/python-machine-learning-book-2nd-edition.svg?token=zvSsJVLJFKzB2yqaeKN1&branch=master)](https://travis-ci.com/rasbt/python-machine-learning-book-2nd-edition) ![Python 3.6](https://img.shields.io/badge/Python-3.6-blue.svg) ![License](https://img.shields.io/badge/Code%20License-MIT-blue.svg) *Please note that a new edition (3rd edition) is now available as of December 2019. The code repository link for the 3rd edition is https://github.com/rasbt/python-machine-learning-book-3rd-edition.* **Python Machine Learning, 2nd Ed.** published September 20th, 2017 Paperback: 622 pages Publisher: Packt Publishing Language: English ISBN-10: 1787125939 ISBN-13: 978-1787125933 Kindle ASIN: B0742K7HYF [<img src="./images/cover_1.jpg" width="348">](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939) ## Links - [Amazon Page](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939) - [Packt Page](https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning-second-edition) ## Table of Contents and Code Notebooks **Helpful installation and setup instructions can be found in the [README.md file of Chapter 1](code/ch01/README.md)** To access the code materials for a given chapter, simply click on the `open dir` links next to the chapter headlines to navigate to the chapter subdirectories located in the [code/](code/) subdirectory. You can also click on the `ipynb` links below to open and view the Jupyter notebook of each chapter directly on GitHub. In addition, the [code/](code/) subdirectories also contain .py script files, which were created from the Jupyter Notebooks. However, I highly recommend working with the Jupyter notebook if possible in your computing environment. Not only do the Jupyter notebooks contain the images and section headings for easier navigation, but they also allow for a stepwise execution of individual code snippets, which -- in my opinion -- provide a better learning experience. **Please note that these are just the code examples accompanying the book, which I uploaded for your convenience; be aware that these notebooks may not be useful without the formulae and descriptive text.** 1. Machine Learning - Giving Computers the Ability to Learn from Data [[open dir](./code/ch01)] [[ipynb](./code/ch01/ch01.ipynb)] 2. Training Machine Learning Algorithms for Classification [[open dir](./code/ch02)] [[ipynb](./code/ch02/ch02.ipynb)] 3. A Tour of Machine Learning Classifiers Using Scikit-Learn [[open dir](./code/ch03)] [[ipynb](./code/ch03/ch03.ipynb)] 4. Building Good Training Sets – Data Pre-Processing [[open dir](./code/ch04)] [[ipynb](./code/ch04/ch04.ipynb)] 5. Compressing Data via Dimensionality Reduction [[open dir](./code/ch05)] [[ipynb](./code/ch05/ch05.ipynb)] 6. Learning Best Practices for Model Evaluation and Hyperparameter Optimization [[open dir](./code/ch06)] [[ipynb](./code/ch06/ch06.ipynb)] 7. Combining Different Models for Ensemble Learning [[open dir](./code/ch07)] [[ipynb](./code/ch07/ch07.ipynb)] 8. Applying Machine Learning to Sentiment Analysis [[open dir](./code/ch08)] [[ipynb](./code/ch08/ch08.ipynb)] 9. Embedding a Machine Learning Model into a Web Application [[open dir](./code/ch09)] [[ipynb](./code/ch09/ch09.ipynb)] 10. Predicting Continuous Target Variables with Regression Analysis [[open dir](./code/ch10)] [[ipynb](./code/ch10/ch10.ipynb)] 11. Working with Unlabeled Data – Clustering Analysis [[open dir](./code/ch11)] [[ipynb](./code/ch11/ch11.ipynb)] 12. Implementing a Multi-layer Artificial Neural Network from Scratch [[open dir](./code/ch12)] [[ipynb](./code/ch12/ch12.ipynb)] 13. Parallelizing Neural Network Training with TensorFlow [[open dir](./code/ch13)] [[ipynb](./code/ch13/ch13.ipynb)] 14. Going Deeper: The Mechanics of TensorFlow [[open dir](./code/ch14)] [[ipynb](./code/ch14/ch14.ipynb)] 15. Classifying Images with Deep Convolutional Neural Networks [[open dir](./code/ch15)] [[ipynb](./code/ch15/ch15.ipynb)] 16. Modeling Sequential Data Using Recurrent Neural Networks [[open dir](./code/ch16)] [[ipynb](./code/ch16/ch16.ipynb)] ### What’s new in the second edition from the first edition? > Oh, there are so many things that we improved or added; where should I start!? The one issue on top of my priority list was to fix all the nasty typos that were introduced during the layout stage or my oversight. I really appreciated all the helpful feedback from readers in this manner! Furthermore, I addressed all the feedback about sections that may have been confusing or a bit unclear, reworded paragraphs, and added additional explanations. Also, special thanks go to the excellent editors of the second edition, who helped a lot along the way! > Also, the figures and plots became much prettier. While readers liked the graphic content a lot, some people criticized the PowerPoint-esque style and layout. Thus, I decided to overhaul every little figure with a hopefully more pleasing choice of fonts and colors. Also, the data plots look much nicer now, thanks to the matplotlib team who put a lot of work in matplotlib 2.0 and its new styling theme. > Beyond all these cosmetic fixes, new sections were added here and there. Among these is, for example, is a section on dealing with imbalanced datasets, which several readers were missing in the first edition and short section on Latent Dirichlet Allocation among others. > As time and the software world moved on after the first edition was released in September 2015, we decided to replace the introduction to deep learning via Theano. No worries, we didn't remove it but it got a substantial overhaul and is now based on TensorFlow, which has become a major player in my research toolbox since its open source release by Google in November 2015. Along with the new introduction to deep learning using TensorFlow, the biggest additions to this new edition are three brand new chapters focussing on deep learning applications: A more detailed overview of the TensorFlow mechanics, an introduction to convolutional neural networks for image classification, and an introduction to recurrent neural networks for natural language processing. Of course, and in a similar vein as the rest of the book, these new chapters do not only provide readers with practical instructions and examples but also introduce the fundamental mathematics behind those concepts, which are an essential building block for understanding how deep learning works. [ [Excerpt from "Machine Learning can be useful in almost every problem domain:" An interview with Sebastian Raschka](https://www.packtpub.com/books/content/machine-learning-useful-every-problem-domain-interview-sebastian-raschka/) ] --- <br> <br> Raschka, Sebastian, and Vahid Mirjalili. *Python Machine Learning, 2nd Ed*. Packt Publishing, 2017. @book{RaschkaMirjalili2017, address = {Birmingham, UK}, author = {Raschka, Sebastian and Mirjalili, Vahid}, edition = {2}, isbn = {978-1787125933}, keywords = {Clustering,Data Science,Deep Learning, Machine Learning,Neural Networks,Programming, Supervised Learning}, publisher = {Packt Publishing}, title = {{Python Machine Learning, 2nd Ed.}}, year = {2017} } # Translations ### German - ISBN-10: 3958457339 - ISBN-13: 978-3958457331 - [Amazon.de link](https://www.amazon.de/Machine-Learning-Python-Scikit-Learn-TensorFlow/dp/3958457339/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1513601461&sr=8-5) - [Publisher link](https://mitp.de/IT-WEB/Programmierung/Machine-Learning-mit-Python-oxid.html) ![](images/cover-german.jpg) ### Japanese - ISBN-10: 4295003379 - ISBN-13: 978-4295003373 - [Amazon.co.jp link](https://www.amazon.co.jp/Python-機械学習プログラミング-達人データサイエンティストによる理論と実践-impress-gear/dp/4295003379/ref=tmm_pap_swatch_0) ![](images/cover-japanese.jpg)

Education & Learning ML Frameworks
7.2K Github Stars
machine-learning-book
Open Source

machine-learning-book

# *Machine Learning with PyTorch and Scikit-Learn* Book ## Code Repository Paperback: 770 pages Publisher: Packt Publishing Language: English ISBN-10: 1801819319 ISBN-13: 978-1801819312 Kindle ASIN: B09NW48MR1 [<img src="./.other/cover_1.jpg" width="248">](https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-scikit-learn-ebook-dp-B09NW48MR1/dp/B09NW48MR1/) ## Links - [Amazon link](https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-scikit-learn-ebook-dp-B09NW48MR1/dp/B09NW48MR1/) - [Packt link](https://www.packtpub.com/product/machine-learning-with-pytorch-and-scikit-learn/9781801819312) - [Blog post summarizing the contents](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html) ## Table of Contents and Code Notebooks **Helpful installation and setup instructions can be found in the [README.md file of Chapter 1](ch01/README.md)**. **In addition, Zbynek Bazanowski contributed [this helpful guide](supplementary/running-on-colab.pdf) explaining how to run the code examples on Google Colab.** **Please note that these are just the code examples accompanying the book, which we uploaded for your convenience; be aware that these notebooks may not be useful without the formulae and descriptive text.** 1. Machine Learning - Giving Computers the Ability to Learn from Data [[open dir](ch01)] 2. Training Machine Learning Algorithms for Classification [[open dir](ch02)] 3. A Tour of Machine Learning Classifiers Using Scikit-Learn [[open dir](ch03)] 4. Building Good Training Sets – Data Pre-Processing [[open dir](ch04)] 5. Compressing Data via Dimensionality Reduction [[open dir](ch05)] 6. Learning Best Practices for Model Evaluation and Hyperparameter Optimization [[open dir](ch06)] 7. Combining Different Models for Ensemble Learning [[open dir](ch07)] 8. Applying Machine Learning to Sentiment Analysis [[open dir](ch08)] 9. Predicting Continuous Target Variables with Regression Analysis [[open dir](ch09)] 10. Working with Unlabeled Data – Clustering Analysis [[open dir](ch10)] 11. Implementing a Multi-layer Artificial Neural Network from Scratch [[open dir](ch11)] 12. Parallelizing Neural Network Training with PyTorch [[open dir](ch12)] 13. Going Deeper -- The Mechanics of PyTorch [[open dir](ch13)] 14. Classifying Images with Deep Convolutional Neural Networks [[open dir](ch14)] 15. Modeling Sequential Data Using Recurrent Neural Networks [[open dir](ch15)] 16. Transformers -- Improving Natural Language Processing with Attention Mechanisms [[open dir](ch16)] 17. Generative Adversarial Networks for Synthesizing New Data [[open dir](ch17)] 18. Graph Neural Networks for Capturing Dependencies in Graph Structured Data [[open dir](ch18)] 19. Reinforcement Learning for Decision Making in Complex Environments [[open dir](ch19)] --- <br> <br> Sebastian Raschka, Yuxi (Hayden) Liu, and Vahid Mirjalili. *Machine Learning with PyTorch and Scikit-Learn*. Packt Publishing, 2022. @book{mlbook2022, address = {Birmingham, UK}, author = {Sebastian Raschka, and Yuxi (Hayden) Liu, and Vahid Mirjalili}, isbn = {978-1801819312}, publisher = {Packt Publishing}, title = {{Machine Learning with PyTorch and Scikit-Learn}}, year = {2022} } ​ ## Coding Environment Please see the [ch01/README.md](ch01/README.md) file for setup recommendations. ## Translations into other Languages - Serbian Translation: [Mašinsko učenje uz PyTorch i Scikit-Learn](https://knjige.kombib.rs/masinsko-ucenje-uz-pytorch-i-scikit-learn). ISBN: 9788673105772

Education & Learning ML Frameworks
5.2K Github Stars
mlxtend
Open Source

mlxtend

[![DOI](https://joss.theoj.org/papers/10.21105/joss.00638/status.svg)](https://doi.org/10.21105/joss.00638) [![PyPI version](https://badge.fury.io/py/mlxtend.svg)](https://badge.fury.io/py/mlxtend) [![Build status](https://ci.appveyor.com/api/projects/status/7vx20e0h5dxcyla2/branch/master?svg=true)](https://ci.appveyor.com/project/rasbt/mlxtend/branch/master) [![codecov](https://codecov.io/gh/rasbt/mlxtend/branch/master/graph/badge.svg)](https://codecov.io/gh/rasbt/mlxtend) ![Python 3](https://img.shields.io/badge/python-3-blue.svg) ![License](https://img.shields.io/badge/license-BSD-blue.svg) [![Discuss](https://img.shields.io/badge/discuss-github-blue.svg)](https://github.com/rasbt/mlxtend/discussions) <img src="./docs/sources/img/logo.png" alt="mlxtend logo" width="300px"> **Mlxtend (machine learning extensions) is a Python library of useful tools for the day-to-day data science tasks.** It is primarily used for: - Ensemble methods such as stacking and voting classifiers - Feature selection and feature extraction techniques - Visualization utilities (e.g., decision regions, confusion matrices) - Plotting helpers for model analysis - Frequent pattern mining, including the Apriori algorithm for association rule mining <br> Sebastian Raschka 2014-2026 <br> ## Links - **Documentation:** [https://rasbt.github.io/mlxtend](https://rasbt.github.io/mlxtend) - PyPI: [https://pypi.python.org/pypi/mlxtend](https://pypi.python.org/pypi/mlxtend) - Changelog: [https://rasbt.github.io/mlxtend/CHANGELOG](https://rasbt.github.io/mlxtend/CHANGELOG) - Contributing: [https://rasbt.github.io/mlxtend/CONTRIBUTING](https://rasbt.github.io/mlxtend/CONTRIBUTING) - Questions? Check out the [GitHub Discussions board](https://github.com/rasbt/mlxtend/discussions) <br> <br> ## Installing mlxtend #### Using uv To add mlxtend to a uv-managed project, run ```bash uv add mlxtend ``` For a one-off command without changing your current project, run ```bash uv run --with mlxtend python -c "import mlxtend; print(mlxtend.__version__)" ``` #### Dev Version The mlxtend version on PyPI may always be one step behind; you can install the latest development version from the GitHub repository by executing ```bash uv add "mlxtend @ git+https://github.com/rasbt/mlxtend.git" ``` Or, you can fork the GitHub repository from https://github.com/rasbt/mlxtend and run mlxtend from your local checkout via ```bash git clone https://github.com/<your_username>/mlxtend.git cd mlxtend uv sync --group dev uv run python -c "import mlxtend; print(mlxtend.__version__)" ``` <br> <br> ## Examples ```python import numpy as np import matplotlib.pyplot as plt import matplotlib.gridspec as gridspec import itertools from sklearn.linear_model import LogisticRegression from sklearn.svm import SVC from sklearn.ensemble import RandomForestClassifier from mlxtend.classifier import EnsembleVoteClassifier from mlxtend.data import iris_data from mlxtend.plotting import plot_decision_regions # Initializing Classifiers clf1 = LogisticRegression(random_state=0) clf2 = RandomForestClassifier(random_state=0) clf3 = SVC(random_state=0, probability=True) eclf = EnsembleVoteClassifier(clfs=[clf1, clf2, clf3], weights=[2, 1, 1], voting='soft') # Loading some example data X, y = iris_data() X = X[:,[0, 2]] # Plotting Decision Regions gs = gridspec.GridSpec(2, 2) fig = plt.figure(figsize=(10, 8)) for clf, lab, grd in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'RBF kernel SVM', 'Ensemble'], itertools.product([0, 1], repeat=2)): clf.fit(X, y) ax = plt.subplot(gs[grd[0], grd[1]]) fig = plot_decision_regions(X=X, y=y, clf=clf, legend=2) plt.title(lab) plt.show() ``` ![](./docs/sources/img/ensemble_decision_regions_2d.png) --- If you use mlxtend as part of your workflow in a scientific publication, please consider citing the mlxtend repository with the following DOI: ``` @article{raschkas_2018_mlxtend, author = {Sebastian Raschka}, title = {MLxtend: Providing machine learning and data science utilities and extensions to Python’s scientific computing stack}, journal = {The Journal of Open Source Software}, volume = {3}, number = {24}, month = apr, year = 2018, publisher = {The Open Journal}, doi = {10.21105/joss.00638}, url = {https://joss.theoj.org/papers/10.21105/joss.00638} } ``` - Raschka, Sebastian (2018) MLxtend: Providing machine learning and data science utilities and extensions to Python's scientific computing stack. J Open Source Softw 3(24). --- ## License - This project is released under a permissive new BSD open source license ([LICENSE-BSD3.txt](https://github.com/rasbt/mlxtend/blob/master/LICENSE-BSD3.txt)) and commercially usable. There is no warranty; not even for merchantability or fitness for a particular purpose. - In addition, you may use, copy, modify and redistribute all artistic creative works (figures and images) included in this distribution under the directory according to the terms and conditions of the Creative Commons Attribution 4.0 International License. See the file [LICENSE-CC-BY.txt](https://github.com/rasbt/mlxtend/blob/master/LICENSE-CC-BY.txt) for details. (Computer-generated graphics such as the plots produced by matplotlib fall under the BSD license mentioned above). ## Contact The best way to ask questions is via the [GitHub Discussions channel](https://github.com/rasbt/mlxtend/discussions). In case you encounter usage bugs, please don't hesitate to use the [GitHub's issue tracker](https://github.com/rasbt/mlxtend/issues) directly.

Developer Tools ML Frameworks
5.2K Github Stars
python-machine-learning-book-3rd-edition
Open Source

python-machine-learning-book-3rd-edition

## Python Machine Learning (3rd Ed.) Code Repository [![Python 3.6](https://img.shields.io/badge/Python-3.7-blue.svg)](#) [![License](https://img.shields.io/badge/Code%20License-MIT-blue.svg)](LICENSE.txt) Code repositories for the 1st and 2nd edition are available at - https://github.com/rasbt/python-machine-learning-book and - https://github.com/rasbt/python-machine-learning-book-2nd-edition **Python Machine Learning, 3rd Ed.** to be published December 12th, 2019 Paperback: 770 pages Publisher: Packt Publishing Language: English ISBN-10: 1789955750 ISBN-13: 978-1789955750 Kindle ASIN: B07VBLX2W7 [<img src="./.other/cover_1.jpg" width="248">](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1789955750/) ## Links - [Amazon Page](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1789955750/) - [Packt Page](https://www.packtpub.com/data/python-machine-learning-third-edition) ## Table of Contents and Code Notebooks **Helpful installation and setup instructions can be found in the [README.md file of Chapter 1](ch01/README.md)** **Please note that these are just the code examples accompanying the book, which we uploaded for your convenience; be aware that these notebooks may not be useful without the formulae and descriptive text.** 1. Machine Learning - Giving Computers the Ability to Learn from Data [[open dir](ch01)] 2. Training Machine Learning Algorithms for Classification [[open dir](ch02)] 3. A Tour of Machine Learning Classifiers Using Scikit-Learn [[open dir](ch03)] 4. Building Good Training Sets – Data Pre-Processing [[open dir](ch04)] 5. Compressing Data via Dimensionality Reduction [[open dir](ch05)] 6. Learning Best Practices for Model Evaluation and Hyperparameter Optimization [[open dir](ch06)] 7. Combining Different Models for Ensemble Learning [[open dir](ch07)] 8. Applying Machine Learning to Sentiment Analysis [[open dir](ch08)] 9. Embedding a Machine Learning Model into a Web Application [[open dir](ch09)] 10. Predicting Continuous Target Variables with Regression Analysis [[open dir](ch10)] 11. Working with Unlabeled Data – Clustering Analysis [[open dir](ch11)] 12. Implementing a Multi-layer Artificial Neural Network from Scratch [[open dir](ch12)] 13. Parallelizing Neural Network Training with TensorFlow [[open dir](ch13)] 14. Going Deeper: The Mechanics of TensorFlow [[open dir](ch14)] 15. Classifying Images with Deep Convolutional Neural Networks [[open dir](ch15)] 16. Modeling Sequential Data Using Recurrent Neural Networks [[open dir](ch16)] 17. Generative Adversarial Networks for Synthesizing New Data [[open dir](ch17)] 18. Reinforcement Learning for Decision Making in Complex Environments [[open dir](ch18)] --- <br> <br> Raschka, Sebastian, and Vahid Mirjalili. *Python Machine Learning, 3rd Ed*. Packt Publishing, 2019. @book{RaschkaMirjalili2019, address = {Birmingham, UK}, author = {Raschka, Sebastian and Mirjalili, Vahid}, edition = {3}, isbn = {978-1789955750}, publisher = {Packt Publishing}, title = {{Python Machine Learning, 3rd Ed.}}, year = {2019} }

Education & Learning ML Frameworks
5K Github Stars
pattern_classification
Open Source

pattern_classification

![logo](./Images/logo.png) <hr> **Tutorials, examples, collections, and everything else that falls into the categories: pattern classification, machine learning, and data mining.** <br> <br> <br> <br> # Sections - [Introduction to Machine Learning and Pattern Classification](#introduction-to-machine-learning-and-pattern-classification) - [Pre-Processing](#pre-processing) - [Model Evaluation](#model-evaluation) - [Parameter Estimation](#parameter-estimation) - [Machine Learning Algorithms](#machine-learning-algorithms) - [Bayes Classification](#bayes-classification) - [Logistic Regression](#logistic-regression) - [Neural Networks](#neural-networks) - [Ensemble Methods](#ensemble-methods) - [Clustering](#clustering) - [Collecting Data](#collecting-data) - [Data Visualization](#data-visualization) - [Statistical Pattern Classification Examples](#statistical-pattern-classification-examples) - [Books](#books) - [Talks](#talks) - [Applications](#applications) - [Resources](#resources) <br> <br> <img src="./Images/supervised_learning_flowchart.png" style="width: 700px; height:600px;"> [[Download a PDF version](https://github.com/rasbt/pattern_classification/raw/master/PDFs/supervised_learning_flowchart.pdf)] of this flowchart. <br> <br> <br> <hr> <br> ### Introduction to Machine Learning and Pattern Classification [[back to top](#sections)] - Predictive modeling, supervised machine learning, and pattern classification - the big picture [[Markdown](./machine_learning/supervised_intro/introduction_to_supervised_machine_learning.md)] - Entry Point: Data - Using Python's sci-packages to prepare data for Machine Learning tasks and other data analyses [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/python_data_entry_point.ipynb)] - An Introduction to simple linear supervised classification using `scikit-learn` [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/scikit_linear_classification.ipynb)] <br> <br> <br> <hr> <br> ### Pre-processing [[back to top](#sections)] - **Feature Extraction** - Tips and Tricks for Encoding Categorical Features in Classification Tasks [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/preprocessing/feature_encoding.ipynb)] - **Scaling and Normalization** - About Feature Scaling: Standardization and Min-Max-Scaling (Normalization) [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/preprocessing/about_standardization_normalization.ipynb)] - **Feature Selection** - Sequential Feature Selection Algorithms [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/feature_selection/sequential_selection_algorithms.ipynb)] - **Dimensionality Reduction** - Principal Component Analysis (PCA) [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/principal_component_analysis.ipynb)] - The effect of scaling and mean centering of variables prior to a PCA [[PDF](https://github.com/rasbt/pattern_classification/raw/master/dimensionality_reduction/projection/scale_center_pca/scale_center_pca.pdf)] [[HTML](http://htmlpreview.github.io/?https://raw.githubusercontent.com/rasbt/pattern_classification/master/dimensionality_reduction/projection/scale_center_pca/scale_center_pca.html)] - PCA based on the covariance vs. correlation matrix [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/pca_cov_cor.ipynb)] - Linear Discriminant Analysis (LDA) [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/linear_discriminant_analysis.ipynb)] - Kernel tricks and nonlinear dimensionality reduction via PCA [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/dimensionality_reduction/projection/kernel_pca.ipynb)] - **Representing Text** - Tf-idf Walkthrough for scikit-learn [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/tfidf_scikit-learn.ipynb)] <br> <hr> <br> ### Model Evaluation [[back to top](#sections)] - An Overview of General Performance Metrics of Binary Classifier Systems [[PDF](http://arxiv.org/pdf/1410.5330.pdf)] - **Cross-validation** - Streamline your cross-validation workflow - scikit-learn's Pipeline in action [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/scikit-pipeline.ipynb)] - Model evaluation, model selection, and algorithm selection in machine learning - Part I [[Markdown](evaluation/model-evaluation/model-evaluation-selection-part1.md)] - Model evaluation, model selection, and algorithm selection in machine learning - Part II [[Markdown](evaluation/model-evaluation/model-evaluation-selection-part2.md)] <br> <hr> <br> ### Parameter Estimation [[back to top](#sections)] - **Parametric Techniques** - Introduction to the Maximum Likelihood Estimate (MLE) [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/parameter_estimation_techniques/maximum_likelihood_estimate.ipynb)] - How to calculate Maximum Likelihood Estimates (MLE) for different distributions [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/parameter_estimation_techniques/max_likelihood_est_distributions.ipynb)] - **Non-Parametric Techniques** - Kernel density estimation via the Parzen-window technique [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/parameter_estimation_techniques/parzen_window_technique.ipynb)] - The K-Nearest Neighbor (KNN) technique - **Regression Analysis** - Linear Regression - Least-Squares fit [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_fitting/regression/linregr_least_squares_fit.ipynb)] - Non-Linear Regression <br> <hr> <br> ### Machine Learning Algorithms [[back to top](#sections)] #### Bayes Classification - Naive Bayes and Text Classification I - Introduction and Theory [[PDF](http://arxiv.org/pdf/1410.5329.pdf)] #### Logistic Regression - Out-of-core Learning and Model Persistence using scikit-learn [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/outofcore_modelpersistence.ipynb)] #### Neural Networks - Artificial Neurons and Single-Layer Neural Networks - How Machine Learning Algorithms Work Part 1 [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/singlelayer_neural_networks/singlelayer_neural_networks.ipynb)] - Activation Function Cheatsheet [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/neural_networks/ipynb/activation_functions.ipynb)] #### Ensemble Methods - Implementing a Weighted Majority Rule Ensemble Classifier in scikit-learn [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/scikit-learn/ensemble_classifier.ipynb)] #### Decision Trees - Cheatsheet for Decision Tree Classification [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/machine_learning/decision_trees/decision-tree-cheatsheet.ipynb)] <br> <hr> <br> ### Clustering [[back to top](#sections)] - **Protoype-based clustering** - **Hierarchical clustering** - Complete-Linkage Clustering and Heatmaps in Python [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/clustering/hierarchical/clust_complete_linkage.ipynb)] - **Density-based clustering** - **Graph-based clustering** - **Probabilistic-based clustering** <br> <hr> <br> ## Collecting Data [[back to top](#sections)] - Collecting Fantasy Soccer Data with Python and Beautiful Soup [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_collecting/parse_dreamteamfc_data.ipynb)] - Download Your Twitter Timeline and Turn into a Word Cloud Using Python [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_collecting/twitter_wordcloud.ipynb)] - Reading MNIST into NumPy arrays [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_collecting/reading_mnist.ipynb)] <br> <hr> <br> ## Data Visualization [[back to top](#sections)] - Exploratory Analysis of the Star Wars API [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_viz/swapi_viz.ipynb)] ![](./Images/data_viz/data_viz_swapi_1.png) - Matplotlib examples -Exploratory data analysis of the Iris dataset [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_viz/matplotlib_viz_gallery.ipynb)] ![](./Images/data_viz/data_viz_iris_1.png) - Artificial Intelligence publications per country [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/data_viz/ai_publication_chloropleth/ai_publication_chloropleth.ipynb)] [[PDF](./data_viz/ai_publication_chloropleth/images/ai_publication_chloropleth_small.pdf)] ![](./data_viz/ai_publication_chloropleth/images/ai_publication_chloropleth_small.png) <br> <hr> <br> ### Statistical Pattern Classification Examples [[back to top](#sections)] - **Supervised Learning** - Parametric Techniques - Univariate Normal Density - Ex1: 2-classes, equal variances, equal priors [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/1_stat_superv_parametric.ipynb)] - Ex2: 2-classes, different variances, equal priors [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/2_stat_superv_parametric.ipynb)] - Ex3: 2-classes, equal variances, different priors [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/3_stat_superv_parametric.ipynb)] - Ex4: 2-classes, different variances, different priors, loss function [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/4_stat_superv_parametric.ipynb)] - Ex5: 2-classes, different variances, equal priors, loss function, cauchy distr. [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/5_stat_superv_parametric.ipynb)] - Multivariate Normal Density - Ex5: 2-classes, different variances, equal priors, loss function [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/5_stat_superv_parametric.ipynb)] - Ex7: 2-classes, equal variances, equal priors [[IPython nb](http://nbviewer.ipython.org/github/rasbt/pattern_classification/blob/master/stat_pattern_class/supervised/parametric/7_stat_superv_parametric.ipynb)] - Non-Parametric Techniques <br> <hr> <br> ## Books [[back to top](#sections)] #### Python Machine Learning <a href='http://sebastianraschka.com/publications.html'>![](./Images/books/pymle_cover.png)</a> - [Amazon link](http://www.amazon.com/Python-Machine-Learning-Sebastian-Raschka/dp/1783555130/ref=sr_1_2?ie=UTF8&qid=1437754343&sr=8-2&keywords=python+machine+learning+essentials) - [Publisher Link](https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning) - [GitHub Code Repository](https://github.com/rasbt/python-machine-learning-book) <br> <hr> <br> ## Talks [[back to top](#sections)] #### An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture <a href='http://www.slideshare.net/SebastianRaschka/nextgen-talk-022015'>![](./Images/talks/nextgentalk022015.png)</a> [[View on SlideShare](http://www.slideshare.net/SebastianRaschka/nextgen-talk-022015)] [[Download PDF](https://github.com/rasbt/pattern_classification/raw/master/PDFs/nextgentalk022015.pdf)] <br> <br> #### MusicMood - Machine Learning in Automatic Music Mood Prediction Based on Song Lyrics <a href='http://www.slideshare.net/SebastianRaschka/musicmood-20140912'> ![](./Images/talks/musicmood20140912.png)</a> [[View on SlideShare](http://www.slideshare.net/SebastianRaschka/musicmood-20140912)] [[Download PDF](https://github.com/rasbt/pattern_classification/raw/master/PDFs/musicmood20140912.pdf)] <br> <hr> <br> ## Applications [[back to top](#sections)] #### MusicMood - Machine Learning in Automatic Music Mood Prediction Based on Song Lyrics This project is about building a music recommendation system for users who want to listen to happy songs. Such a system can not only be used to brighten up one's mood on a rainy weekend; especially in hospitals, other medical clinics, or public locations such as restaurants, the MusicMood classifier could be used to spread positive mood among people. ![](./Images/applications/musicmood_1.png) [[musicmood GitHub Repository](https://github.com/rasbt/musicmood)] <br> #### mlxtend - A library of extension and helper modules for Python's data analysis and machine learning libraries. ![](./Images/applications/mlxtend_1.png) [[mlxtend GitHub Repository](https://github.com/rasbt/mlxtend)] <br> <hr> <br> ## Resources [[back to top](#sections)] - Copy-and-paste ready LaTex equations [[Markdown](./resources/latex_equations.md)] - Open-source datasets [[Markdown](./resources/dataset_collections.md)] - Free Machine Learning eBooks [[Markdown](./resources/machine_learning_ebooks.md)] - Terms in data science defined in less than 50 words [[Markdown](./resources/data_glossary.md)] - Useful libraries for data science in Python [[Markdown](./resources/python_data_libraries.md)] - General Tips and Advices [[Markdown](./resources/general_tips_and_advices.md)] - A matrix cheatsheat for Python, R, Julia, and MATLAB [[HTML](http://sebastianraschka.com/Articles/2014_matrix_cheatsheet_table.html)]

Education & Learning ML Frameworks
4.2K Github Stars