ai-scraping-defense
ai-scraping-defense is a multi-layered, microservice-based system designed to protect creative, FOSS, and documentation websites from unauthorized AI bots and malicious scrapers. Combining Nginx, Lua, and Python microservices, it delivers defense in depth through real-time detection, behavioral analysis, heuristic and machine learning models, optional LLM integration, Markov-based API sequence anomaly detection, AI-driven configuration recommendations, and an AI Labyrinth honeypot. Active countermeasures include a tarpit API, adaptive per-IP rate limiting, community blocklist syncing, and federated peer-to-peer threat sharing. The system supports crawler authentication and pay-per-crawl experiments with multi-provider payment gateway integration and optional blockchain-backed billing logs. It is fully containerized with Docker and ready for Kubernetes deployment, offering multi-tenant isolation, cloud and CDN integrations, RBAC, admin SSO with MFA, and comprehensive audit logging. Security features include CO