silkworm
Silkworm is an async-first web scraping framework built on Rust, designed for high performance and compatibility with Free-threaded Python. It combines the speed of Rust-based libraries for HTTP requests and HTML parsing with a flexible Python API. The framework features a minimal Spider, Request, and Response model that eliminates boilerplate for both quick scripts and large-scale crawlers. Key capabilities include a configurable async engine with priority-aware queueing and backpressure, and a powerful wreq-powered HTTP client supporting browser impersonation, proxy rotation, and redirect loop detection. It offers optional integrations for Tor Onion sites and JavaScript rendering via Servo. Silkworm includes robust middleware for User-Agent rotation, cookie management, retry logic with exponential backoff, robots.txt enforcement, and Cloudflare challenges. Data processing is handled through customizable pipelines that export to JSON Lines, SQLite, XML, or CSV formats. The library also provides fast HTML-to-