About turbowebfetch

# TurboWebFetch [![npm version](https://img.shields.io/npm/v/turbowebfetch?style=flat-square&logo=npm&logoColor=white)](https://www.npmjs.com/package/turbowebfetch) [![npm downloads](https://img.shields.io/npm/dt/turbowebfetch?style=flat-square&logo=npm&logoColor=white)](https://www.npmjs.com/package/turbowebfetch) [![License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square)](LICENSE) [![GitHub stars](https://img.shields.io/github/stars/aza-ali/turbowebfetch?style=flat-square&logo=github)](https://github.com/aza-ali/turbowebfetch/stargazers) **Real browsers. Real content. Full fidelity.** Your AI agents need to read web pages. Documentation, product info, articles, research. But standard fetch tools use plain HTTP - they cannot handle modern client-side rendering or bot mitigation layers, and return empty shells. TurboWebFetch runs actual Chrome browsers. Your agents see what users see. **14 parallel browsers. Zero API keys. Runs locally.** --- ## Prerequisites Before installing, v ...

a

Published by

aza-ali

Visit View Profile

README.md

View on GitHub

TurboWebFetch

Real browsers. Real content. Full fidelity.

Your AI agents need to read web pages. Documentation, product info, articles, research. But standard fetch tools use plain HTTP - they cannot handle modern client-side rendering or bot mitigation layers, and return empty shells.

TurboWebFetch runs actual Chrome browsers. Your agents see what users see.

14 parallel browsers. Zero API keys. Runs locally.

Prerequisites

Before installing, verify you have:

node --version    # Need 18+
python3 --version # Need 3.8+

Google Chrome must be installed (not Chromium).

Quick Start

claude mcp add turbowebfetch npx turbowebfetch

That's it. Your agents now have access to the fetch and fetch_batch tools.

What This Is (And Isn't)

TurboWebFetch helps your AI agents access content you have the right to access. It renders JavaScript-heavy pages that standard tools cannot handle.

It is for:

Fetching documentation that requires JS rendering (React, Stripe, etc.)
Product research on e-commerce sites
Reading articles and news behind JS walls
Multi-source research for your AI agents

It is not for:

Circumventing paywalls
Scraping data you don't have permission to collect
High-volume data harvesting (rate-limited by design)
Violating websites' Terms of Service

The challenge-handling exists because many legitimate sites use broad bot mitigation that affects even authorized access. If a site restricts access and you don't have permission, respect that.

WebFetch vs TurboWebFetch

Scenario	WebFetch	TurboWebFetch
Static HTML pages	Works	Works (overkill)
JavaScript SPAs	Empty content	Full render
Sites with JS challenges	Fails	Negotiates automatically
Bot mitigation layers	Fails	Negotiates automatically
Parallel agents	One at a time	14 simultaneous browsers
JS-heavy sites (docs, e-commerce)	Blocked or empty	Works

Rule of thumb: Use WebFetch for simple pages. Use TurboWebFetch when that fails.

Usage

Single page:

mcp__turbowebfetch__fetch(url: "https://react.dev/learn", format: "markdown")

Response:

{
  "success": true,
  "url": "https://react.dev/learn",
  "title": "Quick Start - React",
  "content": "# Quick Start\n\nWelcome to the React documentation...",
  "status": 200
}

Batch (parallel):

mcp__turbowebfetch__fetch_batch(
  urls: [
    "https://react.dev/learn",
    "https://nextjs.org/docs",
    "https://www.target.com/p/some-product"
  ],
  format: "text"
)

All three fetch simultaneously in separate browsers.

Parameters

Parameter	Default	Description
`url`	required	The URL to fetch
`format`	`"text"`	`"text"`, `"markdown"`, or `"html"`
`timeout`	`60000`	Milliseconds. Increase to 90000+ for slow sites
`wait_for`	-	CSS selector to wait for (rarely needed)

The tool auto-detects when content has loaded. Use wait_for only if auto-detection fails on a specific site.

Known Limitations

Sites that don't work:

Login-required content - This tool doesn't handle authentication
Interactive CAPTCHAs - It handles JS challenges, not image selection tasks
Zillow - Requires interactive verification
Bloomberg - Requires interactive verification

Performance:

Adds 5-10 seconds per page (browser startup + rendering + human-like behavior)
Memory usage: ~200-400MB per browser instance
For 14 parallel fetches, expect ~4GB RAM usage

Not for scale: This is a user assistant, not a scraping service. Rate-limited to 60 requests/minute per domain.

Configuration

Optional environment variables:

Variable	Default	Description
`TURBOFETCH_MAX_PROCESSES`	`14`	Max concurrent browsers
`TURBOFETCH_HUMAN_MODE`	`true`	Human-like scrolling/delays
`TURBOFETCH_HEADLESS`	`true`	Headless mode (auto-switches if blocked)

Most users won't need to change these.

Troubleshooting

"Python not found"

# macOS
brew install python3

# Ubuntu/Debian
sudo apt install python3 python3-venv

"Chrome not launching"

Install Google Chrome from https://google.com/chrome (not Chromium).

"Content is empty"

Some heavily lazy-loaded sites need an explicit selector:

mcp__turbowebfetch__fetch(
  url: "https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
  wait_for: "[class*=\"product\"]",
  timeout: 90000
)

"Page not loading on [site]"

Some sites require interactive verification that automated browsers cannot complete. Open an issue with the URL.

How It Works

Your agent calls the MCP tool
TurboWebFetch spawns a Python process with Chrome (via nodriver)
Chrome loads the page, executes JavaScript, negotiates any browser challenges
Content is extracted and returned as clean text/markdown/HTML
Browser closes, process exits

Each fetch is isolated. No cookies or state persist between requests.

Development

Clone and build locally:

git clone https://github.com/aza-ali/turbowebfetch.git
cd turbowebfetch
npm install
npm run build

Python setup (virtual environment + nodriver) runs automatically during npm install. If it fails, run manually:

npm run setup:python

Then register with Claude Code:

claude mcp add turbowebfetch node /path/to/turbowebfetch/dist/index.js

License

MIT License. See LICENSE for details.

turbowebfetch