browseruse-agent-bench

Landing Page • Issues • Discussions • Leaderboard • Documentation • Dataset

English | 简体中文

Why browseruse-agent-bench

browseruse-agent-bench is a reproducible evaluation framework for browser agents. LexBench-Browser is the built-in public dataset used by the default benchmark workflow. Together they make external results easy to run, compare, cite, and submit back.

Description

browseruse-agent-bench is an all-in-one evaluation framework for AI browser agents, designed to benchmark multiple agents across multiple datasets, browser backends, and models under controlled and reproducible settings. The Python package/CLI is published as browseruse-bench and bubench. It supports both local and cloud browsers, integrates LLM-as-Judge for automated evaluation, and provides a built-in local leaderboard along with efficiency and cost metrics such as agent steps, end-to-end latency, and token usage.

Supported Datasets

• [x] LexBench-Browser — Browser-agent dataset covering e-commerce, social, academic, financial, and other mainstream Chinese/English websites (v1.0, 2026-04-30)
  • All (210, no login required)
  • lexmount (118, mainland-accessible websites) / global (92, international websites)
  • Hugging Face: Lexmount/LexBench-Browser
• [x] Online-Mind2Web — Real website interaction tasks
  • All (300) / Hard (hard subset)
• [x] BrowseComp — Browser operation competition tasks, no login required
  • All (1266)
• [ ] More benchmarks

Details: Benchmarks overview.

Supported Agents & Browsers

Details: Agents overview.

News

• [2026.04.30] 🎉 browseruse-agent-bench v1.0 — initial open-source release. The LexBench-Browser dataset v1.0 ships 210 public tasks across 107 distinct websites with a 6-category × 16-tag robustness label system; reference integrations cover browser-use, skyvern, Agent-TARS and deepbrowse.

Quickstart

1. Clone the repository

git clone https://github.com/lexmount/browseruse-agent-bench.git
cd browseruse-agent-bench

2. Install dependencies (Python>=3.11)

Requires uv (recommended). Select the section for your agent.

Note: browser-use and skyvern have conflicting dependencies and cannot be installed together. If you plan to run multiple agents in parallel, refer to the Environment Isolation section in the documentation.

browser-use

uv sync --extra browser-use
source .venv/bin/activate          # macOS / Linux
.venv\Scripts\Activate.ps1         # Windows PowerShell

skyvern

uv sync --extra skyvern
source .venv/bin/activate          # macOS / Linux
.venv\Scripts\Activate.ps1         # Windows PowerShell

Agent-TARS (requires Node.js 18+)

uv sync
npm install -g @agent-tars/[email protected]
source .venv/bin/activate          # macOS / Linux
.venv\Scripts\Activate.ps1         # Windows PowerShell

After activation, the bubench CLI is available on your PATH. Without activation, prefix every bubench … command in the following steps with uv run (e.g. uv run bubench run …).

3. Configure

Principle: .env holds sensitive credentials (API keys). config.example.yaml → config.yaml (git-ignored) holds all agent, model, browser, and eval settings in one place.

3.1 Shared credentials (.env)

cp .env.example .env
vim .env

3.2 Runtime config (config.yaml)

cp config.example.yaml config.yaml
vim config.yaml

All agents are configured in one file. Runtime config is resolved as: agents.<agent> + models.<model> + browsers.<browser>.

To switch per run, use --model <name> and --browser-id <name>.

4. Install Skills (Optional)

bubench skills

Installs the prebuilt developer-friendly skills pack (browseruse_bench/skills/) into your agent toolchain.

5. Run & Evaluate

Run

bubench run --agent {AGENT} --data {BENCHMARK} --mode first_n --count 3
# Output: experiments/{benchmark}/{split}/{agent}/{model_id}/{timestamp}/

# Example: LexBench-Browser (no login required)
bubench run --agent browser-use --data LexBench-Browser --mode first_n --count 3
# Output: experiments/LexBench-Browser/All/browser-use/gpt-4.1/20260101_120000/

Evaluate

bubench eval --agent {AGENT} --data {BENCHMARK} --model-id {MODEL_ID}

# Example
bubench eval --agent browser-use --data LexBench-Browser --model-id gpt-4.1

--split is optional — the benchmark's default_split (from data_info.json) is used automatically. Pass --split <name> only to override the default. For the full parameter reference, see the Quickstart docs.

Data Loading

Use --data-source to control where benchmark data is loaded from:

Speed up in China: Set HF_ENDPOINT=https://hf-mirror.com in .env. Private datasets: Set HF_TOKEN=hf_your_token_here in .env.

Details: Data Loading.

📖 For complete guides, API reference, and more examples, see the full documentation.

Leaderboard

We provide an interactive local leaderboard to compare agent performance across benchmarks.

Generate leaderboard HTML:

bubench leaderboard

Deploy leaderboard service (temporary process):

bubench server --host 0.0.0.0 --port 8012 &

Deploy leaderboard service (systemd):

sudo bubench service install
sudo bubench service start

See Leaderboard Documentation for more details.

Access URLs (default port 8012):

• Local leaderboard: http://localhost:8012
• Local API docs: http://localhost:8012/docs
• Remote leaderboard: http://<SERVER_IP>:8012/
• Remote API docs: http://<SERVER_IP>:8012/docs

Visualization

An interactive experiment explorer for browsing agent trajectories, evaluation details, and per-task API logs — complements the static leaderboard with task-level drill-down.

# Start server (auto-regenerates index when experiment files change)
bubench viz --watch

# Access at http://localhost:8080

Options:

bubench viz --port 8090              # custom port (default: 8080)
bubench viz --generate-only          # regenerate experiments.json and exit
bubench viz --watch-interval 5       # poll interval in seconds (default: 3)

For remote sharing with tmux and firewall configuration, see Visualization Documentation.

Acknowledgements

Some code in this project is cited and modified from Online-Mind2Web and simple-evals.

Citation

@misc{lexbench_browser_2026,
    title        = {LexBench-Browser: A Real-World Browser Agent Benchmark with Long-Tail and Multilingual Tasks},
    author       = {Lexmount Research and Collaborators},
    year         = {2026},
    howpublished = {\url{https://lexmount.github.io/browseruse-agent-bench/}},
    note         = {Open benchmark; v1.0 reference release},
}

Contact

Questions, benchmark proposals, agent integrations, and result reproductions are welcome:

• Report bugs or request features in GitHub Issues.
• Ask questions and discuss results in GitHub Discussions.
• Email official result, dataset, and collaboration questions to [email protected].
• Track upcoming releases in Milestones.
• Use Contributing when opening pull requests or adding a new agent/benchmark.
• See Governance and Evaluation Protocol for result review rules.

Coming Soon

• 🔐 Login-state preservation — first-class support for reusing browser login across eval runs, so login-gated tasks can be benchmarked end-to-end without manual re-login. Stay tuned.

Roadmap/ Development Plan

Refer to our Milestones for upcoming versions and deadlines.

Star History