Skip to main content
The HybridBrowserToolkit provides a powerful set of browser automation tools for CAMEL agents. It enables web navigation, form interaction, screenshot capture, and data extraction through a unified interface with TypeScript (WebSocket-based) and Python implementations.

Dual Implementation

Choose between TypeScript (WebSocket-based, recommended) or pure Python (Playwright) implementations based on your needs.

Set-of-Marks (SoM)

Capture annotated screenshots with interactive elements highlighted and numbered, enabling visual reasoning for AI agents.

Persistent Sessions

Maintain browser sessions with user_data_dir, keeping login states and cookies across multiple runs.

CDP Connection

Connect to existing Chrome instances via Chrome DevTools Protocol (CDP) for debugging or reusing browser sessions.
Source Code
  • Toolkit: camel/toolkits/hybrid_browser_toolkit/
  • Example: examples/toolkits/hybrid_browser_toolkit_example.py

Installation

The HybridBrowserToolkit requires Node.js for the TypeScript implementation (recommended) or Playwright for Python mode.

Quick Start

Basic Usage

Initialization

The HybridBrowserToolkit supports extensive configuration options.

Configuration Parameters

Timeout Configuration

Available Tools

Default Tools

The default tool set provides essential browser functionality:

All Available Tools

Use enabled_tools=HybridBrowserToolkit.ALL_TOOLS for full functionality:

Custom Tool Selection

Core Tool Methods

browser_visit_page

Navigate to a URL and get the page snapshot.

browser_back / browser_forward

Navigate through browser history.

Interaction

browser_click

Click on an element by ref ID (from SoM screenshot) or pixel coordinates.

browser_type

Type text into an input field.

browser_scroll

Scroll the page in any direction.

Page Observation

browser_get_som_screenshot

Capture a screenshot with Set-of-Marks annotations. Each interactive element is labeled with a ref ID (e.g., e1, e2).

browser_get_page_snapshot

Get the page structure as text, showing all interactive elements with their ref IDs.

Tab Management

Tab Operations

Console Operations

JavaScript Execution

Advanced Usage

Full Visual Mode

Full Visual Mode is designed for vision-capable models that can reason directly from screenshots using pixel coordinates. When enabled, several key behaviors change:
Tools that normally use ref IDs automatically switch to pixel-based parameters. The docstrings are also updated accordingly - you will only see the pixel-based signatures, not both versions simultaneously.Tools that require ref with no pixel alternative (browser_select, browser_get_page_snapshot, browser_get_som_screenshot) are automatically excluded from the tool list.
browser_get_screenshot returns screenshots with pixel rulers added to the top and left edges. This helps vision models accurately identify pixel coordinates for click and type operations.The rulers show:
  • Major tick marks every 100 pixels with numeric labels
  • Medium tick marks every 50 and 10 pixels
  • Minor tick marks every 5 pixels
When a click does not change the page content (snapshot remains the same), the toolkit detects this as a potentially ineffective click and returns helpful feedback including the 5 nearest interactive elements with their clickable coordinates.Example response:
This helps the model correct its click position without needing another screenshot.

Diff Snapshot for Dropdowns and Autocomplete

When interacting with combobox (dropdown) or textbox (input/textarea) elements, the toolkit intelligently returns a diff snapshot instead of the full page snapshot. This optimization is particularly useful for:
  • Dropdown menus that expand with options
  • Autocomplete/typeahead suggestions
  • Search result suggestions

How Diff Snapshot Works

Trigger elements:
  • combobox - dropdown select elements
  • textbox, input, textarea - text input fields
What’s returned:
  • Only new option and menuitem elements that appeared after the interaction
  • For combobox: includes the combobox’s updated state (since its ref may change after expansion)
Example diff snapshot after clicking a dropdown:
This significantly reduces context size compared to returning the entire page snapshot, helping the model focus on the relevant options.

Viewport Limiting

Reduce context size by only including elements visible in the current viewport:

Action Logging

Enable detailed logging for debugging or replay:

Spreadsheet Operations

The toolkit includes specialized tools for interacting with web-based spreadsheets (Google Sheets, Excel Online):

Integration with ChatAgent

Complete Example

Mode Comparison

Troubleshooting

Ensure Node.js is installed for TypeScript mode, or run playwright install chromium for Python mode.
Use browser_get_page_snapshot to see the current page snapshot. Elements may change refs after actions.
Ensure Chrome is started with remote debugging enabled:
Some websites have advanced bot detection. Try using a persistent user_data_dir with realistic browsing history.