WaterCrawl

Description

WaterCrawl is a modern web crawling and content extraction platform that transforms any website into structured, LLM-ready data. Built for developers, it lets users crawl any site instantly with smart crawling controls for depth, domains, and paths, precise content extraction using custom selectors that filter out ads and footers, and JavaScript rendering to capture dynamic content and take screenshots in PDF or JPG.

Beyond crawling, WaterCrawl includes a web search engine for real-time results across the web, automatic sitemap generation to map all URLs on a site, and built-in OpenAI integration to turn raw HTML into structured, meaningful data. It offers an extensible plugin system, real-time monitoring of crawl status and performance via Server-Sent Events, and comprehensive error reporting. The project is open source under a modified MIT license, with commercial use as a competing service requiring permission.

Developers integrate WaterCrawl through a clean REST API with JWT-based authentication and webhook support, and official SDKs for Python, Go, Node.js, PHP, and Rust. The API supports starting single or batch crawl requests, configuring spider and page options, streaming live status, and downloading results as JSON or Markdown, plus dedicated search endpoints.

A free plan provides 1,000 page credits per month with datacenter proxies and a 7-day retention window. Paid Startup, Growth, and Enterprise plans add more page credits, higher crawl depth and concurrency, premium residential proxies, more team seats, and longer retention.

WaterCrawl's Core Features

• Smart crawling controls for depth, domains, and paths

• Precise content extraction with custom selectors

• JavaScript rendering and PDF or JPG screenshots

• Built-in web search engine with real-time results

• Automatic sitemap generation and site mapping

• AI-powered processing via OpenAI integration

• Open source with an extensible plugin system

• REST API, webhooks, and SDKs for multiple languages

How to use WaterCrawl?

1. 1

   Sign up free: Create an account and get 1,000 monthly page credits.

2. 2

   Get an API key: Generate a Team API key from the dashboard.

3. 3

   Configure a crawl: Set the start URL, depth, domains, and page options.

4. 4

   Run and monitor: Start the crawl and track progress in real time via SSE.

5. 5

   Retrieve data: Download results as JSON or Markdown, or receive them via webhook.

WaterCrawl's Use Cases

• LLM data pipelines
• Content aggregation
• Market research
• Search engines
• Website monitoring

FAQ from WaterCrawl

• What is WaterCrawl?

  WaterCrawl is a modern, open-source web crawling and content extraction framework that transforms websites into structured, LLM-ready data, built for developers building search engines, research tools, and data pipelines.

• Can WaterCrawl handle JavaScript-rendered pages?

  Yes. WaterCrawl supports JavaScript rendering with configurable wait times to capture dynamic content, and can take screenshots in PDF or JPG format.

• How do I integrate WaterCrawl?

  WaterCrawl provides a well-documented REST API with JWT-based authentication and webhook support, plus official SDKs for Python, Go, Node.js, PHP, and Rust.

• Is WaterCrawl open source?

  Yes. WaterCrawl is released under a modified MIT license. The software can be freely used, modified, and distributed, though commercial use as a service similar to watercrawl.dev requires explicit permission.

• What does the free plan include?

  The free plan includes 1,000 monthly page credits, 100 daily page credits, a max crawl depth of 2, datacenter proxies, API access, and 7-day data retention.

• What output formats are supported?

  Crawl results can be downloaded as JSON or Markdown, and content can be extracted in formats such as HTML and plain text with options to include or exclude links.