Scrapeer
Extraction

Crawl Website Block

Crawl Website starts at one URL, follows eligible same-site links and XML sitemaps in stable breadth-first order, and stores successful pages as an ordered list.

At a glance

Best for
Multi-page website extraction
Default limit
25 pages, depth 2, 10 minutes
Discovery
Same-site links and XML sitemaps
Local and cloud
Supported

What it does

Crawl Website owns its browser navigation. Connect it after Start, enter a public HTTP or HTTPS URL, and it can begin without a separate Go To URL Block.

The Block admits URLs in stable breadth-first order: the start page is depth 0, pages linked from it are depth 1, and so on. Cloud Runs may read pages at the same depth concurrently, but the output remains in admission order.

Automatic discovery reads same-site page links plus XML sitemap URLs. Every candidate, redirect, and canonical URL must remain on the starting origin, and private or reserved network targets are blocked.

Visible page, depth, duration, retry, pacing, content, and output-size limits keep the dynamic frontier bounded.

Use it when

  • Collect readable content from a documentation, directory, catalogue, or publication website.
  • Discover product, job, article, or profile pages when no complete URL list exists yet.
  • Use an XML sitemap and page links together, while restricting the crawl to selected path prefixes.

Do not use it when

  • You need one known page only; use Go To URL followed by an extraction Block.
  • Pages you need are on another origin or require following cross-site redirects.
  • The target is a private IP address, local service, non-HTTP URL, or downloadable non-HTML file.
  • The site requires an unbounded crawl; increase the visible limits only as far as the Block allows.

Settings

The default view contains the starting URL, page and depth bounds, and output variable. Open Advanced to control discovery, pacing, failures, and page content.

Start URL

Required: Yes
Variables
Yes
Description
Public HTTP or HTTPS page where the crawl starts. This page is depth 0 and must stay on one origin.

Max pages

Required: No
Variables
No
Description
Maximum page attempts, including the start page. Default 25; allowed range 1 to 100.

Link depth

Required: No
Variables
No
Description
Maximum link levels beyond the start page. Default 2; allowed range 0 to 5.

Save pages as

Required: Yes
Variables
Variable name
Description
Name of the ordered list variable that receives successfully read pages. Default crawledPages.

Find pages from

Required: No
Variables
No
Description
Automatic uses page links and XML sitemaps. Advanced alternatives use only links, only sitemaps, or both explicitly.

Sitemap URL

Required: No
Variables
Yes
Description
Optional same-site XML sitemap. Empty checks robots.txt sitemap directives and then /sitemap.xml.

Only these paths

Required: No
Variables
Yes
Description
Optional pathname prefixes to admit, such as /products. Empty allows every eligible same-site path.

Skip these paths

Required: No
Variables
Yes
Description
Pathname prefixes to exclude. Exclusions take priority over inclusions.

Query parameters

Required: No
Variables
No
Description
Remove tracking parameters by default, remove all query parameters, or keep sorted query parameters for deduplication.

Max minutes

Required: No
Variables
No
Description
Maximum wall-clock time for this Block. Default 10 minutes; maximum 60 minutes.

Concurrency

Required: No
Variables
No
Description
Automatic is recommended. Cloud Runs normally use 2 pages at a time and allow a manual request from 1 to 3. Local Runs always use 1.

Delay

Required: No
Variables
No
Description
Minimum delay between page starts on the same website. Automatic uses at least 1 second; a manual value can only slow the crawl.

Page timeout

Required: No
Variables
No
Description
Maximum navigation and extraction time for one page. Default 30 seconds; maximum 120 seconds.

Retries

Required: No
Variables
No
Description
Additional attempts for temporary timeouts, network failures, HTTP 408, HTTP 429, and server errors. Default 1; maximum 2.

On page error

Required: No
Variables
No
Description
Continue records the page error and keeps crawling. Fail Block stops after the first terminal page error and follows the error handle.

Read

Required: No
Variables
No
Description
Read the likely main content area by default, or the full document body.

HTML, links, and metadata

Required: No
Variables
No
Description
Choose optional fields for each successful page. Text is always included; HTML is off by default, while links and metadata are on.

Content caps

Required: No
Variables
No
Description
Bound text and HTML kept per page plus links inspected for discovery and optional output. Defaults are 50000 text characters, 100000 HTML characters, and 200 links.

Outputs

Crawl Website stores successful pages in the configured variable and returns bounded diagnostics for the run.

{{outputVariableName}}array<object>

Successful pages in stable admission order. Each record includes URL, requested URL, depth, admission order, status, title, text, redirect chain, discovery source, attempt count, duration, and truncation state. Enabled content fields add HTML, links, metadata, and canonical URL.

statsobject

Counts for discovery, admission, attempts, successes, failures, exclusions, duplicates, retries, sitemaps, bytes, elapsed time, and configured versus effective pacing and concurrency.

pagesPreview, frontierPreview, errorsPreviewarray<object>

Bounded previews for run inspection. The complete successful-page list remains in pages and the configured output variable.

partial and completionReasonboolean and string

Shows whether a configured limit or page error made the result partial and identifies the reason, such as page, depth, duration, output, or completed-with-errors.

policyVersion, resumable, and sitemapErrorsstring, boolean, and array<string>

Identifies the frontier policy, reports that interrupted crawls cannot resume, and keeps bounded sitemap-discovery errors available for inspection.

Example Flow

Crawl Website can run directly after Start because it owns navigation and produces a page list for downstream Blocks.

  1. Connect Start to Crawl Website.
  2. Set the start URL and visible crawl bounds.
  3. Pass {{crawledPages}} to a downstream transformation, export, or storage Block.

Errors and fixes

Start with symptoms visible in the editor or Inspector, then check the earliest Block that produced the wrong page state, variable, or output.

The Block does not start.

Check
Check that Start URL resolves to a public HTTP or HTTPS page and does not redirect off the starting origin.
Fix
Use the public canonical start URL for the website.

Expected pages are missing.

Check
Inspect Link depth, Max pages, path filters, query policy, completionReason, and frontierPreview.
Fix
Adjust the smallest relevant bound or filter, then run the Flow again.

A sitemap is ignored.

Check
Check sitemapErrors and confirm the sitemap is same-origin XML, reachable, and within the sitemap limits.
Fix
Enter the correct same-site sitemap URL or switch discovery to page links.

The site starts returning temporary errors.

Check
Inspect errorsPreview, retry counts, effective concurrency, and effective delay.
Fix
Set Concurrency to 1 and increase Delay before running the Flow again.

The output contains fewer records than attempted pages.

Check
Compare succeeded and failed in stats. The output variable contains successful pages only.
Fix
Inspect errorsPreview or use Fail Block when any terminal page error should stop the Flow.

Local and cloud runs

Local Runs use the Electron browser sequentially with effective concurrency 1. A manual concurrency setting is retained in the configuration but cannot increase Local Run concurrency.

Cloud Runs use isolated browser pages with Automatic concurrency 2 by default and a hard runtime maximum of 3. The runtime may reduce a requested value.

Both runtimes use the same URL normalization, stable frontier corpus, same-origin policy, visible bounds, retry rules, output shape, and one-second minimum pacing.

Stopping the run cancels active crawl work. Crawl Website reports resumable: false, so an interrupted crawl must be started again rather than continued from its prior frontier.

A Cloud Run keeps the normal 100-credit base estimate and adds 10 credits for Crawl Website once, regardless of pages, redirects, or retries. Ongoing runs reserve another 100 credits at each 50-minute check before the next provider hour; proxy bytes and CAPTCHA solves settle separately. Local Runs currently use zero per-Block credits.

Blocks

Understand how Blocks connect and pass state through a Flow.

Runs and outputs

Inspect Block output, errors, and variables after a run.