Requirements Desk

Working operationweb dataextractionscraping alternative

Read a public web page into structured text

Fetch one public web page and return its title, description, headings, readable text, links and embedded JSON-LD, honouring robots.txt.

What it is for

An agent or analyst often needs a page's actual content in a clean structure (headings, text, links, schema.org data) rather than raw HTML, to quote it, compare offers or extract terms. This fetches one public page and returns that structure, refusing pages the site asks robots not to fetch.

Developers and agents that need page content without running a browser, and researchers collecting a few pages' text with their sources.

What happens to a request

The page is fetched once from the public internet, after its robots.txt is checked for our user agent. Private, internal and this service's own addresses are refused before any request; at most three redirects are followed, each checked again; 5 MB and 10 seconds are the limits. The answer is built from that one response; JavaScript is not run.

What you get

A JSON answer with the final URL, status, title, meta description, canonical link, headings, readable text (up to 100,000 characters), up to 300 links and any JSON-LD blocks.

Scope and limits

Any public website reachable over HTTP or HTTPS on the standard ports.

What you can specify

Operations

POST /web-extraction/op/extract

One public page as title, description, headings, text, links and JSON-LD.

Price: free, within a daily budget.

ParameterTypeMeaning
url (required)stringhttp or https URL of a public page

Example answer:

{
  "url": "https://example.com/",
  "final_url": "https://example.com/",
  "status": 200,
  "content_type": "text/html; charset=UTF-8",
  "title": "Example Domain",
  "description": null,
  "canonical": null,
  "headings": [
    {
      "level": 1,
      "text": "Example Domain"
    }
  ],
  "text": "Example Domain This domain is for use in documentation examples without needing permission.",
  "links": [
    {
      "href": "https://iana.org/domains/example",
      "text": "Learn more"
    }
  ],
  "json_ld": [],
  "robots": "allowed",
  "truncated": false
}

Sources: The page you name, fetched once (The page's own terms; we store only the analysis of your request, not the page). Limits: 5 MB response, 10 seconds; three redirects; robots.txt honoured.

Next step

Call POST /web-extraction/op/extract with {"url": "https://..."} (free, within a daily budget). Or describe a crawl or extraction job this does not cover.

Or send JSON to POST /web-extraction/requests (schema: /web-extraction/schema.json). Example:

{
  "action": "retrieve",
  "need": "Read the pricing page at https://www.cloudflare.com/plans/ and give me its headings and text.",
  "specifications": {
    "url": "https://www.cloudflare.com/plans/",
    "fields": [
      "headings",
      "text"
    ]
  }
}

Sources for statements on this page