Web CrawlerVersion: v3

Start crawling and discover pages for training

POST/knowledge-bases/crawler

Starts a website crawl that discovers pages for later AI training. WHEN TO USE: use this when you need to scan a site before training; use trainDiscoveredUrls to ingest selected pages; use getCrawlingStatusForLatestOperation to poll progress. RETURNS: the discovery operation id, current status, and the URL being crawled.

Authorizations

Authorizationstringheaderrequired

Bearer authentication header of the form Bearer <token>, where <token> is your access token.

Scope:knowledge-bases.write

Auth methods:OAuthPrivate Integration

Token type:Sub-Account Token

Request

Headers

Versionstringrequired

API Version

Available optionsv3

Example:v3

Body

application/json

locationIdstringrequired

Location ID as string

Example:tDtDnQdgm2LXpyiqYvZ6

urlstringrequired

Website URL as string

Example:https://kubernetes.io/tDtDnQdgm2LXpyiqYvZ6

optionstringrequired

Mode as string

Available optionsExactPathDomain

Example:Exact

knowledgeBaseIdstringrequired

knowledge base ID as string

Example:tDtDnQdgm2LXpyiqYvZ6

preSelectedUrlsstring[]

Pre-selected URLs from sitemap preview — when provided, only these URLs are crawled

Example:["https://example.com/page-1","https://example.com/page-2"]

autoTrainboolean

Automatically train the discovered URLs after crawling

Example:false

Response

application/json

Crawling and discovery started successfully

operationIdstringrequired

Operation ID for tracking the discovery process

Example:op_abc123xyz

statusstringrequired

Current status of the website discovery operation

Available optionsPendingProcessingSuccessfulFailedExistingRestrictedCancelledAbortedTraining

Example:Processing