Data collection

Get data from any website with our data collection API.

On this page, we'll dive into the different data collection endpoints you can use to manage data collection programmatically


POST/v1/data-collection/async

Send async data collection job

This endpoint allows you to add a new async job.

When you submit an async job, we will use our most advanced scraping mechanisms to fetch even the most difficult sites in a relatively short time.

Required attributes

  • url
    url
    Type
    string
    Description

    Target url to scrape. Must be url encoded.

Request

Response


GET/v1/data-collection/jobs/{job_id}

Retrieve the status and results of your collection job

This endpoint allows you to retrieve status and result of your asynchronous collection job.

Once your job is finished, you'll receive "response" field which contain the results of your scraping job.

Required attributes

  • job_id
    job_id
    Type
    string
    Description

    Job id, you previously received while sending async job request.

Request

Response


GET/v1/data-collection/jobs

Async jobs list

This endpoint allows you to retrieve a paginated list of all your asynchronous jobs. By default, a maximum of fifteen jobs are shown per page.

Optional attributes

  • Name
    cursor
    Type
    string
    Description

    Use it when you want to fetch the next or previous page.

  • Name
    limit
    Type
    integer
    Description

    Limit the number of items returned.

Request

Response