Request
Represents a URL to be crawled, optionally including HTTP method, headers, payload and other metadata. The Request
object also stores information
about errors that occurred during processing of the request.
Each Request
instance has the uniqueKey
property, which can be either specified manually in the constructor or generated automatically from the
URL. Two requests with the same uniqueKey
are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes, such
as RequestList
, RequestQueue
or PuppeteerCrawler
.
Example use:
const request = new Apify.Request({
url: 'http://www.example.com',
headers: { Accept: 'application/json' },
});
...
request.userData.foo = 'bar';
request.pushErrorMessage(new Error('Request failed!'));
...
const foo = request.userData.foo;
Properties
Param | Type | |
---|---|---|
id | String |
|
Request ID | ||
url | String |
|
URL of the web page to crawl. | ||
loadedUrl | String |
|
An actually loaded URL after redirects, if present. HTTP redirects are guaranteed to be included. When using | ||
uniqueKey | String |
|
A unique key identifying the request.
Two requests with the same | ||
method | String |
|
HTTP method, e.g. | ||
payload | String | Buffer |
|
HTTP request payload, e.g. for POST requests. | ||
noRetry | Boolean |
|
The | ||
retryCount | Number |
|
Indicates the number of times the crawling of the request has been retried on error. | ||
errorMessages | Array |
|
An array of error messages from request processing. | ||
headers | Object |
|
Object with HTTP headers. Key is header name, value is the value. | ||
userData | Object |
|
Custom user data assigned to the request. | ||
handledAt | Date |
|
Indicates the time when the request has been processed.
Is |
new Request(options)
Param | Type | Default |
---|---|---|
options | object | |
All | ||
options.url | String | |
URL of the web page to crawl. It must be a non-empty string. | ||
[options.uniqueKey] | String | |
A unique key identifying the request.
Two requests with the same If The The Pass an arbitrary non-empty text value to the | ||
[options.method] | String | 'GET' |
[options.payload] | String | Buffer | |
HTTP request payload, e.g. for POST requests. | ||
[options.headers] | Object | {} |
HTTP headers in the following format:
| ||
[options.userData] | Object | {} |
Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries, failures etc. | ||
[options.keepUrlFragment] | Boolean | false |
If | ||
[options.useExtendedUniqueKey] | Boolean | false |
If |
request.pushErrorMessage(errorOrMessage, [options])
Stores information about an error that occurred during processing of this request.
You should always use Error instances when throwing errors in JavaScript.
Nevertheless, to improve the debugging experience when using third party libraries that may not always throw an Error instance, the function performs a type inspection of the passed argument and attempts to extract as much information as possible, since just throwing a bad type error makes any debugging rather difficult.
Param | Type | Default |
---|---|---|
errorOrMessage | Error | String | |
Error object or error message to be stored in the request. | ||
[options] | Object | |
[options.omitStack] | Boolean | false |
Only push the error message without stack trace when true. |