web

web / lib/jobs/handlers/url-fetch-job/fetch-utils

lib/jobs/handlers/url-fetch-job/fetch-utils

Fetch utilities for URL fetch jobs.

Contains functions for fetching data from URLs with retry logic, content type detection, error handling, and HTTP caching support.

Interfaces

FetchResult

Properties

data

data: Buffer

contentType

contentType: string

contentLength?

optional contentLength: number

attempts

attempts: number

cacheStatus?

optional cacheStatus: string

FetchOptions

Properties

method?

optional method: string

headers?

optional headers: Record<string, string>

timeout?

optional timeout: number

maxSize?

optional maxSize: number

expectedContentType?

optional expectedContentType: string

Functions

calculateDataHash()

calculateDataHash(data): string

Calculates hash of data for duplicate checking.

Parameters

data

Buffer

Returns

string

detectFileTypeFromResponse()

detectFileTypeFromResponse(contentType, data, sourceUrl): object

Detects file type from content type header or data inspection.

Parameters

contentType

undefined | string

data

Buffer

sourceUrl

string

Returns

object

mimeType

mimeType: string

fileExtension

fileExtension: string

fetchUrlData()

fetchUrlData(sourceUrl, options): Promise<{ data: Buffer; contentType: undefined | string; contentLength: undefined | number; }>

Fetches data from a URL with built-in error handling.

Parameters

sourceUrl

string

options

FetchOptions = {}

Returns

Promise<{ data: Buffer; contentType: undefined | string; contentLength: undefined | number; }>

fetchWithRetry()

fetchWithRetry(sourceUrl, options): Promise<FetchResult>

Parameters

sourceUrl

string

options

FetchOptions & object = {}

Returns

Promise<FetchResult>