web / lib/jobs/handlers/url-fetch-job/paginated-fetch
lib/jobs/handlers/url-fetch-job/paginated-fetch
Paginated JSON API fetching for scheduled ingests.
Handles offset-based, page-based, and cursor-based pagination strategies, collecting all records across multiple pages into a single result array.
Interfaces
PaginationConfig
Properties
enabled
enabled:
boolean
type
type:
"offset"|"cursor"|"page"
pageParam?
optionalpageParam?:string
Query param name for offset or page number. Default: “page”.
limitParam?
optionallimitParam?:string
Query param name for per-page limit. Default: “limit”.
limitValue?
optionallimitValue?:number
Number of records per page. Default: 100.
cursorParam?
optionalcursorParam?:string
Query param name for cursor value (cursor pagination only).
nextCursorPath?
optionalnextCursorPath?:string
Dot-path to next cursor value in the JSON response.
totalPath?
optionaltotalPath?:string
Dot-path to total record count in the JSON response.
maxPages?
optionalmaxPages?:number
Safety limit on pages fetched. Default: 50, hard cap: 500.
PaginatedFetchOptions
Properties
authHeaders?
optionalauthHeaders?:Record<string,string>
timeout?
optionaltimeout?:number
cacheOptions?
optionalcacheOptions?:object
useCache
useCache:
boolean
bypassCache
bypassCache:
boolean
respectCacheControl
respectCacheControl:
boolean
PaginatedFetchResult
Properties
allRecords
allRecords:
Record<string,unknown>[]
pagesProcessed
pagesProcessed:
number
totalRecords
totalRecords:
number
Functions
fetchPaginated()
fetchPaginated(
baseUrl,paginationConfig,recordsPath,options):Promise<PaginatedFetchResult>
Fetches all pages from a paginated JSON API endpoint and collects the records into a single array.
Supports three pagination strategies:
- offset — uses a numeric offset incremented by
limitValue - page — uses a page number starting at 1
- cursor — uses a cursor value extracted from each response
Parameters
baseUrl
string
The API endpoint URL (may already contain query params)
paginationConfig
Pagination strategy and parameter configuration
recordsPath
string | undefined
Dot-path to the records array in each response, or
undefined to auto-detect
options
Authentication headers, timeout, and cache settings
Returns
Promise<PaginatedFetchResult>
All collected records with page and count metadata