web / lib/jobs/handlers/scraper-repo-sync-job
lib/jobs/handlers/scraper-repo-sync-job
Background job that synchronizes a scraper-repo’s manifest with the scrapers collection.
For git-sourced repos the job performs a shallow clone, reads scrapers.yml,
parses it via the manifest parser, then upserts/deletes scraper records so the
database matches the manifest. For upload-sourced repos it reads the manifest
from the inline code JSON field.
Variables
scraperRepoSyncJob
constscraperRepoSyncJob:object
Type Declaration
slug
slug:
string="scraper-repo-sync"
retries
retries:
number=2
handler
handler: (
context) =>Promise<{ output: \{ created: number; updated: number; deleted: number; success: boolean; }; }>
Parameters
context
Returns
Promise<{ output: \{ created: number; updated: number; deleted: number; success: boolean; }; }>
Last updated on