web

web / lib/jobs/handlers/dataset-detection-job

lib/jobs/handlers/dataset-detection-job

Defines the job handler for detecting datasets within an uploaded file.

This job is the first step in the import process after a file is uploaded. It performs the following actions:

• Reads the uploaded file (supports CSV and Excel formats).
• Identifies all the individual sheets (for Excel) or the single data table (for CSV).
• For each detected sheet, it creates a corresponding import-jobs document.
• It either matches the sheet to an existing dataset in the specified catalog or creates a new dataset.
• It populates the import-jobs with initial metadata like row count and sets the first processing stage to DEDUPLICATION.

Variables

datasetDetectionJob

const datasetDetectionJob: object

Type declaration

slug

slug: "dataset-detection" = JOB_TYPES.DATASET_DETECTION

handler()

handler: (context) => Promise<{ output: { sheetsDetected: number; importJobsCreated: number; }; }>

Parameters

context

JobHandlerContext

Returns

Promise<{ output: { sheetsDetected: number; importJobsCreated: number; }; }>