web / lib/jobs/handlers/schema-detection-job

lib/jobs/handlers/schema-detection-job

Defines the job handler for detecting the schema from imported data.

This single job streams all batches from the import file to progressively build a schema. It skips rows that were identified as duplicates to ensure the schema is based on unique data.

Key responsibilities include:

Using a ProgressiveSchemaBuilder to infer data types and properties for each column.
Detecting fields that could be used for geocoding (e.g., address, latitude, longitude).
Storing the evolving schema and the builder’s state in the import-jobs document.

After processing all batches, the import job transitions to the SCHEMA_VALIDATION stage.

Variables

schemaDetectionJob

const schemaDetectionJob: object

Type Declaration

slug

slug: "detect-schema" = JOB_TYPES.DETECT_SCHEMA

handler()

handler: (context) => Promise<{ output: \{ totalBatches: number; totalRowsProcessed: number; }; }>

Parameters

context

JobHandlerContext

Returns

Promise<{ output: \{ totalBatches: number; totalRowsProcessed: number; }; }>