web / lib/types/ingest-transforms

lib/types/ingest-transforms

Types for import transform rules that handle field mapping during data import.

Import transforms enable schema evolution by allowing datasets to map incoming field names to their canonical schema field names, supporting scenarios like column renames, case variations, and JSON path changes.

Interfaces

RenameTransform

Rename transform - maps source field path to target field path.

Extends

BaseTransform

Properties

id

id: string

Unique identifier for this transform rule

Inherited from

BaseTransform.id

active

active: boolean

Whether this transform is currently active

Inherited from

BaseTransform.active

addedAt?

optional addedAt?: Date

Timestamp when this transform was created

Inherited from

BaseTransform.addedAt

addedBy?

optional addedBy?: string

User ID who created this transform

Inherited from

BaseTransform.addedBy

confidence?

optional confidence?: number

Confidence score if auto-detected (0-100)

Inherited from

BaseTransform.confidence

autoDetected

autoDetected: boolean

Whether this transform was suggested by auto-detection

Inherited from

BaseTransform.autoDetected

type

type: "rename"

from

from: string

Source field path (e.g., “date”, “user.email”)

to

to: string

Target field path (e.g., “start_date”)

DateParseTransform

Date parse transform - converts date strings to standardized format.

Extends

BaseTransform

Properties

id

id: string

Unique identifier for this transform rule

Inherited from

BaseTransform.id

active

active: boolean

Whether this transform is currently active

Inherited from

BaseTransform.active

addedAt?

optional addedAt?: Date

Timestamp when this transform was created

Inherited from

BaseTransform.addedAt

addedBy?

optional addedBy?: string

User ID who created this transform

Inherited from

BaseTransform.addedBy

confidence?

optional confidence?: number

Confidence score if auto-detected (0-100)

Inherited from

BaseTransform.confidence

autoDetected

autoDetected: boolean

Whether this transform was suggested by auto-detection

Inherited from

BaseTransform.autoDetected

type

type: "date-parse"

from

from: string

Source field containing date string

inputFormat

inputFormat: string

Expected input format (e.g., “DD/MM/YYYY”, “MM-DD-YYYY”)

outputFormat

outputFormat: string

Output format (typically “YYYY-MM-DD” for ISO 8601)

timezone?

optional timezone?: string

Optional timezone for parsing

StringOpTransform

String operation transform - applies string manipulations.

Extends

BaseTransform

Properties

id

id: string

Unique identifier for this transform rule

Inherited from

BaseTransform.id

active

active: boolean

Whether this transform is currently active

Inherited from

BaseTransform.active

addedAt?

optional addedAt?: Date

Timestamp when this transform was created

Inherited from

BaseTransform.addedAt

addedBy?

optional addedBy?: string

User ID who created this transform

Inherited from

BaseTransform.addedBy

confidence?

optional confidence?: number

Confidence score if auto-detected (0-100)

Inherited from

BaseTransform.confidence

autoDetected

autoDetected: boolean

Whether this transform was suggested by auto-detection

Inherited from

BaseTransform.autoDetected

type

type: "string-op"

from

from: string

Source field to transform

operation

operation: "replace" | "uppercase" | "lowercase" | "expression"

Operation to apply

pattern?

optional pattern?: string

Pattern for replace operation

replacement?

optional replacement?: string

Replacement string for replace operation

expression?

optional expression?: string

Expression for expression operation (uses expr-eval)

ConcatenateTransform

Concatenate transform - joins multiple fields into one.

Extends

BaseTransform

Properties

id

id: string

Unique identifier for this transform rule

Inherited from

BaseTransform.id

active

active: boolean

Whether this transform is currently active

Inherited from

BaseTransform.active

addedAt?

optional addedAt?: Date

Timestamp when this transform was created

Inherited from

BaseTransform.addedAt

addedBy?

optional addedBy?: string

User ID who created this transform

Inherited from

BaseTransform.addedBy

confidence?

optional confidence?: number

Confidence score if auto-detected (0-100)

Inherited from

BaseTransform.confidence

autoDetected

autoDetected: boolean

Whether this transform was suggested by auto-detection

Inherited from

BaseTransform.autoDetected

type

type: "concatenate"

fromFields

fromFields: string[]

Source fields to concatenate

separator

separator: string

Separator between fields

to

to: string

Target field name for the combined value

SplitTransform

Split transform - splits one field into multiple fields.

Extends

BaseTransform

Properties

id

id: string

Unique identifier for this transform rule

Inherited from

BaseTransform.id

active

active: boolean

Whether this transform is currently active

Inherited from

BaseTransform.active

addedAt?

optional addedAt?: Date

Timestamp when this transform was created

Inherited from

BaseTransform.addedAt

addedBy?

optional addedBy?: string

User ID who created this transform

Inherited from

BaseTransform.addedBy

confidence?

optional confidence?: number

Confidence score if auto-detected (0-100)

Inherited from

BaseTransform.confidence

autoDetected

autoDetected: boolean

Whether this transform was suggested by auto-detection

Inherited from

BaseTransform.autoDetected

type

type: "split"

from

from: string

Source field to split

delimiter

delimiter: string

Delimiter to split on

toFields

toFields: string[]

Target field names for split values

TransformSuggestion

A suggested transform detected by comparing schema versions.

When schema changes are detected (e.g., field removed + field added), the system analyzes whether this represents a rename and suggests a transform rule if confidence is high enough.

Properties

type

type: TransformType

Type of transformation being suggested

from

from: string

Source field path (what’s in the new import file).

This is the field name that appears in the incoming data that doesn’t match the existing schema.

to

to: string

Target field path (what’s in the existing schema).

This is the canonical field name in the dataset schema that the incoming field should be mapped to.

confidence

confidence: number

Confidence score (0-100) based on multiple factors:

Name similarity (Levenshtein distance)
Type compatibility
Common rename patterns
Position proximity in schema

Threshold: >= 70 for suggestion, >= 80 for high confidence

reason

reason: string

Human-readable explanation of why this transform was suggested.

Examples:

“Similar names (87%), Compatible types, Matches common rename pattern”
“Similar names (95%), Compatible types”

Type Aliases

TransformType

TransformType = "rename" | "date-parse" | "string-op" | "concatenate" | "split"

Types of transformations that can be applied to import data.

Supported types:

rename: Map source field path to target field path
date-parse: Parse date strings into standardized format
string-op: Apply string operations (uppercase, lowercase, replace, expression)
concatenate: Combine multiple fields into one
split: Split one field into multiple fields

IngestTransform

IngestTransform = RenameTransform | DateParseTransform | StringOpTransform | ConcatenateTransform | SplitTransform

Union of all transform types.

Applied before schema validation and event creation to normalize incoming data structure to match the dataset’s canonical schema.

Variables

TRANSFORM_TYPE_LABELS

const TRANSFORM_TYPE_LABELS: Record<TransformType, string>

Display labels for transform types

TRANSFORM_TYPE_DESCRIPTIONS

const TRANSFORM_TYPE_DESCRIPTIONS: Record<TransformType, string>

Descriptions for transform types

DATE_FORMAT_OPTIONS

const DATE_FORMAT_OPTIONS: readonly [{ value: "DD/MM/YYYY"; label: "DD/MM/YYYY (31/12/2024)"; }, { value: "MM/DD/YYYY"; label: "MM/DD/YYYY (12/31/2024)"; }, { value: "YYYY-MM-DD"; label: "YYYY-MM-DD (2024-12-31)"; }, { value: "DD-MM-YYYY"; label: "DD-MM-YYYY (31-12-2024)"; }, { value: "MM-DD-YYYY"; label: "MM-DD-YYYY (12-31-2024)"; }, { value: "DD.MM.YYYY"; label: "DD.MM.YYYY (31.12.2024)"; }, { value: "YYYY/MM/DD"; label: "YYYY/MM/DD (2024/12/31)"; }, { value: "D MMMM YYYY"; label: "D MMMM YYYY (31 December 2024)"; }, { value: "MMMM D, YYYY"; label: "MMMM D, YYYY (December 31, 2024)"; }]

Common date format options for date-parse transforms

Functions

isTransformValid()

isTransformValid(transform): boolean

Check if a transform has all required fields configured

Parameters

transform

IngestTransform

Returns

boolean

createTransform()

createTransform(type): IngestTransform

Create a new transform with default values

Parameters

type

TransformType

Returns

IngestTransform