Troubleshooting

Diagnosis and recovery for the TimeTiles ingest pipeline.

Error Model

Task handlers communicate results in three ways:

Every sheet runs as its own ingest-process job. A thrown error goes through Payload’s retries; once they are used up, the task’s onFail marks the sheet FAILED with error details. A failure in one sheet does not block others. The ingest run of the file then derives the file status from all sheets (see Architecture).

Common Issues

NEEDS_REVIEW

Symptoms: Ingest job shows NEEDS_REVIEW status. Admin interface shows pending reviews.

Review reasons (defined in review-checks.ts):

Breaking vs non-breaking schema changes:

• Breaking (require approval): field type changes, required fields removed, constraint narrowing, date format changes, enum value restrictions
• Non-breaking (can auto-approve if autoApproveNonBreaking is set): new optional fields, constraint expansion, enum additions, type generalization

Resolution options:

1. Review and approve changes in admin interface (sets schemaValidation.approved = true)
2. Enable autoGrow on the dataset for future auto-approval of non-breaking changes
3. Configure type transformations to handle known mismatches
4. Reject and fix source data

Duplicate Events

Symptoms: Same event appears multiple times, or expected duplicates are not detected.

Check these settings on the dataset:

• idStrategy.type — is it external (field-based) or computed (hash)?
• For external IDs: verify the field path is correct (case-sensitive) and the field exists in all rows
• For computed hashes: verify the selected fields create unique combinations and avoid unstable fields (timestamps, counters)
• deduplicationConfig.enabled — is deduplication on?
• The duplicates summary on the ingest job shows what was detected

Resolution: Fix the ID strategy configuration, add more fields to the computed hash, or re-import after deleting duplicates.

Geocoding Failures

Symptoms: Events created without coordinates. Geocoding stage shows errors.

Field detection issues:

• Address field name does not match common patterns — add manual field mapping override via geocodingCandidates on the ingest job
• Latitude/longitude field names are non-standard, values are outside valid ranges (-90 to 90, -180 to 180), or fields contain non-numeric data

Provider issues:

• Invalid or expired API key (check geocoding provider configuration in Settings global)
• Rate limit exceeded (the geocode-batch task throws on 429 responses, so Payload retries automatically)
• Provider service outage

Resolution: Fix API configuration, add manual field mappings, or switch geocoding providers.

Schema Conflicts

Symptoms: Schema validation errors, type mismatch errors during import.

Common causes:

• Source data types changed between imports (e.g., a numeric field now contains strings)
• New fields in source data but autoGrow is disabled on the dataset schema
• Values exceed existing constraints (min/max, maxLength, enum set)
• Schema is locked (schemaConfig.locked = true)

Resolution: Add type transformations for known mismatches, enable autoGrow, relax constraints, or approve schema changes via NEEDS_REVIEW flow.

Debugging Tools

Version History

Open the ingest job in the admin interface and navigate to the “Versions” tab. Each stage transition is recorded with timestamps. Look for:

• Long gaps between transitions (bottlenecks)
• Stage transitions that failed and were retried
• Changes in progress, errors, or validation results

Error Logs

• ingestJob.errors — up to 500 retained row-error details. Event creation counts all row failures separately; for a high-row-errors review, reviewDetails.errorCount contains the full count, not the length of this array.
• ingestJob.errorLog — the last failure context (message, context, timestamp), set by markSheetFailed
• ingestJob.reviewReason and ingestJob.reviewDetails — why NEEDS_REVIEW was triggered
• Application logs filtered by ingest job ID show the full processing trace

Database Queries

Direct inspection via the admin interface or database:

• Stuck imports: Run timetiles ingest recover, or read the ingest-recovery-runs collection; see Recover Imports Whose Worker Died
• Schema history: Check dataset-schemas for version progression by dataset
• Event counts: Count events by ingestJob relation to verify expected totals

Recovery

Recover Imports Whose Worker Died

A worker killed mid-import (deploy past the stop grace period, OOM, host crash) leaves its job processing, and the sheet never moves on. The hourly ingest-recovery task reports such work in ingest-recovery-runs and the worker log; it changes nothing. Resolve it on the host:

timetiles ingest recover          # report: one line per finding plus counters
timetiles ingest recover --apply  # close dead jobs, requeue or fail their sheets

Each finding names its kind (abandoned, orphaned, obsolete) and action. A requeued sheet resumes from its stage; once its recovery attempts reach the stage task’s retries, it fails instead and can be retried by hand. A finding reported as skipped is retried by the next run. See Ingest Recovery.

Retry a Failed Job

Endpoint: POST /api/ingest-jobs/{id}/retry

Requires authentication and read access to the ingest job. Queues a new ingest-process workflow starting from analyze-duplicates, rebuilding duplicate and quota-review state before downstream processing. Only works when the job is in FAILED state. Unlike an automatic retry within an existing workflow, this new workflow does not inherit the previous workflow’s completed-task cache.

When to use: After fixing the underlying issue (API key, permissions, network) for a job that exhausted automatic retries.

Reset to a Specific Stage

Endpoint: POST /api/ingest-jobs/{id}/reset

Auth: Admin only.

Body: { "targetStage": "detect-schema" | "analyze-duplicates" | "validate-schema" | "geocode-batch" | "create-events", "clearRetries": true }

Only works for failed jobs. Resets the job stage and queues a new ingest-process workflow. The clearRetries flag (default true) clears the error log.

The workflow resume points are coarser than the accepted target stages:

Cautions: Reset does not delete already-created events. Ensure idempotency before resetting to a stage that writes data. Both retry and reset require the retained source-file reference; if cleanup has reclaimed it, upload the file again. Recovery updates and workflow queueing share a transaction, so a queueing failure rolls back the recovery changes.

NEEDS_REVIEW Resolution

Review the issue in the import wizard. Its review panel submits to POST /api/ingest-jobs/{id}/approve, which checks access and requires the job to be in needs-review. Do not bypass this endpoint with a generic job update: ordinary owners have read access to jobs, but no generic update access.

Approval sets schemaValidation.approved = true. The collection hook normally queues ingest-process from the resume point associated with reviewReason, with these exceptions:

• high-row-error-rate: Events already exist. Approval marks the job completed without queueing another workflow; the periodic reconcile-ingest-runs task then settles the run.
• file-too-large: Approval is rejected because duplicate analysis stopped before producing complete results. Split the source into smaller files and upload again.
• quota-exceeded: Approval requires an admin; owners cannot approve their own quota exception.

Scheduled Import Failures

Each scheduled run creates an independent workflow instance and its own ingest run. Future automatic runs still require an enabled, valid schedule. A failed run increments currentRetries once, however many of its sheets failed; when it exceeds retryConfig.maxRetries, the scheduled import is disabled. A run counts as failed as soon as one sheet failed, even if others completed. A successful run resets the counter, while a run paused for review does not increment it.

While a run of the schedule awaits review, also next to a failed sheet, the schedule manager does not trigger that schedule. Approve or discard the review to release it.

If a scheduled import consistently fails, check its last error, source URL, API keys, and schema configuration. If the retry budget has disabled the schedule, fix the cause before re-enabling it; waiting for the next scheduled time alone will not restart it.