Troubleshooting

Diagnosis and recovery for the TimeTiles ingest pipeline.

Error Model

Task handlers communicate results in three ways:

Multi-sheet workflows (manual-ingest, scheduled-ingest, scraper-ingest) wrap each sheet in a try/catch inside Promise.allSettled. A failure in one sheet does not block others. The markSheetFailed function marks the individual ingest job as FAILED with error details. The parent import file status reflects the aggregate state across all sheets.

The single-job ingest-process workflow (queued after NEEDS_REVIEW approval) does not use per-sheet isolation. Errors propagate normally and Payload’s onFail callback fires as expected.

Note: Payload’s onFail callback does not fire when errors are caught by Promise.allSettled. Multi-sheet workflows use explicit markSheetFailed instead.

Common Issues

NEEDS_REVIEW

Symptoms: Ingest job shows NEEDS_REVIEW status. Admin interface shows pending reviews.

Review reasons (defined in review-checks.ts):

Breaking vs non-breaking schema changes:

• Breaking (require approval): field type changes, required fields removed, constraint narrowing, date format changes, enum value restrictions
• Non-breaking (can auto-approve if autoApproveNonBreaking is set): new optional fields, constraint expansion, enum additions, type generalization

Resolution options:

1. Review and approve changes in admin interface (sets schemaValidation.approved = true)
2. Enable autoGrow on the dataset for future auto-approval of non-breaking changes
3. Configure type transformations to handle known mismatches
4. Reject and fix source data

Duplicate Events

Symptoms: Same event appears multiple times, or expected duplicates are not detected.

Check these settings on the dataset:

• idStrategy.type — is it external (field-based) or computed (hash)?
• For external IDs: verify the field path is correct (case-sensitive) and the field exists in all rows
• For computed hashes: verify the selected fields create unique combinations and avoid unstable fields (timestamps, counters)
• deduplicationConfig.enabled — is deduplication on?
• The duplicates summary on the ingest job shows what was detected

Resolution: Fix the ID strategy configuration, add more fields to the computed hash, or re-import after deleting duplicates.

Geocoding Failures

Symptoms: Events created without coordinates. Geocoding stage shows errors.

Field detection issues:

• Address field name does not match common patterns — add manual field mapping override via geocodingCandidates on the ingest job
• Latitude/longitude field names are non-standard, values are outside valid ranges (-90 to 90, -180 to 180), or fields contain non-numeric data

Provider issues:

• Invalid or expired API key (check geocoding provider configuration in Settings global)
• Rate limit exceeded (the geocode-batch task throws on 429 responses, so Payload retries automatically)
• Provider service outage

Resolution: Fix API configuration, add manual field mappings, or switch geocoding providers.

Schema Conflicts

Symptoms: Schema validation errors, type mismatch errors during import.

Common causes:

• Source data types changed between imports (e.g., a numeric field now contains strings)
• New fields in source data but autoGrow is disabled on the dataset schema
• Values exceed existing constraints (min/max, maxLength, enum set)
• Schema is locked (schemaConfig.locked = true)

Resolution: Add type transformations for known mismatches, enable autoGrow, relax constraints, or approve schema changes via NEEDS_REVIEW flow.

Debugging Tools

Version History

Open the ingest job in the admin interface and navigate to the “Versions” tab. Each stage transition is recorded with timestamps. Look for:

• Long gaps between transitions (bottlenecks)
• Stage transitions that failed and were retried
• Changes in progress, errors, or validation results

Error Logs

• ingestJob.errors — up to 500 retained row-error details. Event creation counts all row failures separately; for a high-row-errors review, reviewDetails.errorCount contains the full count, not the length of this array.
• ingestJob.errorLog — the last failure context (message, context, timestamp), set by markSheetFailed
• ingestJob.reviewReason and ingestJob.reviewDetails — why NEEDS_REVIEW was triggered
• Application logs filtered by ingest job ID show the full processing trace

Database Queries

Direct inspection via the admin interface or database:

• Stuck imports: Query ingest-jobs where stage is not COMPLETED or FAILED and updatedAt is stale
• Schema history: Check dataset-schemas for version progression by dataset
• Event counts: Count events by ingestJob relation to verify expected totals

Recovery

Retry a Failed Job

Endpoint: POST /api/ingest-jobs/{id}/retry

Requires authentication and read access to the ingest job. Queues a new ingest-process workflow starting from analyze-duplicates, rebuilding duplicate and quota-review state before downstream processing. Only works when the job is in FAILED state. Unlike an automatic retry within an existing workflow, this new workflow does not inherit the previous workflow’s completed-task cache.

When to use: After fixing the underlying issue (API key, permissions, network) for a job that exhausted automatic retries.

Reset to a Specific Stage

Endpoint: POST /api/ingest-jobs/{id}/reset

Auth: Admin only.

Body: { "targetStage": "detect-schema" | "analyze-duplicates" | "validate-schema" | "geocode-batch" | "create-events", "clearRetries": true }

Only works for failed jobs. Resets the job stage and queues a new ingest-process workflow. The clearRetries flag (default true) clears the error log.

The workflow resume points are coarser than the accepted target stages:

Cautions: Reset does not delete already-created events. Ensure idempotency before resetting to a stage that writes data. Both retry and reset require the retained source-file reference; if cleanup has reclaimed it, upload the file again. Recovery updates and workflow queueing share a transaction, so a queueing failure rolls back the recovery changes.

NEEDS_REVIEW Resolution

Review the issue in the import wizard. Its review panel submits to POST /api/ingest-jobs/{id}/approve, which checks access and requires the job to be in needs-review. Do not bypass this endpoint with a generic job update: ordinary owners have read access to jobs, but no generic update access.

Approval sets schemaValidation.approved = true. The collection hook normally queues ingest-process from the resume point associated with reviewReason, with these exceptions:

• high-row-error-rate: Events already exist. Approval marks the job completed and updates the parent file’s status without queueing another workflow.
• file-too-large: Approval is rejected because duplicate analysis stopped before producing complete results. Split the source into smaller files and upload again.
• quota-exceeded: Approval requires an admin; owners cannot approve their own quota exception.

Scheduled Import Failures

Each scheduled run creates an independent workflow instance rather than resuming the previous workflow’s task cache. Future automatic runs still require an enabled, valid schedule. Failures increment currentRetries; when it exceeds retryConfig.maxRetries, the scheduled import is disabled. A successful run resets the counter, while a run paused for review does not increment it.

If a scheduled import consistently fails, check its last error, source URL, API keys, and schema configuration. If the retry budget has disabled the schedule, fix the cause before re-enabling it; waiting for the next scheduled time alone will not restart it.