Background Jobs

TimeTiles uses Payload CMS’s built-in job queue and workflow system  for all asynchronous processing. Jobs and workflows are defined in lib/jobs/ and registered through ALL_JOBS in lib/jobs/ingest-jobs.ts and the workflow definitions in lib/jobs/workflows/. See the Payload docs for tasks , workflows , and queues .

Key Behavior

• Auto-deletion: Completed jobs are automatically deleted (deleteJobOnComplete: true)
• Workflow-based orchestration: The ingestion pipeline uses 4 Payload Workflows, queued by collection hooks and trigger services
• 3-queue architecture: ingest (user-facing workflows), default (trigger jobs), maintenance (scheduled system jobs)
• Production: One Docker worker container per queue runs scripts/jobs-worker.ts; see the worker settings below
• Development: Payload’s autoRun schedules and processes all queues within the Next.js process after Payload initializes; no polling process or seeded admin login is required

Native job execution, scheduling, queueing and cancellation are admin-only through jobs.access. The payload-jobs collection’s CRUD permissions are separate and do not guard native job endpoints. Trusted server-side workers use the Local API’s default access override. The generated payload-jobs-stats global is admin-readable and not manually writable, including by admins. It holds scheduling state; Payload’s scheduler maintains it through its database adapter.

Production Workers

deployment/docker-compose.prod.yml is the source of truth for worker commands. Each worker runs scripts/jobs-worker.ts through the Payload CLI and consumes exactly one queue:

Every pass first queues the due tasks of the worker’s queue from their native Payload schedule definitions, then starts jobs until the queue has as many running as its job limit. A long job holds one slot and does not delay the jobs behind it. Run one worker per queue: claims are serialized within a worker, so concurrency keys are only reliable there. Scheduling is filtered by queue, so each worker schedules only its own tasks. The ingest queue receives workflows from application triggers and schedules cache-cleanup, which runs in the worker that owns the URL fetch cache.

On SIGTERM the worker claims no new jobs and lets the running ones finish. A container that stops before its stop grace period runs out leaves no half-done work behind; a killed worker leaves jobs for ingest recovery.

For example, the default-queue worker from apps/web:

pnpm payload run scripts/jobs-worker.ts default 20 60

Arguments are positional: <queue|all> [limit=10] [intervalSeconds=10]. all serves every queue, as the all-in-one image and make jobs do.

Ingest Workflows

The ingestion pipeline is orchestrated by 4 Payload Workflows. The three source workflows detect the sheets of a file and hand them to the dispatch-ingest-run task, which creates the file’s ingest run and queues one ingest-process job per sheet (ADR 0043 ). See Data Ingestion Pipeline for detailed stage documentation.

All ingest workflows run on the ingest queue with per-resource concurrency keys (ingest:manual:{id}, ingest:scheduled:{id}, ingest:scraper:{id}, ingest:process:{ingestJobId}).

Each queueing of ingest-process bumps the sheet’s generation and passes it in the job input. A job whose generation is outdated, or whose sheet already completed, returns before its first task, and a task’s late writes land only while its generation is current.

Error Model

• Throw for transient failures — Payload retries the task; the final failure marks the sheet failed
• Return { needsReview: true } for human review — the sheet pauses in needs-review
• Return data for success — the workflow continues to the next task
• Each sheet runs in its own job, so a failed sheet does not block the others; the run finishes once every sheet is terminal, and a run with a failed sheet counts as failed

For scheduled-ingest, Payload owns the retry decision. The workflow propagates failures without incrementing the schedule’s failure counter. On its next pass, schedule-manager reconciles the current run’s latest Payload job when hasError is final, then plans new runs. This normally happens on the next minute tick while scheduled execution is enabled and workers are processing the queue. The reconciliation locks and re-reads the schedule, ignores jobs older than its current lastRun, and records the failed run only once. Jobs marked error.cancelled are excluded: an explicit cancellation must not consume the schedule’s retry budget. Do not duplicate Payload’s retry calculation in workflow catches.

Stuck-run cleanup cancels old queued jobs through payload.jobs.cancel(), with a processing: false filter applied at cancellation time. Payload preserves completed and already failed jobs; cancelled jobs have error.cancelled: true and no completion timestamp. They are removed by the existing seven-day failed-job retention, not the three-day completed-job retention.

Ingest Recovery

A worker that dies mid-job leaves its Payload job processing, and the job’s concurrency key keeps new jobs for the same sheet or schedule from starting. While a workflow or task runs, its worker touches the job’s updatedAt every minute (lib/jobs/utils/job-heartbeat.ts), also while a task waits on a throttled provider. A processing job silent for longer than ingest.leaseMinutes (default 15, see configuration) belongs to a dead worker.

The hourly ingest-recovery task on the maintenance queue only reports. It logs a warning and writes a row to ingest-recovery-runs when it finds something; changes happen only through timetiles ingest recover --apply. Without --apply the command prints the same report. It distinguishes three kinds:

A requeue resumes from the sheet’s stage and counts in recoveryAttempts; once that count reaches the stage task’s retries, the sheet fails instead. Approval, retry and reset start a fresh count. A sheet that died in create-events first has its events of the dead attempt deleted and its event quota reservation released, under the dataset lease; when the lease is busy, the sheet is skipped and the next run tries again. Deleting an import is refused while one of its jobs holds a live lease.

Task Definitions and Scheduling

The current task inventory is ALL_JOBS in lib/jobs/ingest-jobs.ts. Each handler owns its task slug, retry policy, and any native Payload schedule configuration. Inspect those definitions for exact cron expressions and queue assignments instead of copying a schedule table.

Workflow task slugs can differ from filenames: schema-detection-job.ts registers detect-schema, and create-events-batch-job.ts registers create-events. Use the registered slug when invoking a task.

Post-review processing is not always a schema-version restart. ingest-process supports analyze-duplicates, detect-schema, create-schema-version (default), and create-events as resume points.

Adding a New Job

1. Create handler in lib/jobs/handlers/my-job.ts
2. Import and register the job config in ALL_JOBS in lib/jobs/ingest-jobs.ts
3. Configure its queue, retry policy, and schedule if applicable
4. If it is a workflow task, add it to the appropriate workflow in lib/jobs/workflows/
5. Generate a migration with make migrate-create if the job needs schema changes

Testing Jobs

See Integration Testing Patterns for job testing. Key points:

• Query pending jobs before running (completedAt: { exists: false })
• Verify side effects after successful runs (completed job records are deleted). Final failures remain in payload-jobs and can be inspected.
• Use a bounded drain loop with payload.jobs.run({ allQueues: true }) to process chained workflow tasks in tests