Maintenance

Backups, updates, monitoring, and troubleshooting for production instances.

Backups

TimeTiles uses restic  for encrypted, deduplicated backups with optional S3 offsite storage.

Bootstrap installs restic and generates RESTIC_PASSWORD. On a manual install, install restic yourself (sudo apt install restic) and set RESTIC_PASSWORD in .env.production; the template leaves it empty and timetiles backup refuses to run without it.

Commands

timetiles backup              # Full backup (database + uploads)
timetiles backup --offsite    # Full backup with S3 sync
timetiles backup db           # Database only
timetiles backup uploads      # Uploads only
timetiles backup list         # List snapshots
timetiles backup list --offsite  # List S3 snapshots
timetiles backup verify       # Verify integrity
timetiles backup prune        # Apply retention policy
timetiles backup auto         # Set up daily cron job
timetiles backup auto --offsite  # Daily backups with S3 sync
timetiles backup auto --disable  # Remove the user-crontab job created by auto

Choose one subcommand (default: full). --offsite can precede or follow it; --disable is only valid with auto. Unknown arguments are rejected before backup work starts.

Scheduled Backups

Bootstrap already installs these jobs in /etc/cron.d/timetiles, using the host’s cron timezone:

timetiles backup auto is a separate, optional schedule in the invoking deployment user’s crontab. It creates backups/auto-backup.sh and runs a full backup followed by pruning daily at 02:00. --offsite enables S3 synchronization for that generated script. Bootstrap’s backup jobs do not include --offsite just because an S3 repository is configured.

Do not enable both schedules unintentionally: backup auto does not replace /etc/cron.d/timetiles, and backup auto --disable removes only its user-crontab entry and generated script. If replacing bootstrap’s backup schedule, edit only its backup/prune entries; retain the health-check and nginx reload jobs in the same file. Rerunning bootstrap’s monitoring step recreates that file.

After upgrading the CLI, rerun timetiles backup auto to refresh an existing generated script, including --offsite if it was enabled. Updating the checkout alone does not rewrite that script.

Restore

Each snapshot holds either the database or the uploads. A snapshot ID restores that one half; latest restores the newest db snapshot and the newest uploads snapshot together.

timetiles backup list         # Find the snapshot IDs and their tags
timetiles restore abc123      # Restore that snapshot
timetiles restore abc123 --offsite  # Restore it from S3

Full disaster recovery

timetiles restore latest      # Newest db snapshot + newest uploads snapshot

latest refuses to run unless the repository holds at least one snapshot of each tag, and fails unless both halves were restored. To recover an older state, restore the chosen db and uploads snapshot IDs one after the other. The command reports what it restored, for example Restore complete! Restored: database, uploads.

timetiles restore with no arguments lists the available snapshots, same as timetiles backup list.

Offsite Storage (S3)

Configure in .env.production:

RESTIC_OFFSITE_REPOSITORY=s3:s3.amazonaws.com/your-bucket/timetiles
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

For write-only credentials (recommended), use S3 lifecycle rules to expire old backups instead of timetiles backup prune.

Retention Policy

Default retention: 7 daily, 4 weekly, 12 monthly. All backups are encrypted with RESTIC_PASSWORD.

Updates

# 1. Back up first
timetiles backup
 
# 2. Pull latest images and redeploy
timetiles update
 
# 3. Verify
curl https://your-domain.com/api/health

Database migrations run automatically on container startup — no manual intervention needed.

A restart lets the ingest worker finish its running imports for up to five minutes before it is stopped. An import cut off after that is reported by the recovery check below.

Monitoring

Health Checks

# Application health
curl https://your-domain.com/api/health
 
# Container status
timetiles status
 
# Live logs (follows by default)
timetiles logs

Recommended schedule: health endpoint every 5 minutes, service status hourly, log review daily.

Resource Usage

# Container CPU and memory
docker stats
 
# Database size (run from the deployment directory; the image needs -h localhost)
docker compose -f docker-compose.prod.yml --env-file .env.production exec postgres \
  psql -h localhost -U timetiles_user -d timetiles -c \
  "SELECT pg_size_pretty(pg_database_size('timetiles'));"
 
# Disk space
df -h

Log Management

All containers use JSON logging with 50 MB max per file, 5 files retained.

timetiles logs --web            # Web application logs
timetiles logs --files          # Log files in /var/log/timetiles/
 
# Any other service, from the deployment directory
docker compose -f docker-compose.prod.yml --env-file .env.production logs -f worker-ingest

Commands on this page that call docker compose directly use the deployment directory (/opt/timetiles on bootstrapped hosts) as the working directory and the default DB_USER and DB_NAME; adjust them if you changed those.

Import Working Files

The hourly ingest-files-cleanup job reclaims files from UPLOAD_DIR/ingest-files after their import has completed or failed and the retention window has elapsed (24 hours by default). It retains the database records and protects files with active or review-stage ingest jobs.

Files without a database reference are removed only after the orphan grace period (12 hours by default). Generated sheet CSVs remain protected while their source file is referenced. Temporary files left by interrupted CSV writes are eligible for the same age-based orphan sweep.

For Compose deployments, set INGEST_FILE_RETENTION_HOURS and INGEST_FILE_ORPHAN_GRACE_HOURS in .env.production to override these defaults. Both require positive whole hours. Recreate the maintenance worker after changing them so it receives the new environment.

Do not clear the upload directory as if it were a cache: active or paused imports may still need its contents. Check the maintenance worker’s logs for cleanup failures before manually removing files.

Interrupted Imports

An import whose worker was killed stays in progress until someone resolves it. The hourly ingest-recovery job looks for such work, logs a warning in the maintenance worker and records each finding in System → Ingest Recovery Runs. It changes nothing on its own:

timetiles ingest recover          # same report on the command line
timetiles ingest recover --apply  # requeue or fail the affected sheets

A requeued sheet resumes where it stopped. A sheet that keeps losing its worker fails after as many attempts as its stage allows and can then be retried through POST /api/ingest-jobs/{id}/retry. ingest.leaseMinutes in timetiles.yml sets how long a silent import counts as running.

Database Maintenance

Run weekly for optimal performance:

-- Connect: docker compose -f docker-compose.prod.yml --env-file .env.production exec postgres psql -h localhost -U timetiles_user timetiles
VACUUM ANALYZE;

Slow Query Debugging

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
 
SELECT mean_exec_time, calls, query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

SSL Certificates

The certbot container checks for renewal every 12 hours. nginx picks up a renewed certificate only when it reloads: bootstrap installs a daily reload cron job; on a manual install, add one yourself (see SSL/TLS). Verify monthly:

# Check certificate status
docker compose -f docker-compose.prod.yml --env-file .env.production logs certbot
 
# Force renewal if needed
docker compose -f docker-compose.prod.yml --env-file .env.production exec certbot certbot renew --force-renewal
timetiles restart nginx

Health Check

TimeTiles exposes a health endpoint for monitoring and load balancer integration:

curl https://your-instance.com/api/health

Returns 200 with {"status":"ok","database":"connected"} when the system is healthy and 503 when the database is unreachable. Use this for Docker health checks or uptime monitoring (e.g., UptimeRobot , Gatus ).

Troubleshooting

docker system prune affects the entire Docker host, not just the TimeTiles Compose project. Review what other applications use before pruning. Likewise, adding NODE_OPTIONS to .env.production does not automatically pass it into containers: any deliberate heap tuning needs an explicit service environment override and enough memory left for PostgreSQL, workers, buffers, and the operating system.

Debug Mode

# Enable detailed logging
# Edit deployment/.env.production:
LOG_LEVEL=debug
 
timetiles restart
timetiles logs --web

Next Steps

• Configure environment variables and feature flags
• Customize the UI with themes and branding
• Set usage limits for trust levels and quotas