Maintenance

Backups, updates, monitoring, and troubleshooting for production instances.

Backups

TimeTiles uses restic  for encrypted, deduplicated backups with optional S3 offsite storage.

Commands

timetiles backup              # Full backup (database + uploads)
timetiles backup --offsite    # Full backup with S3 sync
timetiles backup db           # Database only
timetiles backup uploads      # Uploads only
timetiles backup list         # List snapshots
timetiles backup list --offsite  # List S3 snapshots
timetiles backup verify       # Verify integrity
timetiles backup prune        # Apply retention policy
timetiles backup auto         # Set up daily cron job
timetiles backup auto --offsite  # Daily backups with S3 sync
timetiles backup auto --disable  # Remove the user-crontab job created by auto

Choose one subcommand (default: full). --offsite can precede or follow it; --disable is only valid with auto. Unknown arguments are rejected before backup work starts.

Scheduled Backups

Bootstrap already installs these jobs in /etc/cron.d/timetiles, using the host’s cron timezone:

timetiles backup auto is a separate, optional schedule in the invoking deployment user’s crontab. It creates backups/auto-backup.sh and runs a full backup followed by pruning daily at 02:00. --offsite enables S3 synchronization for that generated script. Bootstrap’s backup jobs do not include --offsite just because an S3 repository is configured.

Do not enable both schedules unintentionally: backup auto does not replace /etc/cron.d/timetiles, and backup auto --disable removes only its user-crontab entry and generated script. If replacing bootstrap’s backup schedule, edit only its backup/prune entries; retain the health-check and nginx reload jobs in the same file. Rerunning bootstrap’s monitoring step recreates that file.

After upgrading the CLI, rerun timetiles backup auto to refresh an existing generated script, including --offsite if it was enabled. Updating the checkout alone does not rewrite that script.

Restore

Each snapshot holds either the database or the uploads. A snapshot ID restores that one half; latest restores the newest db snapshot and the newest uploads snapshot together.

timetiles backup list         # Find the snapshot IDs and their tags
timetiles restore abc123      # Restore that snapshot
timetiles restore abc123 --offsite  # Restore it from S3

Full disaster recovery

timetiles restore latest      # Newest db snapshot + newest uploads snapshot

latest refuses to run unless the repository holds at least one snapshot of each tag, and fails unless both halves were restored. To recover an older state, restore the chosen db and uploads snapshot IDs one after the other. The command reports what it restored, for example Restore complete! Restored: database, uploads.

timetiles restore with no arguments lists the available snapshots, same as timetiles backup list.

Offsite Storage (S3)

Configure in .env.production:

RESTIC_OFFSITE_REPOSITORY=s3:s3.amazonaws.com/your-bucket/timetiles
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

For write-only credentials (recommended), use S3 lifecycle rules to expire old backups instead of timetiles backup prune.

Retention Policy

Default retention: 7 daily, 4 weekly, 12 monthly. All backups are encrypted with RESTIC_PASSWORD.

Updates

# 1. Back up first
timetiles backup
 
# 2. Pull latest images and redeploy
timetiles update
 
# 3. Verify
curl https://your-domain.com/api/health

Database migrations run automatically on container startup — no manual intervention needed.

Monitoring

Health Checks

# Application health
curl https://your-domain.com/api/health
 
# Container status
timetiles status
 
# Live logs
timetiles logs -f

Recommended schedule: health endpoint every 5 minutes, service status hourly, log review daily.

Resource Usage

# Container CPU and memory
docker stats
 
# Database size
docker compose -f deployment/docker-compose.prod.yml exec postgres \
  psql -U timetiles_user -d timetiles -c \
  "SELECT pg_size_pretty(pg_database_size('timetiles'));"
 
# Disk space
df -h

Log Management

All containers use JSON logging with 50 MB max per file, 5 files retained.

timetiles logs web              # Web application logs
timetiles logs worker-ingest    # Import worker logs
timetiles logs postgres         # Database logs
timetiles logs nginx            # Reverse proxy logs

Import Working Files

The hourly ingest-files-cleanup job reclaims files from UPLOAD_DIR/ingest-files after their import has completed or failed and the retention window has elapsed (24 hours by default). It retains the database records and protects files with active or review-stage ingest jobs.

Files without a database reference are removed only after the orphan grace period (12 hours by default). Generated sheet CSVs remain protected while their source file is referenced. Temporary files left by interrupted CSV writes are eligible for the same age-based orphan sweep.

For Compose deployments, set INGEST_FILE_RETENTION_HOURS and INGEST_FILE_ORPHAN_GRACE_HOURS in .env.production to override these defaults. Both require positive whole hours. Recreate the maintenance worker after changing them so it receives the new environment.

Do not clear the upload directory as if it were a cache: active or paused imports may still need its contents. Check the maintenance worker’s logs for cleanup failures before manually removing files.

Database Maintenance

Run weekly for optimal performance:

-- Connect: docker compose exec postgres psql -U timetiles_user timetiles
VACUUM ANALYZE;

Slow Query Debugging

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
 
SELECT mean_exec_time, calls, query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

SSL Certificates

Certificates auto-renew every 12 hours via the certbot container. Verify monthly:

# Check certificate status
timetiles logs certbot
 
# Force renewal if needed
docker compose -f deployment/docker-compose.prod.yml exec certbot certbot renew --force-renewal
docker compose -f deployment/docker-compose.prod.yml restart nginx

Health Check

TimeTiles exposes a health endpoint for monitoring and load balancer integration:

curl https://your-instance.com/api/health

Returns 200 when the system is healthy. Use this for Docker health checks or uptime monitoring (e.g., UptimeRobot , Gatus ).

Troubleshooting

docker system prune affects the entire Docker host, not just the TimeTiles Compose project. Review what other applications use before pruning. Likewise, adding NODE_OPTIONS to .env.production does not automatically pass it into containers: any deliberate heap tuning needs an explicit service environment override and enough memory left for PostgreSQL, workers, buffers, and the operating system.

Debug Mode

# Enable detailed logging
# Edit deployment/.env.production:
LOG_LEVEL=debug
 
timetiles restart
timetiles logs -f web

Next Steps

• Configure environment variables and feature flags
• Customize the UI with themes and branding
• Set usage limits for trust levels and quotas