HTTP Caching

How TimeTiles caches outgoing HTTP responses for scheduled URL imports.

Purpose

The URL fetch cache reduces redundant network requests when importing data from external APIs:

Avoids re-downloading unchanged data
Respects API rate limits
Saves bandwidth and improves performance

The cache stores complete GET responses. Other methods and requests with a Range header bypass it. 206 Partial Content responses are not stored, and partial responses retained by older versions are not reused.

Supported HTTP Cache Headers

ETag (Entity Tag)

Server provides unique identifier for response version:


Response: ETag: "abc123xyz"
Next Request: If-None-Match: "abc123xyz"
Server Response: 304 Not Modified (if unchanged)

Last-Modified

Server indicates when resource was last updated:


Response: Last-Modified: Wed, 21 Oct 2025 07:28:00 GMT
Next Request: If-Modified-Since: Wed, 21 Oct 2025 07:28:00 GMT
Server Response: 304 Not Modified (if unchanged)

Cache-Control Directives

Server specifies caching behavior:

max-age=N - Freshness lifetime of N seconds, reduced by the response’s age
no-cache - Do not retain the response; the next import fetches it again
no-store - Never cache
private - Not cacheable (response is rejected by the cacheability check, same effect as no-store)
must-revalidate - Once stale, reuse requires successful validation; HTTP or network failures are propagated instead of returning old content

Vary

Vary selects the request headers that must match before a stored response can be reused. For example, Vary: Accept-Language prevents an English response from being reused for a German request. Header names are case-insensitive; an absent header differs from an explicitly empty one. Headers not named by Vary do not affect this comparison.

The cache stores a fingerprint of the selected header values, not an additional plaintext copy. A mismatch fetches a new response and replaces the single entry for that URL/user/authentication key; multiple variants are not retained simultaneously. Legacy entries with Vary but without a fingerprint are fetched again.

Vary: * responses are not retained, even when respectCacheControl is disabled. Changes to Vary supplied by a 304 response are applied before the response is stored again.

Cache Flow

Cache Hit


Request → Check cache → Fresh & Vary matches → Return cached response ✓

Cache Miss


Request → Check cache → Not found → Fetch from URL → Store → Return response

Revalidation


Request → Check cache → Found but stale → Send conditional request with ETag
  → 304 Not Modified → Update cache metadata → Return cached response ✓
  → 200 OK with new data → Update cache → Return new response

HTTP freshness uses the deadline calculated when storing or revalidating the response. Expired entries may still be read for ETag or Last-Modified revalidation until regular cleanup or size-based eviction removes them. Entries without validators are fetched again. Older entries without a freshness deadline are treated as stale, never as fresh hits.

A 304 reuses the stored body and original success status, rather than rewriting the status to 200. Revalidated responses use the same storage policy as fresh responses, including TTL, Vary, and cacheability checks. Its Age and Date replace the old response’s age information; absent values are not inherited from the old response.

TTL Calculation

The cache determines Time-To-Live using this priority order:

Cache-Control: no-store → Don’t cache (TTL = 0)
Cache-Control: no-cache → Don’t cache (TTL = 0)
Cache-Control: max-age=N → Use N seconds minus the response’s current age
Expires header → Derive the lifetime from Expires - Date (using the current time when Date is absent), then subtract the response’s current age
Default TTL → Use cache.urlFetch.defaultTtlSeconds from config/timetiles.yml, minus the response’s current age

The current age is the greater of the apparent age from Date and the upstream Age plus elapsed request/body-transfer time, never below zero. For example, max-age=60 with Age: 45 leaves at most 15 seconds of freshness, not another minute. Non-positive remaining lifetimes produce TTL = 0 and are not retained.

Positive TTLs are capped by cache.urlFetch.maxTtlSeconds. With respectCacheControl: false, freshness uses the configured default TTL (still capped by the maximum); responses marked no-store or private remain excluded by the cacheability check.

Cache Key Generation

Cache identity preserves the URL sent to the server:

Normalization steps:

Hostname lowercased (API.Example.com → api.example.com)
Default ports removed (:80 and :443 stripped)
Fragments removed (#section, not sent in HTTP requests)

Paths (including trailing slashes), query parameter order, and query encoding remain unchanged. Servers can return different responses for these differences.

Example:


Original: https://API.Example.com:443/events/?limit=100&format=json#top
Normalized: https://api.example.com/events/?limit=100&format=json

For the same user and authentication identity, these URLs cache as the same entry:

https://api.example.com/data
https://API.Example.com/data
https://api.example.com:443/data

/data, /data/, /data?b=2&a=1, and /data?a=1&b=2 are distinct cache identities. Keys also include the HTTP method, user ID (or an anonymous marker), and an authentication fingerprint when supplied by the caller. The http:v2: storage namespace prevents reuse of older entries whose normalization conflated these URLs; old entries remain subject to normal cache cleanup.

The normalized URL is hashed with SHA-256 before it becomes part of a stored key. This avoids copying URL credentials into the index and payload metadata. It does not encrypt cached response bodies or headers, and existing entries with plaintext URL keys are not rewritten by this change.

Storage Architecture

File System Backend

The cache uses persistent disk storage:

Structure:


/tmp/url-fetch-cache/
├── index.json          # Cache metadata and key index
├── ab/
│   └── ab….cache       # SHA-256 key hash, first two characters select directory
└── de/
    └── de….cache

Characteristics:

Persists while the configured directory exists; surviving container replacement requires a persistent volume
Configurable directory and size limits (cache.urlFetch.dir and cache.urlFetch.maxSizeBytes)
The cache-cleanup job runs every 6 hours on the ingest queue, in the worker that owns the cache. That worker must run with --handle-schedules, as configured in the production Compose stack. A separate maintenance container cannot clean the ingest container’s private /tmp directory. Size-based eviction also runs when entries are written.
Ordinary cache reads remove expired entries; HTTP revalidation reads may use them until cleanup or eviction
New cache files use Node’s native binary serialization in a versioned, length-checked envelope. User-data keys are not interpreted as binary markers; truncated or appended payloads are rejected. Legacy JSON files and version-one binary envelopes remain readable, but ambiguous user-data markers in old envelopes cannot be repaired retroactively.
Size accounting uses the actual serialized payload-file bytes, including metadata, rather than source-body size or text character counts. The separate index file is not included in this limit.
Index writes are serialized and atomically renamed within one storage instance; the directory must not be shared by independently initialized instances

Relationship to the Generic Cache

UrlFetchCache adds HTTP-specific behavior on top of the shared Cache wrapper and FileSystemCacheStorage. It owns conditional requests, response metadata, and URL/user/authentication cache keys; the storage backend owns serialization, disk accounting, expiration, and eviction.

Cache Headers

The cache adds X-Cache header for debugging:

HIT - Served from cache
MISS - Fetched from origin server
STALE - Cached but expired, fallback used
REVALIDATED - 304 response, cache metadata updated

Integration Points

Used by:

url-fetch-job - Scheduled import URL fetching
Scheduled imports with advancedOptions.useHttpCache: true

Not used by:

Manual file uploads (no URL involved)
Geocoding API calls (uses separate cache)
API endpoint responses (no caching layer)

Design Decisions

Why Filesystem Only?

Responses can survive process restarts when the cache directory is retained
Large response bodies don’t fit well in memory
Disk space more scalable than RAM for caching

Why Not Use CDN?

Scheduled imports run server-side, not from client
Need control over revalidation logic
Privacy: API tokens shouldn’t go through CDN

Usage Limits Configuration - HTTP cache YAML settings
Resource Protection - How caching affects quotas