shared-http-cache

---

title: Shared HTTP Cache

description: Node.Js Utility for fetching multiple HTTP resources with browser-like cache management.

---

Overview

$3

This implementation models a shared HTTP cache that follows the semantics defined in RFC 9111, with explicitly documented assumptions and controlled decisions. It uses a content-addressed cache through cacache providing lockless, high-concurrency cache access, that stores along with the downloaded content the associated HTTP response headers in the cache metadata. Overall, the design aims to provide browser-like caching behavior in a Node.js environment.

$3

The cache is shared (not private) and applies shared-cache rules.

- Request methods other than GET are not stored.
- Private responses (request Authorization, response Cache-Control="private" and Set-Cookie headers) not stored.
- Variant responses (response header Vary="*") are not stored.
- Partial content (response Content-Range header) is not stored.
- Time calculations rely exclusively on locally recorded timestamps, not on server-provided Date.
- No heuristic freshness is used.
- Storage and eviction are deterministic; no background or implicit cleanup is assumed.

$3

Each request begins by determining whether a cached response exists.

If no cached entry exists:

- If the request Cache-Control header has only-if-cached directive, the cache returns a 504 HTTP status.
- Otherwise, the request is sent to the origin server.

$3

Two Cache-Control directives may short-circuit normal cache usage:

- no-cache (request or response): cached data cannot be used without revalidation.
- no-store (request or response): the response must not be stored.
When no-store applies, the response is served directly and bypasses storage entirely.

$3

If a cached response exists and is not excluded, strict freshness is evaluated first.

Freshness lifetime is computed from response headers:

- Cache-Control header's s-maxage directive first, then max-age, if present
- otherwise Expires header, if present

Current age is derived from local metadata:

``excel-formula
currentAge = now − storedTime + incomingAge
`

The incomingAge is taken from the stored response Age header, if present.

Remaining freshness:

`excel-formula
remainingFreshness = freshnessLifetime − currentAge
`

If request Cache-Control header includes min-fresh directive, its value is deducted from the remainingFreshness:

`excel-formula
remainingFreshness = remainingFreshness − minimumFreshness
`

If remainingFreshness ≥ 0, the response is served as fresh.

$3

If the response is stale, request Cache-Control header's max-stale directive is evaluated, if present.

If max-stale is present, but its value is unspecified → accept any staleness; otherwise the response is acceptable if:

`excel-formula
currentAge ≤ freshnessLifetime + maximumStaleness
`

If staleness exceeds the acceptable max-stale, the cache proceeds toward revalidation or origin fetch.

$3

Even though the request Cache-Control header's max-stale directive allows use of stale data:

- response Cache-Control header's must-revalidate or proxy-revalidate directives forbid serving stale.
- In that case, the cache must revalidate or fetch from the origin.
- If request Cache-Control header's only-if-cached directive also applies, the cache returns a 504 HTTP status instead of revalidating.
- If no revalidation constraint applies, stale content may be served.

On revalidation, if the cached content includes ETag or Last-Modified, If-None-Match or respectively If-Modified-Since headers are automatically added to the request. Revalidated entries are explicitly replaced during each successful fetch to avoid unbounded growth in the index.

$3

The subresource integrity specifications are implemented on both, fetch and storage:
1. if a request does not provide an integrity hash, then cacache will compute it using its default algorithm: sha512, and subsequent requests may use that hash to retrieve the resource directly from cache using store.get.byDigest function along with the regular search by url store.get.
1. if a request does provide an integrity hash:
- if the related resource is not stored, then node:fetch will use it to verify the incoming response,
- if the related resource is stored and fresh, or is stale but revalidated with the origin server, then cacache will use the hash to:
- get the resource directly from cache if the stored integrity hash matches the one provided, or,
- recompute the path and rebase the resource on the new path if the provided integrity hash is different. In other words, multiple integrity hashes may validate a resource, but only the last provided hash is responsible for its storage path, as cacache can work with a single algorithm at a time. This situation may only be encountered when a resource was initially stored without an integrity hash provided in the request, or when a different integrity hash is provided in the request for the same resource. An exception for resource rebase is the case when a different integrity hash is provided in the request along with Cache-Control header's max-stale directive.

$3

When a request is sent to the origin:

- 2xx: response is stored (unless restricted) and served.
- 304 Not Modified: cached metadata is updated; response is served as fresh.
- 410 Gone: cached entry is removed.
- Other responses: treated as errors and returned directly.

$3

The cache does not:

- apply heuristic freshness
- perform automatic eviction based on staleness

However:

- a 410 Gone response explicitly removes the cached entry.
- additional cleanup mechanisms are available to the user via the underlying storage system.

State diagram

The accompanying state diagram represents the full decision flow:

!Diagram

Legend

1. no-cache may appear on request or response and always requires revalidation.
2. no-store may appear on request or response; See scope and assumptions for storage limitations.
3. Freshness evaluation excludes max-stale that is evaluated only after strict freshness fails.
4. 410 Gone cleanup is an explicit design choice to keep the cache coherent; no heuristic eviction is used.

Install

`bash
npm i shared-http-cache
`

Usage

$3

`ts
new SharedHttpCache(options?) -> SharedHttpCache
`

`js
const SharedHttpCache = require('shared-http-cache');
const sharedHttpCache = new SharedHttpCache();
`

$3

`ts
new SharedHttpCache({ cacheDir?: string, requestTimeoutMs?: number, awaitStorage?: boolean, deferGarbageCollection: boolean }) -> SharedHttpCache
`

- cacheDir: cache storage directory (default .cache).
- requestTimeoutMs: amount of time in milliseconds after which a request is timed out (default: 5000).
- awaitStorage: await cache writes before continuing (default false).
- deferGarbageCollection: defer garbage collection to a later action (default true). If false, the stored content index file is replaced with a clean new one impacting performance.

`js
const sharedHttpCache = new SharedHttpCache({ cacheDir: '/tmp/http-cache', awaitStorage: true, requestTimeoutMs: 1000 });
`

$3

fetch is the only method available. On success, fetch resolves to the same instance, enabling chained workflows.

`ts
sharedHttpCache.fetch(requests) -> Promise
`

Syntax:

`ts
fetch([{ url: string, integrity?: string, options?: RequestInit, callback?: function }]) -> Promise
`

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://example.com/data.txt',
callback: ({ buffer }) => console.log(buffer.toString()),
},
]);
`

$3

Errors encountered during fetches are collected, and the returned promise either resolves with the instance itself for successful fetches or rejects with a list of errors for failed requests.

The response is converted into a Buffer served to callback, then stored in the cache along with the response headers.

`ts
callback({ buffer: Buffer, headers: Headers, fromCache: boolean, index: number }) -> void
`

The callback provided for each request is executed before storing new content, allowing implementers to inspect, transform or validate the data before it's cached. The errors thrown by the callback are also caught and stored in the errors delivered by the Promise.reject().

`js
await sharedHttpCache
.fetch([
{
url: 'https://example.com/data.txt',
callback: ({ buffer, headers, fromCache, index }) => {
console.log(buffer.toString());
console.log(headers);
console.log(index, fromCache);
},
},
])
.catch((errors) => errors.forEach((entry) => console.error(entry.index, entry.url, entry.error.message)));
`

$3

`js
const urls = ['https://example.com/file1', 'https://example.com/file2'];
const parser = ({ url, buffer, headers, fromCache, index }) => {
console.log(index, fromCache, url);
console.log(headers);
console.log(buffer.toString());
};

const requests = urls.map((url) => ({ url, callback: (response) => parser({ ...response, url }) }));

sharedHttpCache.fetch(requests).catch((errors) => errors.forEach((entry) => console.error(entry.index, entry.url, entry.error.message)));
`

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://example.com/file.bin',
integrity: 'sha256-abcdef...',
callback: ({ buffer }) => console.log(buffer.length),
},
]);
`

$3

`ts
fetch.options -> RequestInit
`

fetch.options are passed directly to node:fetch.
They follow standard RequestInit semantics (method, credentials, headers, mode, cache-mode, etc.).

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://api.example.com/list',
options: { headers: { Accept: 'application/json' } },
callback: ({ buffer }) => console.log(buffer.toString()),
},
]);
`

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://example.com/data',
options: { headers: { 'Cache-Control': 'no-cache' } },
callback: ({ fromCache }) => console.log(fromCache),
},
]);
`

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://example.com/data',
options: { headers: { 'Cache-Control': 'max-stale=3600' } },
callback: ({ fromCache }) => console.log(fromCache),
},
]);
`

$3

`js
await sharedHttpCache.fetch([
{
url: 'https://example.com/resource',
options: { method: 'HEAD' },
callback: ({ headers }) => console.log(headers),
},
]);
`

$3

The underlying cache store (cacache) is exposed directly.

`ts
sharedHttpCache.store -> cacache
`

Listing (example with promise)

`js
sharedHttpCache
.fetch(requests)
.then((sharedHttpCache) => sharedHttpCache.store.ls(sharedHttpCache.cacheDir))
.then(console.log)
.catch((errors) => console.error('Errors:', errors));
`

Compacting (example with await)

`ts
sharedHttpCache.store.verify(cacheDir) -> Promise