JPEG 2000 and the pool

Every ECCC Datamart field is JPEG 2000 packed (DRT 5.40). The core package never carries a codec: decodeFieldValues takes a DecodeJ2k function through an injection seam, so the codec dependency never enters the browser-safe barrel. Node callers get the wiring from the @azohra/meteo.grib/j2k-node subpath.

The codecs

The default codec is @azohra/meteo.j2k, the workspace’s own pure-TypeScript decoder, and the deciding fact is the production shape: builders sample fields — a handful of site gridpoints out of millions — and under the default codec a sampled decode is a region decode (decodeJ2kRegion): only the codeblocks the points touch entropy-decode, bit-identical to a full decode by contract, ~16× faster per core on the largest ECCC field (692 → 44 ms single-threaded, 4 points, Apple M5 Max — the measured tables are in Performance, honestly). Only a decoder whose internals this workspace owns can do that; a whole-image codec pays the full-decode price for every sample. Explainability made the same choice first — every marker and lifting step here is readable against its clause of the spec.

Two options remain selectable on the decoder and pool constructors:

• codec: "wasm" — @cornerstonejs/codec-openjpeg, the incumbent WASM OpenJPEG build, ~2.6x faster per thread on ≤16-bit full-frame decodes, kept selectable for whole-image workloads. It decodes whole images only, so under it a ≤16-bit sampled request decodes the full frame in the worker and gathers the points — same values, none of the region speedup. Its main entry is a wasm2js transpile, seconds per multi-megapoint field; j2k-node loads the real WebAssembly build behind its ./wasmjs export instead — an order of magnitude faster, bit-identical. It clamps samples wider than 16 bits (RAQDPS packs 20), so deep codestreams — sampled or full — route to @azohra/meteo.j2k even under this option. (OpenJPEG.js, the asm.js artifact that used to carry them, is retired — slower than @azohra/meteo.j2k and readable by nobody.)
• strategy: "codeblock" — one field’s EBCOT codeblocks fanned across the whole pool over a SharedArrayBuffer tile, the within-field dimension OpenJPEG cannot use at all: one full decode runs across every worker instead of one, cutting its latency several-fold while saturated throughput ties per-field fan-out exactly. Sampled decodes ignore the strategy — the region path already skips the work a fan-out would parallelize. Requires the pure-TS codec (selected automatically when the option is unpinned) — the WASM codec decodes whole images only.

Every configuration sits behind the one golden gate, full-decode and sampled (test/j2k-configs.test.ts).

One decoder or a pool

createNodeJ2kDecoder() returns one in-process, synchronous decoder — what the decode example uses. createNodeJ2kDecoderPool() spawns a worker_threads pool whose async decode drops into decodeFieldValuesAsync and fans whole-field decodes out across cores.

The pool exists because an HRDPS run is thousands of JPEG 2000 fields: decodes fan out one per worker while fetch slots stay a separate budget. The pool’s decodeSampled is the production path for point extraction — under the default codec the worker region-decodes: it entropy-decodes only the codeblocks the requested gridpoints touch (decodeJ2kRegion), GRIB-scales the exact values, and transfers one double per point instead of the whole grid. Measured through this call path (HRDPS-continental, 4 points, 2-worker pool, Apple M5 Max): 25.5 ms per field sustained against 394 ms per field for full decodes — 15.5× per core. The gate holding that mechanism, plus bit-exactness of sampled against full values, is test/production-codec-throughput.test.ts.

Sizing and lifetime

• size defaults to min(availableParallelism(), 8), at least 1. Size the pool for the machine.
• Heap: with the default codec the workers carry no WASM heap; under codec: "wasm" each worker’s Emscripten heap grows to the largest field seen — roughly 60–90 MB after multi-megapoint ECCC fields — and never shrinks.
• Close it. Workers are real threads; the pool holds the process open until close() is called. Queued and in-flight decodes reject on close.

The wiring lives in src/j2k-node.ts; the codecs and the pool’s worker entry in src/j2k-worker.ts.