JPEG 2000 and the pool

Every ECCC Datamart field is JPEG 2000 packed (DRT 5.40). The core package carries no codec. decodeFieldValues takes a DecodeJ2k function through an injection seam, which keeps the codec dependency out of the browser-safe barrel. Node callers get the wiring from the @azohra/meteo.grib/j2k-node subpath.

The codecs

The default codec is @azohra/meteo.j2k, the workspace’s own pure-TypeScript decoder. It is the default because of how production uses fields. Builders sample fields, reading a handful of site gridpoints out of millions. Under the default codec a sampled decode is a region decode (decodeJ2kRegion), in which only the codeblocks the points touch are entropy-decoded. By contract the result is bit-identical to a full decode, and it is ~16× faster per core on the largest ECCC field. The measured tables are in Performance. Region decoding needs a decoder whose internals this workspace owns. A whole-image codec pays the full-decode price for every sample. Every marker and lifting step in the in-workspace decoder can also be read against its clause of the spec.

Two other options can be selected on the decoder and pool constructors.

codec: "wasm" selects @cornerstonejs/codec-openjpeg, the incumbent WASM OpenJPEG build. It is ~2.6x faster per thread on ≤16-bit full-frame decodes and stays selectable for whole-image workloads. It decodes whole images only, so under it a ≤16-bit sampled request decodes the full frame in the worker and gathers the points. The values are the same, without the region speedup. The package’s main entry is a wasm2js transpile that takes seconds per multi-megapoint field. j2k-node loads the real WebAssembly build behind its ./wasmjs export instead. That build is an order of magnitude faster and bit-identical. The codec clamps samples wider than 16 bits (RAQDPS packs 20), so deep codestreams, sampled or full, route to @azohra/meteo.j2k even under this option. OpenJPEG.js, the asm.js artifact that used to decode them, is retired. It was slower than @azohra/meteo.j2k, and nobody could read its source.

strategy: "codeblock" fans one field’s EBCOT codeblocks across the whole pool over a SharedArrayBuffer tile. OpenJPEG cannot parallelize within a field at all. With this strategy one full decode runs across every worker instead of one, which cuts its latency several-fold, and saturated throughput ties per-field fan-out exactly. Sampled decodes ignore the strategy, because the region path already skips the work a fan-out would parallelize. The strategy requires the pure-TS codec, which is selected automatically when the codec option is unpinned. The WASM codec decodes whole images only.

Every configuration sits behind the one golden gate, in both full and sampled decodes (test/j2k-configs.test.ts).

One decoder or a pool

createNodeJ2kDecoder() returns one in-process, synchronous decoder. The decode example uses it. createNodeJ2kDecoderPool() spawns a worker_threads pool whose async decode drops into decodeFieldValuesAsync and fans whole-field decodes out across cores.

The pool exists because an HRDPS run is thousands of JPEG 2000 fields. Decodes fan out one per worker, and fetch slots stay a separate budget. The pool’s decodeSampled is the production path for point extraction. Under the default codec the worker runs decodeJ2kRegion, GRIB-scales the exact values, and transfers one double per point instead of the whole grid. test/production-codec-throughput.test.ts gates that mechanism and the bit-exactness of sampled values against full ones.

The script below constructs a pool with its options, runs a sampled decode through sampleFieldValuesAsync, and closes the pool, which is mandatory. The output after it comes from this exact script over the committed fixture.

// pool-sample.mjs — run inside grib/ after `mise run build`
import { readFileSync } from "node:fs";
import {
  nearestGridpoint,
  parseFields,
  parseGrid,
  sampleFieldValuesAsync,
  splitMessages,
} from "./dist/index.js";
import { createNodeJ2kDecoderPool } from "./dist/j2k-node.js";

const pool = await createNodeJ2kDecoderPool({
  size: 2, // sizing defaults below
  codec: "j2k", // the default; "wasm" selects the whole-image WASM build
  strategy: "field", // the default; "codeblock" fans one full decode across the pool
});

// The corpus fixture (grib/test/fixtures/ in the repository); any GRIB2
// file works here.
const bytes = readFileSync("test/fixtures/hrdps-continental-tmp-2m.grib2");
const [field] = parseFields(splitMessages(bytes)[0]);
const grid = parseGrid(field.section3);
const site = nearestGridpoint(grid, 49.3634, -117.2361);

// The sampled path: the worker region-decodes and returns one double per point.
const { values } = await sampleFieldValuesAsync(field, [site.index], {
  decodeJ2k: pool.decode, // whole-field decodes and the bitmap fallback
  decodeJ2kSampled: pool.decodeSampled,
});
console.log(`pool of ${pool.size}: 2 m temperature ${(values[0] - 273.15).toFixed(2)} C`);

await pool.close();

pool of 2: 2 m temperature 23.05 C

Sizing and lifetime

size defaults to min(availableParallelism(), 8), at least 1. Size the pool for the machine.

With the default codec the workers carry no WASM heap. Under codec: "wasm" each worker’s Emscripten heap grows to the largest field seen (roughly 60–90 MB after multi-megapoint ECCC fields) and never shrinks.

Workers are real threads, so the pool holds the process open until you call close(). Queued and in-flight decodes reject on close.

The wiring lives in src/j2k-node.ts, and the codecs and the pool’s worker entry live in src/j2k-worker.ts.