Skip to content

Example 3: Without HDF5 in the loop: find the chunks once, then read the file and decode by byte offset

Each frame's compressed chunks are contiguous bytes and locate_chunks reads the locations of chunks in the HDF5 file. This only works whenever the HDF5 chunking is set to one frame and will fail otherwise. After indexing, the file is read with ordinary file reads. from_buffer decodes a batch of frames straight from the bytes read, with one release of the GIL for the whole batch.

Source: examples/ex3_batches_and_offsets.py, run as python examples/ex3_batches_and_offsets.py, or download the notebook (figures and output as run when this page was built).

Output, from running it when this page was built, on a synthetic stand-in for your files (powder rings with noise on an Eiger-like detector, tools/standin.py; with your own data the numbers are yours):

import h5py
import hdf5plugin  # registers the bitshuffle filter, so that h5py can decompress too
import numpy as np

import bslz4_to_sparse

h5filename = "eiger_0000.h5"  # your file: bitshuffle-compressed, one frame per chunk
h5dataset = "/entry_0000/ESRF-ID11/eiger/data"  # the (nframes, ni, nj) dataset inside it
mask_filename = (
    "mask.h5"  # your detector mask: a dataset "mask", pyFAI's convention (non-zero = masked)
)
  1. once, with HDF5: where is each chunk, and what are the shape and dtype of the frames?
with h5py.File(h5filename, "r") as h5:
    ds = h5[h5dataset]
    nframes, shape, dtype = ds.shape[0], ds.shape[1:], ds.dtype
    codec = bslz4_to_sparse.detect_codec(ds)
    where = bslz4_to_sparse.locate_chunks(ds)  # {frame: (byte offset, byte size)}
print(
    "%d frames of %s %s; frame 0 is %d bytes at offset %d"
    % ((nframes, shape, dtype) + (where[0][1], where[0][0]))
)
with h5py.File(mask_filename, "r") as h5:
    bad = h5["mask"][()] != 0  # the masked pixels
mask = (~bad).astype(np.uint8)  # ours: 1 = use this pixel

12 frames of (1065, 1030) uint16; frame 0 is 227498 bytes at offset 7632
  1. no HDF5 from here on: the file is read as bytes
file_bytes = np.fromfile(h5filename, dtype=np.uint8)
nbatch = min(nframes, 8)  # the output buffers have one full-size row per frame of the batch
offsets, lengths = bslz4_to_sparse.offsets_lengths(where, range(nbatch))

c2s = bslz4_to_sparse.chunk2sparse(mask, dtype=dtype, codec=codec)
cut = 3
npx, (values, indices) = c2s.from_buffer(file_bytes, offsets, lengths, cut)
print("pixels above the cut of %d, per frame:" % cut, npx.tolist())
print("values and indices have one row per frame:", values.shape, indices.shape)

pixels above the cut of 3, per frame: [7023, 6860, 6918, 6803, 6902, 6985, 6947, 7013]
values and indices have one row per frame: (8, 1096950) (8, 1096950)

with h5py.File(h5filename, "r") as h5:  # only to check the result
    frames = h5[h5dataset][:nbatch]
for k in range(nbatch):
    ref = np.flatnonzero((frames[k].ravel() > cut) & (mask.ravel() > 0))
    assert (indices[k, : npx[k]] == ref).all()
print("every frame matches numpy on the frames HDF5 decompressed")

every frame matches numpy on the frames HDF5 decompressed

A list of chunks held in memory goes through multi() in the same way

chunks = [file_bytes[o : o + n].tobytes() for o, n in zip(offsets.tolist(), lengths.tolist())]
batch_npx = npx.copy()
npx, _ = c2s.multi(chunks, cut)
assert (npx == batch_npx).all()
print("multi() on the same chunks gives the same counts")

multi() on the same chunks gives the same counts

Generated by tools/generate_docs.py from bslz4_to_sparse 0.0.22a2 (git v0.0.22-13-g610ed27, sources sha256 6929bd75ab7a).