API reference
Everything here is read from the installed module: the signatures by introspection, the text from the docstrings in src/.
The package
bslz4_to_sparse: decode bitshuffle-lz4/zstd detector frames straight
into sparse pixel lists, optionally with a sparse matrix product (pyFAI
integration) in the same pass.
chunk2sparse(mask, dtype, codec, pipeline) sparsify
chunk2sparseCSC(mask, csc, dtype, codec, pipeline) sparsify + matrix dot
Each object has a `pipeline` (src/_pipeline.py): one value per processing
step, resolved when the object is made (0 / None: the best guess for this
CPU, data and matrix) and kept in `.pipeline`; describe() names the values.
An automatic mask step on uint16 data with masked pixels is settled by the
first frame decoded: 'planes' when most masked pixels hold 65535 there (and
the decode block is at most 8192 pixels, a multiple of 64), 'pixel' otherwise.
Nothing is global, so objects with different pipelines can run in
different threads.
plan(mask, csc, dtype, codec, pipeline) the matrix of a chunk2sparseCSC, prepared once
plan.context(howmany, thresholds) a chunk2sparseCSC on that plan, per thread
A chunk2sparseCSC owns a copy of its matrix (mask folded, analysed, laid out for the dot). A plan
is that copy made once and shared: every object made from it (chunk2sparseCSC(mask, plan, ...) or
plan.context()) builds each layout at most once between them, and only reads the plan's arrays,
which are read-only. Workspace, output buffers and the pipeline stay per object.
Threads: the GIL is released while decoding, and on free-threaded Python
(3.14t, 3.15t) the module runs without a GIL. There are no locks (a plan has one, for building a layout), so:
* use one chunk2sparse / chunk2sparseCSC object per thread; an object
keeps its workspace and output arrays and reuses them on every call;
* the arrays a call returns are those buffers (except coo(), which
copies): the next call on the same object overwrites them;
* leave a call's inputs alone until it returns: do not modify, resize or
close the chunks, the from_buffer base buffer (e.g. an mmap) or the
mask from another thread meanwhile.
Sharing an object between threads, or breaking these rules, gives wrong
results or a crash rather than an error.
chunk2sparse
class chunk2sparse(mask, dtype=numpy.uint16, codec=None, pipeline=None, howmany=None, thresholds=None)
Decode bitshuffle-lz4/zstd chunks of (ni, nj) frames into the pixels
above a cut (and not masked).
mask = detector mask, (ni, nj) uint8, active pixels > 0
dtype = the pixel dtype of the dataset
codec = None (default): found from the first chunk decoded, or CODEC_LZ4 /
CODEC_ZSTD (detect_codec(dataset) reads it from the HDF5 file).
Until then .codec is CODEC_LZ4 and .pipeline is resolved for lz4;
it is resolved again if the first chunk is zstd.
pipeline = None (best guess), or step values: see describe()
howmany = None (default): every pixel above the cut is returned. An
integer: at most that many per frame, chosen as ImageD11's
top_pixels does. A frame with more pixels above the cut keeps
those above the first of the sorted `thresholds` that leaves
fewer than howmany (else above the last), and then the first
howmany of them in pixel order.
thresholds = None: max(cut, 1) * 2**i for i in 0..15, or up to 16 sorted
values. Only used when howmany is set.
cut is compared in the pixel type: for integer pixels a fractional cut is
floored and a cut above the type's maximum selects nothing; a negative cut
is an error.
c(chunk, cut) -> npixels, (values, indices) for one frame (the arrays
are full size: slice [:npixels]); c.multi(chunks, cut) -> the same per
frame for a batch, npixels an array and the arrays (nframes, ni*nj).
After a call, last_cut[i] is the cut frame i ended up with (the call's
cut unless howmany reduced it) and last_nabove[i] the number of pixels
above that cut, before the list was truncated to howmany.
chunk2sparse.call
__call__(self, buffer, cut)
Decode one compressed chunk (one frame): returns npixels, (values,
indices). values and indices are full-frame-size views of this
object's buffers: the first npixels entries are the pixels above
cut (and not masked), in pixel order, and the next call overwrites
them.
chunk2sparse.multi
multi(self, chunks, cut)
Decode a batch of chunks (one frame each) of this dataset:
returns npixels[nframes], (values, indices) of shape (nframes, ni*nj).
chunk2sparse.from_buffer
from_buffer(self, base, offsets, lengths, cut)
As multi, for the chunks at byte offsets[i] (lengths[i]) inside
the one buffer base (an mmap of the HDF5 file, or the file read into
memory): see locate_chunks and offsets_lengths.
chunk2sparse.from_frames
from_frames(self, frames, cut)
As multi, for uncompressed frames: an array (nframes, ni, nj) or (nframes, ni*nj) of this
object's pixel dtype. Only the back half of the pipeline runs (mask, collect, matrix
product, howmany): the same results as multi on the compressed frames. A C-contiguous array
of the right dtype is used in place, else a copy is made.
chunk2sparse.coo
coo(self, buffer, cut)
Decode one chunk to (npixels, row, col, values): uint16 row and
column of each pixel above cut, and its value. These are new arrays
(copies), unlike the buffers __call__ returns.
chunk2sparseCSC
class chunk2sparseCSC(mask, csc, dtype=numpy.uint16, codec=None, pipeline=None, howmany=None, thresholds=None)
As chunk2sparse, and every frame also goes through a sparse matrix
product (a pyFAI integration): powder = csc @ frame.
mask = detector mask, (ni, nj) uint8, active pixels > 0
csc = scipy.sparse matrix (nout, ni*nj), or a pyFAI CSC integrator
(data, indices, indptr, bins): used as it is, float32 weights, float64
powder. Or a plan(mask, csc, ...), which holds the matrix prepared once (mask folded,
analysed, layouts built on demand) for any number of objects; mask must be the one
the plan was made with, else ValueError. Or the result of
fixed_point_matrix(csc, mask, dtype): integer
weights, and the powder is the int64 sums of weight x pixel, exact, not
converted to float: float powder = powder / powder_scale (an integer power
of two, 2**b). With a matrix of weight 1 per pixel (no pixel splitting) these are
exactly the photon counts summed per bin. Either way the matrix is not
changed here: preparing it is a step of its own.
dtype, codec, pipeline: as chunk2sparse; pipeline's dot step picks the
matrix layout (auto: from what the matrix is; "csc-int" for a
fixed_point_matrix, which also takes csc-run-int and, made with wbits=16,
csc-run-int16). A matrix of weight 1 and one bin per pixel (a histogram) of integer
pixels of at most 32 bits is auto "csc-nosplit-int": the sums are accumulated in int64
(no weights, no conversion to float) and returned as float64 like any float dot; a bin
sum above 2**53 is rounded in that last cast
howmany, thresholds: as chunk2sparse; they limit the pixels returned
only: the powder always uses every unmasked pixel
The array of the powder is the object's own and is overwritten by the next call.
The object owns copies of the matrix and the mask, taken here (read-only): later changes to
the arrays you passed do not reach it. The mask is folded into a raw matrix once, here:
masked pixels contribute nothing to the powder (fixed_point_matrix does it for the matrix it
makes). Objects made again from the same csc each repeat this and hold their own copy: for
one object per thread, make a plan(mask, csc, ...) and give it to each in place of csc (or
use plan.context()); they then share the copy and the layouts, each built once.
c(chunk, cut) -> npixels, (values, indices), powder; c.multi(chunks, cut)
-> the same per frame for a batch.
chunk2sparseCSC.call
__call__(self, buffer, cut)
Decompress one chunk, put the pixels above cut into (values, indices) and every pixel
into the matrix product.
returns npixels, (values, indices), powder_sum: values and indices are full-frame-size
views of buffers owned by this object, overwritten by the next call (slice
values[:npixels] and indices[:npixels]; copy what you keep); powder_sum is float64, or
int64 when this object was made with a fixed_point_matrix (divide by powder_scale for float).
chunk2sparseCSC.multi
multi(self, chunks, cut)
Decode a batch of chunks (one frame each) of this dataset:
returns npixels[nframes], (values, indices) of shape (nframes,
ni*nj), powder (nframes, nout).
chunk2sparseCSC.from_buffer
from_buffer(self, base, offsets, lengths, cut)
As multi, for the chunks at byte offsets[i] (lengths[i]) inside
the one buffer base: see locate_chunks and
offsets_lengths.
chunk2sparseCSC.from_frames
from_frames(self, frames, cut)
As multi, for uncompressed frames: an array (nframes, ni, nj) or (nframes, ni*nj) of this
object's pixel dtype. Only the back half of the pipeline runs (mask, collect, matrix
product, howmany): the same results as multi on the compressed frames. A C-contiguous array
of the right dtype is used in place, else a copy is made.
chunk2sparseCSC.coo
coo(self, buffer, cut)
Decode one chunk to (npixels, row, col, values, powder): as
chunk2sparse.coo, plus the frame's powder. All are new arrays
(copies), unlike the buffers __call__ returns.
bslz4_to_sparse
bslz4_to_sparse(ds, num, cut, mask=None, pixelbuffer=None, workspace=None, codec=None, pipeline=None)
Reads a bitshuffle compressed hdf5 dataset and converts this directly
into a sparse format (indices, values) when decoding the data.
ds = hdf5 dataset containing [nframes, ni, nj] pixels
num = frame number to read
cut = threshold, pixels not above this value are ignored (compared in the pixel type)
mask = detector mask. Active pixels > 0.
pixelbuffer = None or (values, indices) storage space
workspace = None or a uint8 array to use as decode scratch space
codec = None to auto-detect lz4 vs zstd, or an explicit CODEC_LZ4 / CODEC_ZSTD
pipeline = None (best guess), or step values: see describe()
returns (number_of_pixels, (values, indices))
plan
plan(mask, csc, dtype=numpy.uint16, codec=None, pipeline=None)
A matrix prepared once for any number of chunk2sparseCSC objects, one per thread: the plan
holds its own read-only copy of csc (mask folded in, or the fixed_point_matrix copied), what
the matrix is (for the automatic dot) and the layouts the objects ask for, each built once
however many ask at the same time. Later changes to the arrays passed in do not reach it.
p = plan(mask, csc, dtype, codec, pipeline)
c = p.context(howmany, thresholds) # or chunk2sparseCSC(mask, p, dtype, ...)
mask, csc, dtype, codec, pipeline: as chunk2sparseCSC. They are the defaults of context();
chunk2sparseCSC(mask, p, ...) takes its own. A plan is never changed by its objects except
by adding a built layout.
fixed_point_matrix
fixed_point_matrix(csc, mask, dtype, maxval=None, wbits=32)
Prepare a csc matrix (scipy csc, pyFAI CSC engine, ...) for exact integer sums: fold the mask
in (masked pixels add nothing), then store the weights as fixed-point integers of wbits (32
or 16) bits. Give the result to chunk2sparseCSC in place of the matrix.
dtype: the pixel dtype of the data (integer). The weights get b fractional bits, b <= wbits
(their width) and b <= 62 - log2(maxval + 1) - log2(the largest sum of weights in a bin), so
that the int64 sums cannot overflow; each doubling of maxval costs a bit. maxval is the
dtype's maximum unless given: the largest value the data can hold. For 16-bit pixels b is
about 31, for 32-bit pixels at their maximum about 17 for a 1D matrix (weight step 2**-17: a
relative error of a few 1e-6 in a bin sum). Pixels above maxval can overflow the sums,
silently: it is a promise about the data. tools/fixed_point_precision.py prints b for a
matrix and data.
Raises ValueError for a matrix that cannot be fixed-point (negative weights beyond rounding
noise, no room for 4 fractional bits). The result has .scale (the integer 2**b to divide the
int64 sums by) and .nbins.
describe
describe(pipeline=None)
The step names of a pipeline (a dict step -> name; None for a step left
auto (0), or not used), or, with no argument, every step's values
with whether this CPU/build can run them (a dict step -> list of
(value, name, usable, what it needs)).
detect_codec
detect_codec(ds)
Detect which block codec (CODEC_LZ4 or CODEC_ZSTD) a dataset's
bitshuffle filter was configured with, from its HDF5 filter pipeline
(cd_values[4]). Falls back to CODEC_LZ4 if it can't be determined.
locate_chunks
locate_chunks(ds)
Where each frame's compressed chunk lies in the HDF5 file: a dict
{frame: (byte_offset, byte_size)} for a (nframes, ni, nj) dataset chunked
one frame per chunk. With offsets_lengths, it lets from_buffer
read the chunks from one buffer holding the file (e.g. an np.memmap)
with no h5py call per frame. Raises ValueError for other chunk shapes,
and for a chunk stored with the bitshuffle filter skipped.
offsets_lengths
offsets_lengths(frame_offsets, frames)
The (offsets int64, lengths int32) arrays from_buffer takes, for the
given frame numbers, from locate_chunks' {frame: (offset, size)}.
decompress
decompress(chunks, dtype, unshuffle=True, out=None, codec=None)
Decompress bitshuffle-lz4/zstd chunks (one frame each, as h5py's
read_direct_chunk gives them) without any sparse processing: the front
half of the pipeline alone. The GIL is released, so threads decompress in
parallel; nothing touches HDF5.
dtype = the pixel dtype of the dataset
unshuffle = True: the pixel values, array (nframes, npixels) of dtype.
False: each block as the HDF5 filter's decompressor yields it,
before the bit unshuffle: uint8 array (nframes, nbytes), for
a later stage (the GPU, from_frames after a separate unshuffle)
out = None, or a writeable C-contiguous array to fill (at least
nframes * the decoded size in bytes), returned
codec = None (found from the first chunk), CODEC_LZ4 or CODEC_ZSTD
decompress_from_buffer
decompress_from_buffer(base, offsets, lengths, dtype, unshuffle=True, out=None, codec=None)
As decompress, for the chunks at byte offsets[i] (lengths[i]) inside the
one buffer base (an mmap of the HDF5 file, or the file read into memory):
see locate_chunks and offsets_lengths.
block_starts
block_starts(chunks, dtype, out=None)
Where each compressed block of each chunk starts, found on the CPU
without decoding: the block index the GPU decoder needs (cuda
unblock='cpu'). Block b holds pixels [b * B, (b + 1) * B) with B = the
chunk's block size in pixels.
Returns starts, a uint32 array (nchunks, nblocks): the byte offset, from
the start of its chunk, of each block's 4-byte big-endian compressed
size; the compressed block follows it. The last block may be partial,
and up to 8 * dtype.itemsize - 1 bytes of pixels after it are stored raw
at the end of the chunk. lz4 or zstd. Every chunk must have the same
total size and block size; a chunk whose blocks do not tile it exactly
raises (corrupt or truncated).
out = None, or a uint32 array of nchunks * nblocks elements to fill
block_starts_from_buffer
block_starts_from_buffer(base, offsets, lengths, dtype, out=None)
As block_starts, for the chunks at byte offsets[i] (lengths[i]) inside
the one buffer base; the starts stay relative to each chunk.
build_info
build_info()
How the loaded native extension was built, as a dict: the package
"version", "git" (`git describe --tags --always --dirty` of the source
tree: a tag when built clean on a tagged commit, "-dirty" when it had
local changes, None outside a git checkout), "modified" (the changed
tracked files when dirty), "src_sha256" (of the compiled sources and
headers, also for builds without git), "compiler", "platform" and
"built_utc".
Constants
| name | value |
|---|---|
version |
'0.0.22a2' |
CODEC_LZ4 |
2 |
CODEC_ZSTD |
3 |
Decode errors
A negative return from the native decode is raised as an Exception with this text (_DECODE_ERRORS in src/__init__.py):
| code | meaning |
|---|---|
| -2 | a chunk failed to decompress (its compressed data is corrupt) |
| -98 | a chunk's decompressed size does not fit in an int |
| -99 | a chunk's decompressed size needs more pixels than the mask has |
| -100 | the cut must not be negative or NaN, and the top_pixels request must be well formed |
| -103 | the workspace or the output buffer is too small |
| -104 | the untranspose kernel reported a failure (corrupt chunk data) |
| -105 | no chunks were given |
| -106 | the chunks do not share the same total size and block size |
| -107 | a chunk is corrupt or truncated (its header, a block length or its raw tail lies outside the chunk) |
| -108 | a chunk offset or size lies outside the buffer it refers to |
| -109 | the dot's matrix layout does not match the decode entry point, or the decoded block size differs from the layout's |
| -110 | a chunk's decompressed size is smaller than the mask (frame and mask shapes differ) |
| -111 | a pipeline step value is unknown, auto (0: resolve it first) or not valid here |
| -112 | a pipeline step value needs an instruction set this CPU/build lacks |
| -113 | the pixel dtype is out of range or unsupported |
| -114 | the cut must be one element of the pixel type |
Generated by tools/generate_docs.py from bslz4_to_sparse 0.0.22a2 (git v0.0.22-13-g610ed27, sources sha256 6929bd75ab7a).