ftsdata.apply_blockwise

ftsdata.apply_blockwise(func, data, axis, n_out, *, mask=None, out_dtype, bytes_work_per_pix, mem_safety=0.8, small_cube_bytes=10000000.0, ncpu=None, mem_available=None, use_dask=None)

Apply func to a cube in parallel, over blocks of pixels.

The transform axis is never sliced: every task receives the full extent of data along axis and returns n_out samples along that axis. The remaining (pixel) axes are split into contiguous blocks, one per thread of a concurrent.futures.ThreadPoolExecutor, or one per dask worker if use_dask is True.

Parameters:

funccallable

Function applied to each block, called as func(block, mask_block, workers=...). block is a numpy.ndarray with the pixel axes restricted to the block and the transform axis complete. mask_block is the matching slice of mask, or None if mask is None. It must return an array with the same shape as block, except along axis where it must have n_out samples.

dataarray_like

Input cube. It is only read, never modified in place.

axisint

Index of the transform axis, kept whole in every block.

n_outint

Size of the output along axis.

maskarray_like, optional

Boolean mask with the same shape as data, passed to func block by block. Default is None.

out_dtypenumpy.dtype

dtype of the allocated output array.

bytes_work_per_pixfloat

Estimated peak memory, in bytes, used by func per output pixel. Together with the available memory and the number of threads, it sets the size of a block.

mem_safetyfloat, optional

Fraction of the currently available memory that the output plus the working blocks may use. Default is 0.8.

small_cube_bytesfloat, optional

If the total estimated work is below this value, or if data has no pixel axis, func is called once on the whole cube with workers=ncpu instead of block by block. Default is 1e7.

ncpuint, optional

Number of worker threads. Default is None, which uses ftsdata._utils.cpu_count().

mem_availablefloat, optional

Available memory, in bytes. Default is None, which uses ftsdata._utils.available_memory().

use_daskbool, optional

If True, use dask to process the cube in parallel. If None (default), use dask if available. If False, use ThreadPoolExecutor.

Returns:

outnumpy.ndarray

Output array with the same shape as data, except along axis where it has n_out samples.

Raises:

MemoryError

If the output alone does not fit in the available memory, or if one row of pixels does not fit in the per-thread memory budget.

Notes

When ncpu and mem_available are not given, they are read with ftsdata._utils.cpu_count() and ftsdata._utils.available_memory().

In the block path, func is called with workers=1 because parallelism already comes from the thread pool/dask; the single-call path uses workers=ncpu. func may ignore that keyword if it does not parallelise internally.