xdas.processing.DataArrayWriter#

class xdas.processing.DataArrayWriter(dirpath, encoding=None, max_buffers=1, max_workers=1, create_dirs=False)[source]#

A class to handle chunked data egress.

Parameters:

  • dirpath (str or Path) – The directory to store the output of a processing pipeline. The directory needs to exist and be empty.

  • encoding (dict) – The encoding to use when dumping the DataArrays to bytes.

  • max_buffers (int, default=1) – The maximum number of chunks to load into memory at the same time.

  • max_workers (int, default=1) – The maximum number of thread used to load the chunks.

  • create_dirs (bool, optional) – Whether to create parent directories if they do not exist. Default is False.

Examples

>>> import xdas as xd
>>> import xdas.processing as xp

>>> expected = xd.DataArray(np.random.rand(1000, 100), dims=("time", "distance"))

>>> dw = DataArrayWriter("some_path")
>>> for chunk in chunks:
...     dw.submit(chunk)
>>> result = dw.result

>>> assert result.equals(expected)
__init__(dirpath, encoding=None, max_buffers=1, max_workers=1, create_dirs=False)[source]#

Methods

__init__(dirpath[, encoding, max_buffers, ...])

result()

Flush all pending writes and return the concatenated DataArray.

shutdown()

Shut down the internal thread pool.

submit(chunk)

Asynchronously write chunk to disk and register the path for later concat.

write(chunk)

Alias for submit().