xdas.processing.DataArrayLoader#

class xdas.processing.DataArrayLoader(da, chunks, max_buffers=1, max_workers=1)[source]#

A class to handle data chunked data ingestion.

To optimize I/O latencies, chunks are loaded before they are used asynchronously in a buffer as soon as the iterator is created.

Parameters:

  • da (DataArray) – The (virtual) DataArray that contains the data to be chunked

  • chunks (dict) – The sizes of the chunks along each dimension. Needs to be of the form: {"dim": int}. The key correspond with the dimension (usually “time”), and the value is an integer indicating the size of the chunk (in samples) along that dimension.

  • max_buffers (int, default=1) – The maximum number of chunks to load into memory at the same time.

  • max_workers (int, default=1) – The maximum number of thread used to load the chunks.

Examples

>>> import xdas as xd
>>> from xdas.processing import DataArrayLoader
>>> da = xd.open_dataarray(...)

Create chunks along the time dimension

>>> chunks = {"time": 1000}
>>> dl = DataArrayLoader(da, chunks)

Iterate over the chunks

>>> for chunk in dl:
...     process(chunk)
__init__(da, chunks, max_buffers=1, max_workers=1)[source]#

Methods

__init__(da, chunks[, max_buffers, max_workers])

Attributes

nbytes

Total bytes of the underlying DataArray.