xdas.atoms.Atom

class xdas.atoms.Atom

The base class for atoms. Used to implement new Atom objects.

Atoms are the building blocks to perform massive computations. They represent processing units that can be combined to create complex data processing pipelines. Each Atom object is a callable that takes a unique input data object and returns a unique processed data object. Atoms can be stateful meaning that they can store some memory between calls to ensure the continuity of the processing across chunks. The memory of the Atom is stored in the state attribute. The state is a dictionary that is updated during each execution of the Atom. The Atom object is initialized with an initial state at the first call. In subsequent calls, the state is updated if and only if the `chunk_dim` flag is provided along with the dimension along wich chunking was performed. If this flag is not provided, the state is reset between calls. The Atom can be reset manually to its initial state by calling the reset method.

When implementing a new Atom object, the user must sublcass the Atom class, and at minima define the initialize and the call methods. The initialize method is called at the first call to the Atom and is used to initialize the Atom with the input data. The call method is called at each subsequent call to the Atom and is used to perform the main processing logic. The Atom class handles when an how those two methods are called.

To reduce the size of the state that need to be stored, a good practive is to also define the initialize_from_state method. This method is called in the initialize as soon as the minimal set of states is initialized. The other states that are usefull for the processing but that can be recomputed from the minimal set are initialized in the initialize_from_state method.

state

Returns the current state of the atom recursively including the state of nested atoms.

Type:

dict

initialized

Wether the atom has been initialized or not.

Type:

bool

initialize(x, \*\*flags)

Initializes the atom with the given input.

initialize_from_state()

Initializes the atom from its minimal state.

call(x, \*\*flags)

Performs the main processing logic of the atom.

reset()

Resets the atom to its initial state.

__init__()

Methods

__init__()
call(x, **flags)	Process a chunk of data.
initialize(x, **flags)	Initialise the atom from a first chunks of data.
initialize_from_state()	Initialise the atom from its current state.
load_state(path)	Load the atom state from the NetCDF4 file at path.
reset()	Reset all state entries to ... (uninitialised sentinel).
save_state(path)	Serialise the current state to a NetCDF4 file at path.
set_state(state)	Restore the atom state from a previously saved state dict.

Attributes

initialized	True if every state key has been initialised (no ... sentinels remain).
state	Dict of the current state, including nested atom states.