xdas.processing.DataFrameWriter#

class xdas.processing.DataFrameWriter(path, parse_dates=None, create_dirs=False)[source]#

A class for writing pandas DataFrames to a CSV file asynchronously.

Parameters:

  • path (str) – The path to the csv file.

  • parse_dates (bool, int, optional) – Whether to parse dates when reopening the csv file at the end of the process

  • create_dirs (bool, optional) – Whether to create parent directories if they do not exist. Default is False.

Examples

>>> import pandas as pd
>>> import xdas.processing as xp

>>> dw = xp.DataFrameWriter("output.csv")
>>> for df in dfs:
...     dw.submit(dfs).
>>> result = dw.result()

>>> expected = pd.concat(dfs, ignore_index=True)
>>> assert result.equals(expected)
__init__(path, parse_dates=None, create_dirs=False)[source]#

Methods

__init__(path[, parse_dates, create_dirs])

result()

Flush pending writes and return the full CSV as a pandas.DataFrame.

shutdown()

Shut down the internal thread pool.

submit(df)

Asynchronously append df to the CSV file.

write(df)

Alias for submit().