Safe way to periodically add arrow RecordBatch to a file

[RayZ0rr]

I have a use case where I want to save some information, which can consist of numpy nd array of variable shape, numpy 1D arrays, objects like pytorch model state_dict, pytorch optimizer state_dict, scalars like floats, ints, strings, custom types like MetricsInfo etc. These formats can be encoded as various datatypes in arrow which is straight forward for primitive types, tricks like #48099 for variable shape tensors and binary type for other python objects.

I only want to use a single file for all these. The information will be generated periodically like after each epoch in training deep learning models. So at each period, say epoch end, I need to save this information to the file. This is important because if the run is interrupted, I don't want to lose all information till the current epoch. Nor do I want to pressure the memory by buffering all information for a single write.

Can this be done with files of arrow or parquet format?

EDIT: after adding the data, I would like to get random access reads to the saved data

[drin]

yes, you can use either. you just need to be able to represent your data as a recordbatch as a precondition (or any type a write function accepts).

[RayZ0rr]

Hey @drin ,

I forgot to mention in my top post but after saving the record batches I would like to get random access reads to the data at specific epoch. For eg: after training the model and doing further analysis, I'll have a list of epoch number with best performance. So, I would like to only get data at these best epoch numbers or single best epoch number

[drin]

you can make epoch a column of your record batch or put some index-like structure in the schema metadata that you can use to identify record batch index from epoch.