[ideas] save and read targets as on disk formats

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I see a format = "parquet" but when I tar_readall data is loaded into memory, and crashed the session (> 300Tb).

I was wondering if the was a way to save and read

• data.frame in parquet
• format= HDF5 gene expression as on-disk H5

[multimeric]

As described here, format = "parquet" tells Targets to use arrow::write_parquet() and arrow::read_parquet(). Unfortunately those functions don't actually exploit the parquet format's ability to be queried directly on disk.

My workaround for this type of thing has been to treat the parquet as a file (ie format = "file", and I return the path to that file from my target), but this is admittedly not taking advantage of Targets' abstractions. I think the better idea would be to define a custom format using tar_format that saves to parquet using arrow but that loads it using something like duckdb/dbplyr/duckplyr to take advantage of the disk backing.

[stemangiola]

Same thing If I want to tar_read a huge list of tibbles, and just select few items, instead of needing 300Gb of memory just to read the list, I would use little in case I just need few elements.

[multimeric]

I think that your targets should return a data frame with data as a character vector of filepaths to the parquet files. Then when you need the actual data you can load the parquet as needed (with dbplyr etc). I'm not sure we can expect Targets to do what you want here automatically. Nesting a disk-backed data frame inside another data frame is a bit complex.

[stemangiola]

I see. I will think about it, having files outside the targets framework can have its drawbacks. @wlandau any opinion? How hard would be to implement native on-disk file-system?

[wlandau]

Natively, it doesn't seem feasible because of the number of things in targets which assume a whole target is loaded at a time. But there could be a tar_format()-powered workaround in which each saved file is just a reference to a piece of a large data file, as described in #1232 (reply in thread) for a different situation.

[wlandau]

You might also try #1281 (reply in thread)