git-stable version of meta

[jameelalsalam]

I used to .gitignore the entire _targets folder, but I recently tried git tracking the /meta/meta file. So far this seems quite valuable first as a low-effort way to verify whether or not targets have changed and second to detect differences/trigger builds between collaborators.

One problem with git-tracking meta is that there can be changes in it that don't represent changes in the results. To make extra sure I don't have stale cache issues, I occasionally run:

tar_invalidate(everything())
tar_make()

Unfortunately, because the rows are ordered differently, this usually shows as many diffs in the meta file.

Right now I am experimenting with a target which writes out a stable version of the meta in alphabetical order and omitting time and seconds:

tar_target(meta_stable, {
    # mention late targets to push to the end
    my_late_target
    
    targets::tar_meta() %>%
      select(-time, -seconds) %>%
      arrange(type, name) %>%
      write_delim(meta_out, delim = "|", fs::path(tar_config_get("store"), "meta", "meta-stable"))
  }, cue = tar_cue("always"))

This seems to work, but also not ideal to reference meta from inside the pipeline. Is there potential for a way to configure meta to be git-stable?

[wlandau]

Good point on row order, especially since tracking the metadata file is how continuous data versioning with the cloud works in targets (https://books.ropensci.org/targets/storage_amazon.html). 8bf16cb re-sorts the rows after each tar_make(), so I recommend giving that a try.

Removing metadata fields is trickier. seconds is completely for user information, but time is important because it is used in internal decision-making (e.g. to figure out whether a file should be rehashed). As a user, it is best to leave the metadata alone.

[jameelalsalam]

Thanks! I gave the new commit a try, and I think this is a really nice step forward in the usefulness of git-tracking meta. My Git GUI of choice (GitKraken) does a pretty good job of highlighting the characters changed when the row order stays the same, so looking through the diff after invalidate/make cycle, I can definitely see that it is only the time and seconds fields that are changing.

I think my workaround still has some added value, so I will keep experimenting with that as well. Since I am not replacing meta (just making another copy for git purposes) I don't think I am messing up targets' internal workings.

As an aside, I can't overstate how much I appreciate your work on targets. It is has completely changed how I organize my projects -- I don't think of it as a package, but as a piece of infrastructure or an alternative runtime for R. Thank you.