A function to capture all dependency for dynamic branches?

[33Vito]

Hi everyone,

I just starting to use this package, it's absolutely amazing! Thanks @wlandau

I am not sure if there is a better way to do this. I find myself in need of a table that captures all the dependencies between the dynamic branches, which can be joined to the tar_meta() table.

This is my current approach:

Suppose we have a target called raw_data_grouped, which is a grouped data.frame with "group" as the iteration. Each group of the data.frame will enter task1, task2 and task3 in downstream targets based on the map pattern supplied.

tar_branch_mapping <- bind_cols(
  tar_read(raw_data_grouped) %>% 
    distinct(tar_group) %>% 
    arrange(tar_group),
  list(
    tar_branches(task1, pattern = map(raw_data_grouped)),
    tar_branches(task2, pattern = map(task1)),
    tar_branches(task3, pattern = map(task2)),
  ) %>% 
    reduce2(c("task1", "task1", "task2"), 
                function(x, y, by) left_join(x, y, by = by))

My use case is to group the table from tar_meta() by the grouping variable tar_group, so that we can visualise a subset of the table by tar_group in e.g. a Gantt chart (with parallel processing, arranging all targets by time is not ideal sometimes); or to generate better label such as "task1_groupA", "task1_groupB" rather than "task1_25a7518f", etc.

Tony

[wlandau]

That's one way to do it. Another is to have the individual downstream branches keep track of the metadata you want. So instead of a branch that just returns list(x = 1), you could instead have the branch return tibble(value = list(x = 1), task_group = "groupA"). Then tar_read(downstream_dynamic_target) will automatically aggregate all the metadata next to the branches.

[wlandau]

However, the table from tar_meta() does not have a key for me to join the task group, it only has the name of the branch which is like "task1_25a7518f". That's why I need to join the branch dependencies.

What about the parent column?

Taking a step back, what motivates you to join on tar_meta() metadata to the value of a target? If you do this during the actual computation of a target, it could risk being circular.

I am also thinking, if the dynamic branch names generated actually respect the order of the group_keys from the grouped_raw_data, can I just rely on the order/name of the hash keys in the branch name to join the tar_group column, without joining all the dependencies?

The purpose of dynamic branch hashes is to be independent of order. Otherwise, reordering or inserting upstream elements would invalidate more branches than necessary. So if we change this:

# _targets.R
library(targets)
list(
  tar_target(x, c("a", "b")),
  tar_target(y, x, pattern = map(x))
)

to this:

# _targets.R
library(targets)
list(
  tar_target(x, c("a", "INSERTED", "b")),
  tar_target(y, x, pattern = map(x))
)

then reliance on hashes ensures the last branch of target y will stay up to date.

[33Vito]

I am joining the tar_meta() data to the target group name to visualise the start time and finish time for each branch in a Gantt chart. Since the process is parallel, it is useful to understand for each "group" in the raw data, how long does it take for it to go though task1, task2 and task3.

Additionally, I find it often useful to look at the group name and the error messages side-by-side for debug purpose, because the group names can carry a lot of useful information regarding why the process failed for this particular set of interdependent branches.

I see you point about the branch hashes to be independent of order, then is it possible to maybe have an additional column in the tar_meta() to capture the hierarchy somehow?

[33Vito]

What I meant is that for each dynamic branch, it might have upstream or downstream also as dynamic branches with hash names that is not informative. Therefore, we would like to search the hierarchy up and down until we find a branch that is not dynamic, and record the name or group name or map/cross variable names so that it become an meaningful ID for this chain of dynamic branches.

[wlandau]

I am joining the tar_meta() data to the target group name to visualise the start time and finish time for each branch in a Gantt chart. Since the process is parallel, it is useful to understand for each "group" in the raw data, how long does it take for it to go though task1, task2 and task3.

That sound really cool.

I see you point about the branch hashes to be independent of order, then is it possible to maybe have an additional column in the tar_meta() to capture the hierarchy somehow?

Although sorting dynamic branch names by hash does not have any meaning, the children column of tar_meta() does list branch names in the order they are defined. So the information is there.

[33Vito]

Ah... I missed that column, that's great. Thanks!