[help] Adding custom "iteration" options?

[Aariq]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Is there a way to extend targets to add custom options for the iteration argument? For example, terra SpatRaster or SpatVector objects with multiple layers can be branched with [[]], but need to be re-combined with terra::c() to end up with a SpatRaster or SpatVector object. "vector" iteration doesn't work and "list" iteration doesn't work with crew (njtierney/geotargets#52). What would it take to create a new option for iteration, say "terra_layers"? I'd be happy to attempt a PR to targets if necessary.

[wlandau]

It would not be feasible to customize dynamic branching iteration at the level of targets, and I doubt it would address the source of the problem. njtierney/geotargets#52 is a marshaling issue. It seems like something in terra or geotargets is sometimes producing NULL objects. You could try to reproduce this with a simpler parallel computing package without targets, in e.g. crew or mirai or clustermq directly. Maybe there is a terra function that depends on some part of the global environment which gets lost on parallel workers.

[wlandau]

As a temporary workaround, you could set storage = "worker" and retrieval = "worker" at the level of tar_terra_rast().

[Aariq]

I haven't been able to reproduce this with just crew, but it does seem like a marshaling issue. I'm not sure if this is actually equivalent to how dynamic branching works though: njtierney/geotargets#52 (comment)

[wlandau]

Turns out this was a marshaling bug in targets: #1266. Fixed in #1267.