[help] Pipeline error traces to tar_make instead of a job, makes debugging hard

[dshemetov]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi there, we're having a mysterious error in our pipeline and just wanted to see if any of the errors here could give us a hint for where to look (it's a large pipeline so it takes a long time to debug). What's confusing here is that the error message isn't traced to a particular job, but to tar_make itself. The line "received data could not be converted to integer" might be informative, but that string doesn't appear in targets or crew.

# Lots of dispatched/startup messages prior
# ...
▶ dispatched target forecast_by_ahead_nonmystic.teary.14
received data could not be converted to integer
✔ skipped pipeline [11.184 minutes]
There were 50 or more warnings (use warnings() to see the first 50)
Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    dims [product 315] do not match the length of object [1]
Last error traceback:
    base::tryCatch(base::withCallingHandlers({ NULL base::saveRDS(base::do.c...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]), na...
    doTryCatch(return(expr), name, parentenv, handler)
    tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    base::withCallingHandlers({ NULL base::saveRDS(base::do.call(base::do.ca...
    base::saveRDS(base::do.call(base::do.call, base::c(base::readRDS("/tmp/R...
    base::do.call(base::do.call, base::c(base::readRDS("/tmp/RtmpGb0jWW/call...
    (function (what, args, quote = FALSE, envir = parent.frame()) { if (!is....
    (function (targets_function, targets_arguments, options, envir = NULL, s...
    tryCatch(out <- withCallingHandlers(targets::tar_callr_inner_try(targets...
    tryCatchList(expr, classes, parentenv, handlers)
    tryCatchOne(expr, names, parentenv, handlers[[1L]])
    doTryCatch(return(expr), name, parentenv, handler)
    withCallingHandlers(targets::tar_callr_inner_try(targets_function = targ...
    targets::tar_callr_inner_try(targets_function = targets_function, target...
    do.call(targets_function, targets_arguments)
    (function (pipeline, path_store, names_quosure, shortcut, reporter, seco...
    crew_init(pipeline = pipeline, meta = meta_init(path_store = path_store)...
    self$run_crew()
    self$iterate()
    self$process_target(queue$dequeue())
    self$run_target(name)
    if_any(target_should_run_worker(target), self$run_worker(target), self$r...
    self$run_worker(target)
    self$controller$push(command = command, data = data, globals = globals, ...
    .subset2(control, "push")(command = command, data = data, globals = glob...
    .subset2(self, "scale")(throttle = throttle)
    private$.launcher$scale(demand = self$unresolved(), throttle = throttle)
    self$tally()
    daemons %|||% daemons_info(name = private$.name, seconds_interval = priv...
   daemons_info(name = private$.name, seconds_interval = private$.seconds_i...
    crew_retry(fun = ~{ daemons <- mirai::status(.compute = name)$daemons va...
    isTRUE(do.call(what = fun, args = args, envir = envir))
    do.call(what = fun, args = args, envir = envir)
    (function (..., .x = ..1, .y = ..2, . = ..1) { daemons <- mirai::status(...
    mirai::status(.compute = name)
    query_status(envir)
# ...

We're just stumped and can't tell if it's a bug in our pipeline or it's some sort of parallelization issue that we're having. We were also started receiving this error, but strangely were able to fix it by setting launch_max = 100.

    {crew} worker 15 launched 5 times in a row without completing any tasks. Either troubleshoot or raise launch_max asLast error traceback:

[MilesMcBain]

It looks like this could be coming from somewhere in targets, it's hard to tell.

The way to find out for sure is use the callr_function = NULL method outlined in the manual such that the entire pipeline runs within your interactive session and you'll be able to use all R's debugging tools.

In this context I am a fan of using options(error = recover) to pause evaluation when the error is hit, letting me poke around at various bits of data to see what the problem is. If you've never used recover before, I can say it is WELL WORTH the investment to figure it out, possibly only second to {targets} itself in this regard.

[dsweber2]

I think we're also getting way more R processes spawning than we have workers; parallel::detectCores() - 1L=63, but when I look at ps -e | grep R | wc -l we have 135 lines (1-4 of these may be unrelated, but it's over double the count!). I was expecting to see ~ the same number of R processes as the intended worker count, though this may just be a lack of familiarity with the multi-processing used here.

[shikokuchuo]

Thanks for reporting back @dsweber2. I am glad you are no longer seeing the received data could not be converted to integer error. We will maintain this fix in the main branch going forward. I believe @wlandau has a set of guidance for errors of the type you are getting.

@wlandau FYI the fix in {mirai} reverts the dispatcher control sockets to req/rep - although we were not able to break it in testing, it evidently has in the wild. Although it may only be a symptom of a connection error, the low level error is not easy for you to handle, so this is the best course of action to ensure robustness going forward.

[wlandau]

Yes, glad the error is more interpretable. A more complete reprex would help diagnose the worker crash problem: https://books.ropensci.org/targets/help.html. You could also turn on local logs, c.f. local_log_directory: https://wlandau.github.io/crew/reference/crew_controller_local.html

[wlandau]

If there are still issues with crew workers, please open a new thread and include a small reprex.

[dshemetov]

It's been tough to isolate the issue on our end, but we'll keep you posted.