Add LLM tools for R package help & vignettes #8421
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
E2E Tests 🚀 |
seeM
previously approved these changes
Jul 8, 2025
| ]); | ||
| } | ||
|
|
||
| if (!options.input.packageName) { |
There was a problem hiding this comment.
Can we use inputSchema: { required: ['packageName']} in the package.json instead, here and for the other tools? I haven't looked into what exactly it does.
There was a problem hiding this comment.
Sure, done here: 752df47
| ]); | ||
| } | ||
|
|
||
| const helpTopics = await session.callMethod('list_package_help_topics', options.input.packageName); |
There was a problem hiding this comment.
Should we try-catch callMethod here and for the other tools? I got this error at first:
Failed to process positron.ui request: Error evaluating '.ps.rpc.get_help_page("glyphInfo", "grDevices")': pandoc is not available. Please install pandoc or set the RSTUDIO_PANDOC environment variable to the directory containing the pandoc executable.
R backtrace: 0. pandoc_convert(tmp_rd_file, to = to, ...)
format_help_page_markdown(help_page, output = md_file, options = c("--shift-heading-level-by=1")) at modules/positron/llm_tools.R:219:5
.ps.rpc.get_help_page("glyphInfo", "grDevices") at modules/positron/help.R:460:5
R thread backtrace: 0: std::backtrace_rs::backtrace::libunwind::trace at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9 1: std::backtrace_rs::backtrace::trace_unsynchronized at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14 2: std::backtrace::Backtrace::create at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/backtrace.rs:331:13 3: harp::exec::try_catch::handler::{{closure}} at ./posit/ark/crates/harp/src/exec.rs:237:30 4: harp::exec::try_catch::handler at ./posit/ark/crates/harp/src/exec.rs:213:27 5: do_internal at /Volumes/Builds/R4/R-4.3.2/src/main/names.c:1404:11 6: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 7: R_execClosure 8: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 9: bcEval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:7414:12 10: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1013:8 11: R_execClosure 12: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 13: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1140:12 14: vsignalError at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:1903:3 15: Rf_errorcall at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:912:5 16: do_stop at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:1365:7 17: bcEval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:7446:14 18: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1013:8 19: R_execClosure 20: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 21: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1140:12 22: do_begin at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2798:10 23: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 24: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 25: do_begin at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2798:10 26: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 27: R_execClosure 28: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 29: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1140:12 30: do_begin at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2798:10 31: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 32: R_execClosure 33: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 34: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1140:12 35: do_begin at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2798:10 36: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1092:12 37: R_execClosure 38: Rf_applyClosure at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:2113:16 39: Rf_eval at /Volumes/Builds/R4/R-4.3.2/src/main/eval.c:1140:12 40: libr::r::Rf_eval at ./posit/ark/crates/libr/src/functions.rs:31:21 41: harp::exec::try_eval::{{closure}} at ./posit/ark/crates/harp/src/exec.rs:96:41 42: harp::exec::try_catch::callback at ./posit/ark/crates/harp/src/exec.rs:200:31 43: R_withCallingErrorHandler at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:2613:16 44: libr::r::R_withCallingErrorHandler at ./posit/ark/crates/libr/src/functions.rs:31:21 45: harp::exec::try_catch::{{closure}} at ./posit/ark/crates/harp/src/exec.rs:272:9 46: harp::exec::top_level_exec::callback at ./posit/ark/crates/harp/src/exec.rs:342:31 47: R_ToplevelExec at /Volumes/Builds/R4/R-4.3.2/src/main/context.c:799:2 48: libr::r::R_ToplevelExec at ./posit/ark/crates/libr/src/functions.rs:31:21 49: harp::exec::top_level_exec at ./posit/ark/crates/harp/src/exec.rs:345:14 50: harp::exec::try_catch at ./posit/ark/crates/harp/src/exec.rs:271:20 51: harp::exec::try_eval at ./posit/ark/crates/harp/src/exec.rs:96:19 52: harp::exec::RFunction::call_in at ./posit/ark/crates/harp/src/exec.rs:88:9 53: harp::exec::RFunction::call at ./posit/ark/crates/harp/src/exec.rs:83:9 54: ark::ui::ui::UiComm::handle_call_method::{{closure}} at ./posit/ark/crates/ark/src/ui/ui.rs:191:26 55: ark::r_task::r_task::{{closure}} at ./posit/ark/crates/ark/src/r_task.rs:214:44 56: core::ops::function::FnOnce::call_once{{vtable.shim}} at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 57: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1966:9 58: harp::exec::try_catch::callback at ./posit/ark/crates/harp/src/exec.rs:200:31 59: R_withCallingErrorHandler at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:2613:16 60: libr::r::R_withCallingErrorHandler at ./posit/ark/crates/libr/src/functions.rs:31:21 61: harp::exec::try_catch::{{closure}} at ./posit/ark/crates/harp/src/exec.rs:272:9 62: harp::exec::top_level_exec::callback at ./posit/ark/crates/harp/src/exec.rs:342:31 63: R_ToplevelExec at /Volumes/Builds/R4/R-4.3.2/src/main/context.c:799:2 64: libr::r::R_ToplevelExec at ./posit/ark/crates/libr/src/functions.rs:31:21 65: harp::exec::top_level_exec at ./posit/ark/crates/harp/src/exec.rs:345:14 66: harp::exec::try_catch at ./posit/ark/crates/harp/src/exec.rs:271:20 67: harp::exec::r_sandbox at ./posit/ark/crates/harp/src/exec.rs:388:5 68: ark::interface::RMain::handle_task::{{closure}} at ./posit/ark/crates/ark/src/interface.rs:1393:63 69: tracing::span::Span::in_scope at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/span.rs:1102:9 70: ark::interface::RMain::handle_task at ./posit/ark/crates/ark/src/interface.rs:1393:30 71: ark::interface::RMain::handle_task_interrupt at ./posit/ark/crates/ark/src/interface.rs:1365:34 72: ark::interface::RMain::read_console at ./posit/ark/crates/ark/src/interface.rs:891:21 73: ark::interface::r_read_console::{{closure}} at ./posit/ark/crates/ark/src/interface.rs:2263:31 74: harp::exec::try_catch::callback at ./posit/ark/crates/harp/src/exec.rs:200:31 75: R_withCallingErrorHandler at /Volumes/Builds/R4/R-4.3.2/src/main/errors.c:2613:16 76: libr::r::R_withCallingErrorHandler at ./posit/ark/crates/libr/src/functions.rs:31:21 77: harp::exec::try_catch::{{closure}} at ./posit/ark/crates/harp/src/exec.rs:272:9 78: harp::exec::top_level_exec::callback at ./posit/ark/crates/harp/src/exec.rs:342:31 79: R_ToplevelExec at /Volumes/Builds/R4/R-4.3.2/src/main/context.c:799:2 80: libr::r::R_ToplevelExec at ./posit/ark/crates/libr/src/functions.rs:31:21 81: harp::exec::top_level_exec at ./posit/ark/crates/harp/src/exec.rs:345:14 82: harp::exec::try_catch at ./posit/ark/crates/harp/src/exec.rs:271:20 83: harp::exec::r_sandbox at ./posit/ark/crates/harp/src/exec.rs:388:5 84: r_read_console at ./posit/ark/crates/ark/src/interface.rs:2263:18 85: Rf_ReplIteration at /Volumes/Builds/R4/R-4.3.2/src/main/main.c:210:10 86: R_ReplConsole at /Volumes/Builds/R4/R-4.3.2/src/main/main.c:314:11 87: run_Rmainloop at /Volumes/Builds/R4/R-4.3.2/src/main/main.c:1200:5 88: libr::r::run_Rmainloop at ./posit/ark/crates/libr/src/functions.rs:31:21 89: ark::sys::unix::interface::run_r at ./posit/ark/crates/ark/src/sys/unix/interface.rs:96:9 90: ark::interface::RMain::start at ./posit/ark/crates/ark/src/interface.rs:517:9 91: ark::start::start_kernel at ./posit/ark/crates/ark/src/start.rs:128:5 92: ark::main at ./posit/ark/crates/ark/src/main.rs:370:5 93: core::ops::function::FnOnce::call_once at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 94: std::sys::backtrace::__rust_begin_short_backtrace at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:152:18 95: std::rt::lang_start::{{closure}} at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/rt.rs:199:18 96: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/ops/function.rs:284:13 97: std::panicking::try::do_call at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:589:40 98: std::panicking::try at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:552:19 99: std::panic::catch_unwind at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panic.rs:359:14 100: std::rt::lang_start_internal::{{closure}} at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/rt.rs:168:24 101: std::panicking::try::do_call at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:589:40 102: std::panicking::try at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:552:19 103: std::panic::catch_unwind at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panic.rs:359:14 104: std::rt::lang_start_internal at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/rt.rs:164:5 105: std::rt::lang_start at ./.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/rt.rs:198:5 106: _main
(request: {"jsonrpc":"2.0","method":"call_method","params":{"method":"get_help_page","params":["glyphInfo","grDevices"]}})
There was a problem hiding this comment.
That's a good question; I think allowing this exception to propagate is okay here; if we catch the error, the model doesn't show the error to the user and moves along without explaining why.
In practice this pandoc error won't happen in release builds, since they have a pandoc executable.
There was a problem hiding this comment.
I have bumped my shins a few times against the lack of pandoc in dev builds; filed that here: #8449
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds 4 new LLM tools that can be used to read an R package's help and vignettes.
Requires posit-dev/ark#868 on the Ark side.
Addresses #8016 (for R)
Release Notes
New Features
Bug Fixes
QA Notes
The LLM has been instructed to use help when it needs to, but a fun thing about LLMs is that they mostly don't know what they don't know. Asking for help with a package that is newer than the LLM's knowledge cutoff date is a good way to get it to fall back to tool calls to look up help; if you ask it about common usage for long-standing base R functions, it is not likely to need help pages. You can also just tell it to look up a help page or vignette and it'll do the right thing.