Accessing files in a requester pays bucket? #194
Description
I'm using googleCloudStorageR on the All Of Us Researcher Workbench via a Jupyter notebook with an R kernel. I can work with files in a bucket that I own without issue, but I'm running into trouble when I try to list files in a requester pays bucket. I'm using R 4.5.0 and googleCloudStorageR 0.7.0.
I first authenticate like this:
## Fetch token. See: https://developers.google.com/identity/protocols/oauth2/scopes
scope <-c("https://www.googleapis.com/auth/cloud-platform")
token <- token_fetch(scopes = scope)
## Pass your token to gcs_auth
gcs_auth(token = token)
When I try to list files:
gcs_list_objects(bucket="gs://fc-aou-datasets-controlled")
I get the following error:
Error in `abort_http()`:
! http_400 Bucket is a requester pays bucket but no user project provided.
Traceback:
1. gar_api_page(lo, page_f = function(x) attr(x, "nextPageToken"),
. page_method = "param", page_arg = "pageToken")
2. f()
3. doHttrRequest(req_url, request_type = http_header, the_body = the_body,
. customConfig = customConfig, simplifyVector = simplifyVector)
4. retryRequest(do.call("RETRY", args = arg_list, envir = asNamespace("httr")))
5. abort_http(status_code, error)
6. rlang::abort(paste0("http_", status_code), message = paste0("http_",
. status_code, " ", msg))
7. signal_abort(cnd, .file)
8. signalCondition(cnd)
Normally, with gsutil, I can specify the google project that the gsutil ls command should use for a requester pays bucket, eg:
gsutil -u $GOOGLE_PROJECT ls gs://fc-aou-datasets-controlled
Is there a way to have the gcs_list_objects function use my GOOGLE_PROJECT environment variable so I can list files in a requester pays bucket?
Full session info:
R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.5 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: Etc/UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] GGally_2.3.0 glue_1.8.0
[3] lubridate_1.9.4 forcats_1.0.0
[5] stringr_1.5.1 dplyr_1.1.4
[7] purrr_1.0.4 readr_2.1.5
[9] tidyr_1.3.1 tibble_3.2.1
[11] ggplot2_3.5.2 tidyverse_2.0.0
[13] gargle_1.5.2 googleCloudStorageR_0.7.0
[15] IRdisplay_1.1
loaded via a namespace (and not attached):
[1] generics_0.1.3 stringi_1.8.7 hms_1.1.3 digest_0.6.37
[5] magrittr_2.0.3 RColorBrewer_1.1-3 evaluate_1.0.3 grid_4.5.0
[9] timechange_0.3.0 pbdZMQ_0.3-14 fastmap_1.2.0 jsonlite_2.0.0
[13] zip_2.3.2 httr_1.4.7 scales_1.3.0 textshaping_1.0.0
[17] cli_3.6.4 rlang_1.1.6 crayon_1.5.3 googleAuthR_2.0.2
[21] bit64_4.6.0-1 munsell_0.5.1 base64enc_0.1-3 withr_3.0.2
[25] repr_1.1.7 cachem_1.1.0 yaml_2.3.10 parallel_4.5.0
[29] tools_4.5.0 tzdb_0.5.0 uuid_1.2-1 memoise_2.0.1
[33] colorspace_2.1-1 ggstats_0.10.0 curl_6.2.2 assertthat_0.2.1
[37] vctrs_0.6.5 R6_2.6.1 lifecycle_1.0.4 bit_4.6.0
[41] fs_1.6.6 vroom_1.6.5 ragg_1.4.0 pkgconfig_2.0.3
[45] pillar_1.10.2 gtable_0.3.6 systemfonts_1.2.2 tidyselect_1.2.1
[49] IRkernel_1.3.2 farver_2.1.2 htmltools_0.5.8.1 labeling_0.4.3
[53] compiler_4.5.0 S7_0.2.0 askpass_1.2.1 openssl_2.3.2