Side effect from region override to credentials[["region"]]

[yilun11]

This is related to #340 since the fix is not ideal and has side effects. I'd like opinions on best way to resolve. I have AWS_S3_ENDPOINT environment variable set and region set in my .aws/credentials file. Even if region isn't set, aws.signature returns us-east-1 by default.

The fix is line 62 from s3HTTP.R:

# allow region="" to override any config - the only way to use 3rd party URLs without region
region <- if (length(region) && !nzchar(region)) region else credentials[["region"]]

Before the fix, region is NULL but after, it is set to credentials[["region"]]. Later, in setup_s3_url(), this change prepends region to my AWS_S3_ENDPOINT, which is not what I want and is a fatal error. See line 282-292:

# handle S3-compatible storage URLs
if (base_url != "s3.amazonaws.com") {
    if (isTRUE(verbose)) {
        message("Non-AWS base URL requested.")
    }
    accelerate <- FALSE
    dualstack <- FALSE
    if (!is.null(region) && region != "") {
         base_url <- paste0(region, ".", base_url)
    }
}

What are recommendations to resolve? Only solution I have so far is to fork code and take out line 62.

[yilun11]

I just saw #371 and #363, which seem to be related, and the suggestion to try arrow. This issue can be closed unless its helpful for others.