v0.10.0 - http_proxy behaviour changed - rclone via proxy might be affected

[kapitainsky]

I have been using rclone backend successfully with v0.9.5. Suddenly with v0.10.0 it stopped working and I really struggle where to look for clues why. I only narrowed this down to something related to rclone using proxy.

As I use rclone with local proxy my toml contains (which is the way to tell rclone to use proxy):

[global.env]
HTTPS_PROXY = "socks5://127.0.0.1:1080"
HTTP_PROXY = "socks5://127.0.0.1:1080"

It used to work perfectly but with the latest rustic when I try:

$ rustic snapshots --use-profile rustic_socks5
[INFO] using config ./rustic_socks5.toml
[WARN] Error error sending request for url (http://127.0.0.1:36769/config) at 1s, retrying
[WARN] Error error sending request for url (http://127.0.0.1:36769/config) at 2s, retrying
[WARN] Error error sending request for url (http://127.0.0.1:36769/config) at 4s, retrying
[WARN] Error error sending request for url (http://127.0.0.1:36769/config) at 8s, retrying
[WARN] Error error sending request for url (http://127.0.0.1:36769/config) at 16s, retrying
error: `rustic_core` experienced an error related to `the backend`.

Message:
Backoff failed, please check the logs for more information.


Some additional details ...

Caused by:
error sending request for url (http://127.0.0.1:36769/config)
 : (source: client error (Connect))

Backtrace:
disabled backtrace (set 'RUST_BACKTRACE="1"' environment variable to enable)

removing proxy env variables from toml file solves the problem. But for various reasons I have to use proxy.

I have read all v0.10.0 release notes and I am puzzled what change caused it. It looks like suddenly rustic is picking up these proxy env variables for something else but failing to use them.

Short term I can roll back rustic to v0.9.5 so it is definitely not critical but at some stage I would like to sort it out.

I would appreciate any expert help here... as I run out of ideas.

I have not created any github issue yet as I am not sure (yet) whether problem is with rustic or between chair and keyboard.

[kapitainsky]

In the meantime I have found how to fix it for my particular use case. Might be useful for others with similar problem.

To make only rclone to use proxy I have changed my toml file proxy/rclone related settings to to:

[repository.options]
rclone-command = '/usr/local/bin/rclone serve restic --addr localhost:0 --http-proxy "socks5://127.0.0.1:1080"' 

[global.env]
# HTTPS_PROXY = "socks5://127.0.0.1:1080"
# HTTP_PROXY = "socks5://127.0.0.1:1080"

effectively making only rclone to have knowledge about proxy. Now rustic v0.10.0 works again.

The question remains why does rustic v0.10.0 attempt to utilise global env proxy settings? There is no need for it when using rclone backend. And it was not the case in v0.9.5

[aawsome]

Hi @kapitainsky! Thanks for reporting!

First, about setting env in the config profile, I must say that I could not reproduce any issue here. I tried using a hook to print an env variable and it worked when I set it in the [global.env] section. I suppose that the behavior you are describing is exactly the same as running HTTP_PROXY=.. HTTPS_PROXY=.. rustic ..., but can you please also test this, to be sure?

Second, from the errors it looks like rclone did actually not start correctly. Can you please run with --log-level debug to get more debug output here? Also it might be necessary to set a higher value for the RCLONE_VERBOSE env variable to get more logging output from rclone itself. However, if this is an rclone issues it is still strange as there hasn't been any modification w.r.t. the rclone integration between 0.9.5 and 0.10.0...

[kapitainsky]

yes it is strange. For sure this behaviour is not present in rustic v0.9.5. Only what was changing between my tests was rustic version hence I asked this question here.

But you are right and the root cause might more nuanced and rustic could be only involuntary trigger. Thank you for tips what and how to test. I will try to gather more info.

[kapitainsky]

I have had to do some tests before tonight:)

removing all proxy reference from toml:

[repository.options]
# rclone-command = '/usr/local/bin/rclone serve restic --addr localhost:0 --http-proxy "socks5://127.0.0.1:1080"'                               

[global.env]
# HTTPS_PROXY = "socks5://127.0.0.1:1080"
# HTTP_PROXY = "socks5://127.0.0.1:1080"

and setting proxy env variable inline with rustic command works:

$ HTTPS_PROXY="socks5://127.0.0.1:1080"; HTTP_PROXY="socks5://127.0.0.1:1080"; rustic snapshots --use-profile rustic_socks5
[INFO] using config ./rustic_socks5.toml
[INFO] repository rclone:TEST01-ENC:rustic_kptsky_backup: password is correct.
[INFO] using cache at /media/external_ext4/rustic_cache/ef16129f0f3470822a35de5534465238e31046d5c50f6d7e76ea1c5dbefbac4a
[00:00:00] getting snapshots...           ████████████████████████████████████████         54/54                                                                 
....

but rclone is not using any proxy. As it also works when I turn proxy off:) I do not think rustic is passing its initial env to rclone session.

Anyway - I will keep digging:)

[kapitainsky]

I have set RCLONE_LOG_LEVEL = "DEBUG" but it does not reveal anything. In both cases (working and not) rclone output looks exactly the same. Only difference is rustic v10 output with non working toml (proxy set as env variables) producing:

[WARN] Error error sending request for url (http://127.0.0.1:35849/config) at 1s, retrying

Please note that HTTPS_PROXY and HTTP_PROXY are not just rclone specific variables. Many other programs/modules can use them as it is sort of de facto standard of setting proxies.

So far I am puzzled:)

[aawsome]

And for rustic, use --log-level=debug as CLI option, please.

[kapitainsky]

RCLONE_VERBOSE = "2"

FYI

RCLONE_LOG_LEVEL = "DEBUG" and RCLONE_VERBOSE = "2" are exactly the same options. And produce the same results. Rclone has a bit of redundancy when it comes to its flags:)

[kapitainsky]

And for rustic, use --log-level=debug as CLI option, please.

I think we are getting somewhere here. With rustic v0.10.0 and proxy env variables set:

[DEBUG] (1) rustic_backend::rclone: using REST backend with url TEST01-ENC:rustic_kptsky_backup.
[DEBUG] (4) reqwest::connect: starting new connection: http://127.0.0.1:39271/
[DEBUG] (4) reqwest::connect: proxy(socks5://127.0.0.1:1080/) intercepts 'http://127.0.0.1:39271/'
[WARN] Error error sending request for url (http://127.0.0.1:39271/config) at 1s, retrying

this line I think would explain the problem:

[DEBUG] (4) reqwest::connect: proxy(socks5://127.0.0.1:1080/) intercepts 'http://127.0.0.1:39271/'

of course when connecting to local IP via proxy it fails. But rustic should not do this I think.

This is how v0.9.5 was doing it for the same toml (all working):

[DEBUG] (1) rustic_backend::rclone: using REST backend with url TEST01-ENC:rustic_kptsky_backup.
[DEBUG] (4) reqwest::connect: starting new connection: http://127.0.0.1:39313/
[DEBUG] (4) hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:39313
[DEBUG] (4) hyper_util::client::legacy::connect::http: connected to 127.0.0.1:39313
[DEBUG] (4) hyper_util::client::legacy::pool: pooling idle connection for ("http", 127.0.0.1:39313)
[DEBUG] (4) hyper_util::client::legacy::pool: reuse idle connection for ("http", 127.0.0.1:39313)
[INFO] rclone output: 2025/09/14 22:10:51 DEBUG : keys: list request

[kapitainsky]

And GOOD news.

Whatever (it is not rclone but something new in rustic) is responsible for this issue is also respecting another de facto standard proxy related env variable NO_PROXY

Below toml env variables work with rustic v0.10.0:

[global.env]
HTTPS_PROXY = "socks5://127.0.0.1:1080"
HTTP_PROXY = "socks5://127.0.0.1:1080"
NO_PROXY = "127.0.0.1"

Maybe it was actually rustic v0.9.5 not working properly? I am not sure if proxy (when set) should be used for 127.0.0.1 by default or not.

From my perspective now I know fully how to deal with this in my setup:)

Thank you very much for your help.

[aawsome]

Thanks for analyzing this @kapitainsky
It could be that the behavior of the dependency reqwest changed and that the default proxy env variables are now respected. The reqwest version has been updated with 0.10.0. But this IMO is actually a nice feature we now have!

In any case, I would say that the current behavior (env variables in the config profile are respected by rustic itself and are sent to child processes) is the desired one.

[kapitainsky]

In any case, I would say that the current behavior (env variables in the config profile are respected by rustic itself and are sent to child processes) is the desired one.

Agree. Either way it is now clear how things work and how to configure required behaviour.

[kapitainsky]

FYI. I have done a bit of Google and it seems that there is no strict standard what application should do when HTTP_PROXY is set in regards to traffic to localhost.

Looks like most apps like browsers (Chrome. Firefox) and common cmd tools like curl by default apply localhost exclusion.

But there are quite few (e.g. Cypress) which do not do it.

I think there is no problem with rustic not to do this neither as long as it is documented. I wonder where we could put the line that rustic might need the NO_PROXY variable when traffic to localhost should not be proxied.

It is rather edge case but when needed such info would be a gold for somebody trying to configure it:)