org.eclipse.ecf.provider.filetransfer.httpclientjava does not support proxy authentication

[merks]

See the following p2 issue for background details:

eclipse-equinox/p2#381

It's not clear (to me) if the proxy needs to be configured by java.net.http.HttpClient.Builder.proxy(ProxySelector) as suggested here:

https://openjdk.org/groups/net/httpclient/intro.html

Note that with org.eclipse.ecf.internal.provider.filetransfer.httpclientjava.Activator.USE_SHARED_CLIENT_DEFAULT being true, I think there is only a singe HttpClient that is shared.

These two fields

ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/ECFHttpClientFactory.java

Lines 35 to 38 in bbf5ffa

	@SuppressWarnings("unused")
	private static final List<String> DEFAULT_PREFERRED_AUTH_SCHEMES_NO_NTLM = Arrays.asList("Basic","Digest");
	@SuppressWarnings("unused")
	private static final List<String> DEFAULT_PREFERRED_AUTH_SCHEMES = Arrays.asList("Basic", "Digest", "NTLM");

which are copied from here:

ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/internal/provider/filetransfer/httpclient5/ECFHttpClientFactory.java

Lines 62 to 63 in bbf5ffa

	private static final List<String> DEFAULT_PREFERRED_AUTH_SCHEMES_NO_NTLM = Arrays.asList("Basic","Digest");
	private static final List<String> DEFAULT_PREFERRED_AUTH_SCHEMES = Arrays.asList("Basic", "Digest", "NTLM");

are not actually used. It's not clear what, if anything, needs to be configured in this regard.

The following implementation detail is clearly incomplete:

ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/HttpClientProxyCredentialProvider.java

Line 77 in bbf5ffa

Credentials credential = getCredentials(null); // TODO

Perhaps it would be correctly implemented like this:

Credentials credentials = getCredentials(new AuthScope(getRequestingHost(), getRequestingPort()));

It's possible that if this were fixed, some additional proxy configurations would "just work"...

[scottslewis]

The following implementation detail is clearly incomplete:

ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/HttpClientProxyCredentialProvider.java

Line 77 in bbf5ffa

Credentials credential = getCredentials(null); // TODO

Perhaps it would be correctly implemented like this:

Credentials credentials = getCredentials(new AuthScope(getRequestingHost(), getRequestingPort()));

It's possible that if this were fixed, some additional proxy configurations would "just work"...

I agree with @merks that this might fix some (or even all) of the observed proxy creds regressions. I have no way of testing whether it will help or not however, so we should line up some testing assistance and/or environment before making this fix.

[SebasHein]

I am one of the users behind an authentification proxy.
I know you would prefere some kind of unit test work environment but thats apparently hard to set up.

If you could provide me with a build of the eclipse sdk that contains the fix @merks suggests, I could do a test and see wether it works again - if that is of any help of course.

[merks]

Yes, my sense is that the fix can do no harm. In fact the called getCredentials method looks to me highly likely to produce NPEs when passed null as it current does so I feel we should change this regardless of there not being good automated tests...

@SebasHein

Locally for testing this technology stack, I use the installer with these projects for the workspace:

That lets me change and test everything (ECF, p2, and Oomph) locally.

If you wish to help with testing such a change before there is an ECF build, let's discuss that further in this Oomph issue that I just opened:

eclipse-oomph/oomph#46

Thanks again for reporting the issue...

[merks]

@SebasHein

I produced this update site that you could use for testing the potential fix:

https://ci.eclipse.org/emf/job/test-ecf/lastSuccessfulBuild/artifact/releng/org.eclipse.ecf.releng.repository/target/repository/

[sratz]

I have looked into this in more detail and have come to the conclusion that the implementation is incomplete regarding proxy authentication.

The whole code base of java 11 client was copied from apache http client 5 and adapted.

However, apache 5 worked significantly different:

• Base HTTP client with a default Proxy provider and Authentication handler:

  ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/internal/provider/filetransfer/httpclient5/ECFHttpClientFactory.java

  Line 90 in 3485821

  builder.setDefaultCredentialsProvider(new HttpClientProxyCredentialProvider());

• Re-use of the HTTP client instance, but request-specific proxy set inside the context of the request:

  ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java

  Line 362 in 3485821

  requestConfigBuilder.setProxy(new HttpHost(address.getHostName(), address.getPort()));

• and a request-specific credential provider in the context of the request:
  

  ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java

  Line 140 in 3485821

  httpContext.setCredentialsProvider(contextCredentialsProvider);

  ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient5/src/org/eclipse/ecf/provider/filetransfer/httpclient5/HttpClientFileSystemBrowser.java

  Line 224 in 3485821

  requestConfigBuilder = Activator.getDefault().getHttpClientFactory().newRequestConfig(httpContext, null);

The Java 11 HTTP client however, does not allow request-specific credential providers or proxy selectors. It only allows a global one set on the HttpClient instance during its construction.

The current ECF wrapper for Java 11 tries to imitate such a 'request context' object by introducing a custom interface
https://github.com/eclipse/ecf/blob/3485821a7c5c8948d8a82caca5a454a04c45607d/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/IHttpClientContext.java

and the proxy and credential information goes to the implementation of that interface:

ecf/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclientjava/src/org/eclipse/ecf/internal/provider/filetransfer/httpclientjava/ECFHttpClientFactory.java

Lines 64 to 91 in 3485821

	IHttpClientContext context = new IHttpClientContext() {
	private Map<String, Object> values = new HashMap<>();
	@SuppressWarnings("unused")
	private Authenticator authenticator;
	@SuppressWarnings("unused")
	private HttpHost httpHost;
	@Override
	public void setAttribute(String key, Object value) {
	values.put(key, value);
	}
	@Override
	public void setCredentialsProvider(Authenticator contextCredentialsProvider) {
	this.authenticator = contextCredentialsProvider;
	}
	@Override
	public Object getAttribute(String key) {
	return values.get(key);
	}
	@Override
	public void setProxy(HttpHost httpHost) {
	this.httpHost = httpHost;
	}
	};

The problem is - as can be seen by the @SuppressWarnings("unused") - this information is simply going nowhere, because there is no notion of such a client request context in the Java 11 HTTP client.

I don't think there is an easy way around this.

Two options have come to mind:

1. Somehow hack in this per-request-context by passing along the information in some static Map / via ThreadLocals. The problem is that the Java 11 HTTP client is async by default and we are not in the same thread in HttpClientFileSystemBrowser and in the ProxySelector / Authenticator. So there is no key for a mapping of request -> IHttpClientContext that we could leverage for a lookup.
2. Do not re-use the HTTP client instances at all. Instead, create a fresh one for every ECF call. This probably results in worse performance, as sockets / caches cannot be shared and re-used. It also would change the public API of org.eclipse.ecf.internal.provider.filetransfer.httpclientjava / org.eclipse.ecf.provider.filetransfer.httpclientjava significantly.

[sratz]

Another thing:

The Java HTTP client - by default - disallows Basic Authentication for proxies since Java 8u111:

• https://www.oracle.com/java/technologies/javase/8u111-relnotes.html [JDK-8160838 (not public) (security-libs/java.security)]
• https://github.com/openjdk/jdk17u/blob/df82c1c6490bb5114fe52e4e1a4faa6aa3a56cfd/src/java.base/share/conf/net.properties#L69-L93

So after solving the problem mentioned above, we would - additionally - need the following system properties to be set:

jdk.http.auth.proxying.disabledSchemes = ""
jdk.http.auth.tunneling.disabledSchemes = ""

to get things working. This would need to happen probably via eclipse.ini, as these are initialized statically very early:
https://github.com/openjdk/jdk17u/blob/df82c1c6490bb5114fe52e4e1a4faa6aa3a56cfd/src/java.net.http/share/classes/jdk/internal/net/http/common/Utils.java#L197-L216
so a System.setProperty() would probably be too late.

[laeubi]

@sratz Prox Auth is done on the http level, so one always could set these headers manually (we do that at Tycho already)...

[merks]

@scottslewis

Note that final platform build is on Wednesday so it will be good to have a final repository URL for that:

eclipse-platform/eclipse.platform.releng.aggregator#1390

Also for contribution to SimRel. If you let me know the URL I will do that for ECF.

Thanks.

[scottslewis]

@merks

Here is release build repo for ECF 3.14.40:

https://download.eclipse.org/rt/ecf/3.14.40/site.p2/3.14.40.v20231114-1017/

[scottslewis]

@merks @laeubi

As @laeubi appears disinclined to implement this (he is the implementer of the httpclientjava provider). I'm going to close this issue. If it should be reopened then please do so.

[jonahgraham]

Can you consider reopening this for planning because we have started a funded dev effort initiative to cover this issue.

[laeubi]

This is somehow related:

• A "org.eclipse.equinox.p2.transport.javahttpclient" transport using directly Java HTTP client or OSGi services eclipse-equinox/p2#526

[jonahgraham]

There seems to be an open question as to which approach for handling http going forward should happen. The ECF and Equinox/p2 project teams need to decide on the correct course of action so that the planning council can recommend funding this work to the IDE WG. (See this comment in gitlab)

[merks]

This is a related problem that I too encounter only with the newer provider:

eclipse-equinox/p2#529