Service Fabric On Premise with AD Integration 401 Unauthorized

[pregress]

Do you want to request a feature or report a bug?

Bug

What did you do?

• We run an on-premise Service Fabric cluster, with Active Directory Integration (Not Azure AD)
• Our FabricSvc service is running under an Active Directory account, which has admin rights on the cluster and on the nodes
• The Traefik.exe is running under the same Active Directory Account
• When we start Traefik.exe in the cluster / or from the command line with that account we get the following errors in the log file: Provider connection error: Service Fabric responded with error code 401 Unauthorized to request http://localhost:19080/Applications/?api-version=3.0 with body {}; retrying in 588.391017ms
• Is it possible the windows account is not send with the request to the Service Fabric api ?
• We are able to access http://localhost:19080/Applications/?api-version=3.0 with the Account that is running the service.

What did you expect to see?

• When we browse to traefik dashboard we see the following: No providers found.

What did you see instead?

• We would like to see our Service Fabric services, like we see on our development environments.

Output of traefik version: (What version of Traefik are you using?)

Traefik version v1.7.7 built on 2019-01-08_10:21:03AM

What is your environment & configuration (arguments, toml, provider, platform, ...)?

################################################################
# Global configuration
################################################################

# Enable debug mode
#
# Optional
# Default: false
#
debug = true

# Traefik logs file
# If not defined, logs to stdout
#
# Optional
#
traefikLogsFile = "E:/ServiceFabric/traefik/traefik.log"

# Log level
#
# Optional
# Default: "ERROR"

logLevel = "DEBUG"

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
#
# Optional
# Default: ["http"]
#
defaultEntryPoints = ["http", "https"]

# Entrypoints definition
#
# Optional
# Default:
[entryPoints]
[entryPoints.http]
address = ":21500"
[entryPoints.traefik]
address = ":21550"

# Enable access logs
# By default it will write to stdout and produce logs in the textual
# Common Log Format (CLF), extended with additional fields.
#
# Optional
#
# [accessLog]

# Sets the file path for the access log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
#
# Optional
# Default: os.Stdout
#
# filePath = "/path/to/log/log.txt"

# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format = "common"

################################################################
# API definition
################################################################

[api]
  # Name of the related entry point
  #
  # Optional
  # Default: "traefik"
  #
  entryPoint = "traefik"

  # Enabled Dashboard
  #
  # Optional
  # Default: true
  #
  dashboard = true

  # Enable debug mode.
  # This will install HTTP handlers to expose Go expvars under /debug/vars and
  # pprof profiling data under /debug/pprof.
  # Additionally, the log level will be set to DEBUG.
  #
  # Optional
  # Default: false
  #
  debug = true

################################################################
# Service Fabric provider
################################################################

# Enable Service Fabric configuration backend
[servicefabric]

# Service Fabric Management Endpoint
clustermanagementurl = "http://localhost:19080"
# Note: use "https://localhost:19080" if you're using a secure cluster

# Service Fabric Management Endpoint API Version
apiversion = "3.0"

# Enable TLS connection.
#
# Optional
#
#[serviceFabric.tls]
#  cert = "certs/servicefabric.crt"
#  key = "certs/servicefabric.key"
#  insecureskipverify = true

Part of our cluster config.

"security": {
		"ClusterCredentialType": "Windows",
		"ServerCredentialType": "Windows",
		"WindowsIdentities": {
			"ClientIdentities": [
				{
					"Identity": "DOMAIN\\SA_XXXXXXXXXXXX",
					"IsAdmin": true
				},
				....
			]
		}
	},

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)

time="2019-01-28T08:17:41+01:00" level=info msg="Using TOML configuration file D:\\ServiceFabric\\Data\\A250DOMZ01S0\\Fabric\\work\\Applications\\TraefikType_App12\\TraefikPkg.Code.1.7.7\\traefik.toml"
time="2019-01-28T08:17:41+01:00" level=info msg="Traefik version v1.7.7 built on 2019-01-08_10:21:03AM"
time="2019-01-28T08:17:41+01:00" level=debug msg="Global configuration loaded {\"LifeCycle\":{\"RequestAcceptGraceTimeout\":0,\"GraceTimeOut\":10000000000},\"GraceTimeOut\":0,\"Debug\":true,\"CheckNewVersion\":true,\"SendAnonymousUsage\":false,\"AccessLogsFile\":\"\",\"AccessLog\":null,\"TraefikLogsFile\":\"E:/ServiceFabric/traefik/traefik.log\",\"TraefikLog\":null,\"Tracing\":null,\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Address\":\":21500\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"traefik\":{\"Address\":\":21550\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}}},\"Cluster\":null,\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\",\"https\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"IdleTimeout\":0,\"InsecureSkipVerify\":false,\"RootCAs\":null,\"Retry\":null,\"HealthCheck\":{\"Interval\":30000000000},\"RespondingTimeouts\":null,\"ForwardingTimeouts\":null,\"AllowMinWeightZero\":false,\"KeepTrailingSlash\":false,\"Web\":null,\"Docker\":null,\"File\":null,\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":null,\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null,\"DynamoDB\":null,\"ServiceFabric\":{\"Watch\":false,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"ClusterManagementURL\":\"http://localhost:19080\",\"APIVersion\":\"3.0\",\"RefreshSeconds\":0,\"TLS\":null,\"AppInsightsClientName\":\"\",\"AppInsightsKey\":\"\",\"AppInsightsBatchSize\":0,\"AppInsightsInterval\":0},\"Rest\":null,\"API\":{\"EntryPoint\":\"traefik\",\"Dashboard\":true,\"Debug\":true,\"CurrentConfigurations\":null,\"Statistics\":null},\"Metrics\":null,\"Ping\":null,\"HostResolver\":null}"
time="2019-01-28T08:17:41+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2019-01-28T08:17:41+01:00" level=warning msg="clientTLS is nil"
time="2019-01-28T08:17:41+01:00" level=info msg="Preparing server traefik &{Address::21550 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0004f01e0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2019-01-28T08:17:41+01:00" level=info msg="Preparing server http &{Address::21500 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0004f01a0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2019-01-28T08:17:41+01:00" level=info msg="Starting server on :21550"
time="2019-01-28T08:17:41+01:00" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2019-01-28T08:17:41+01:00" level=info msg="Starting server on :21500"
time="2019-01-28T08:17:41+01:00" level=info msg="Starting provider *servicefabric.Provider {\"Watch\":false,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"ClusterManagementURL\":\"http://localhost:19080\",\"APIVersion\":\"3.0\",\"RefreshSeconds\":10000000000,\"TLS\":null,\"AppInsightsClientName\":\"\",\"AppInsightsKey\":\"\",\"AppInsightsBatchSize\":0,\"AppInsightsInterval\":0}"
time="2019-01-28T08:17:51+01:00" level=info msg="Checking service fabric config"
time="2019-01-28T08:17:51+01:00" level=error msg="Provider connection error: Service Fabric responded with error code 401 Unauthorized to request http://localhost:19080/Applications/?api-version=3.0 with body {}; retrying in 588.391017ms"
time="2019-01-28T08:18:01+01:00" level=info msg="Checking service fabric config"
time="2019-01-28T08:18:01+01:00" level=error msg="Provider connection error: Service Fabric responded with error code 401 Unauthorized to request http://localhost:19080/Applications/?api-version=3.0 with body {}; retrying in 507.777275ms"
time="2019-01-28T08:18:12+01:00" level=info msg="Checking service fabric config"
.... (Trimmed, because this goes on)

[jjcollinge]

Please refer to this issue for more details.
In summary it's a known limitation of the integration, it only supports certificate authentication.

[pregress]

I Solved it by deploy a Proxy service for the Service fabric service, that handles the authentication for us.

private const int StreamCopyBufferSize = 81920;

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            // send every request to Service fabric api
            app.MapWhen(ctx => true, appBuilder =>
            {
                appBuilder.Run(async context =>
                {
                    var requestMessage = context.Request.ToHttpRequestMessage("localhost",19080);

                    try
                    {
                        var response = await Client.SendAsync(requestMessage);

                        using (var responseStream = await response.Content.ReadAsStreamAsync())
                        {
                            await responseStream.CopyToAsync(context.Response.Body, StreamCopyBufferSize, context.RequestAborted);
                        }
                    }
                    catch (Exception e)
                    {
                        await context.Response.WriteAsync(e.ToString());
                        context.Response.StatusCode = 500;
                    }
                });
            });
        }

        private static readonly HttpClient Client = new HttpClient(
            new HttpClientHandler
            {
                UseDefaultCredentials = true,
            })
            {
                BaseAddress = new Uri("http://localhost:19080")
            };

And I changed the clustermanagementurl to the new Proxy Service Url

[tastyeggs]

To build on @pregress's answer, this is now even simpler with the Microsoft.AspNetCore.Proxy package (assuming you're using ASP.NET Core):

<PackageReference Include="Microsoft.AspNetCore.Proxy" Version="0.2" />

public class Startup
{
        public void ConfigureServices(IServiceCollection services)
        {

        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            app.RunProxy(new ProxyOptions
            {
                Host = "localhost",
                Port = "19080",
                Scheme = "http",
                BackChannelMessageHandler = new HttpClientHandler() { UseDefaultCredentials = true }
            });
        }
    }