Thredds do not behave well behind Nginx proxy serving as SSL termination

[tlvu]

This is our Nginx proxy config. We are using Nginx to provide SSL termination.

    location /thredds/ {
        proxy_pass http://thredds:8080/thredds/;
        proxy_set_header Host $host;  # pass the original public hostname to Thredds
        proxy_set_header X-Forwarded-Proto $scheme;  # pass the original httpS proto to Thredds
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  # pass the original client IP to Thredds
    }

We are using docker image unidata/thredds-docker:4.6.14.

What we found out is Thredds is ignoring the X-Forwarded-Proto http header and the X-Forwarded-For http header as well since in the logs, we only see IP of our Nginx proxy.

Below we see the IP 172.21.0.1 which is the Nginx proxy, not the real client IP requesting data from Thredds. It's the same IP everywhere.

$ docker exec -it thredds grep 'Remote host' /usr/local/tomcat/content/thredds/logs/threddsServlet.log | tail -1
2019-09-19T20:49:01.911 +0000 [   4657961][     144] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1"

The original public hostname is preserved by Thredds so the Host http header seems to be honored properly.

To work-around the X-Forwarded-Proto http header not being honored, we added scheme="https" to the Connector in server.xml:

    <Connector scheme="https" server="Apache" secure="true" port="80
80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

That server.xml file is from the docker image so we had to insert an entrypoint to modify that file right before the regular startup.

We still have no work-around for the wrong client IP being logged because X-Forwarded-For http header is being ignored.

Also, using Nginx as a reverse proxy should be documented here https://www.unidata.ucar.edu/software/tds/current/reference/TomcatBehindProxyServer.html

[lesserwhirls]

Thank you for passing along your nginx config - I think we're getting close to being able to document this setup, but not quite yet.

Here is my understanding of the overall situation:

When obtaining information about incoming requests (ip addresses, for example), the TDS uses the Java Servlet API, so it only "sees" what the servlet container passes it. This gives server administrators the ability to choose a servlet container of their choice (e.g. Jetty, GlassFish, WebSphere, JBoss, etc.), as long as it supports the proper version of the servlet specification (3.0 in the case of the TDS). The TDS Docker image uses Tomcat as the servlet container.

When you add a proxy, you have to make sure that the proxy server and the servlet container are configured properly so that the correct information makes it from the proxy through the servlet container to the application (i.e. the TDS), and that they servlet container can get the response information from the application back to the proxy. Because there are a number of combinations of proxy server / servlet container combinations, there isn't one single configuration that can be used. We've documented the one we know and use.

The solution you found for getting https to work adds a Connector to Tomcat to properly interface with nginx to make sure the protocol makes it to the servlet container (this approach was mentioned here. We modify the Tomcat server.xml config in a similar way to get the apache proxy working, too (see here). To ease the process, perhaps the TDS Docker image should allow one to supply their own Tomcat configuration (although I think it sort of does, as the docs hint at it: "Tomcat configuration can be done by mounting over the appropriate directories in CATALINA_HOME (/usr/local/tomcat).")

In terms of logging, the IP address logged by the TDS is the one passed in by Tomcat via the servlet API. The question is, how do you configure Tomcat to pass along the "correct" IP when nginx is the proxy? We use mod_jk, which provides a smooth interaction between the Apache proxy and the servlet container, and it does not require us to consider that question, as it's handled under-the-hood, so to say. With other proxies (like nginx), a more careful consideration of the configuration is needed to make sure the proxy and Tomcat can fully "talk".

It looks like everything could be done by adding a special <Valve> configuration to Tomcat's server.xml (no need for the <Connector>), but I'm not certain quite yet. I do not manage the proxy / servlet setup on our machines, and locally I only ever run Tomcat, but at this point I think I have enough information to get started and give things a try on my local machine. I'll update this ticket as things progress.

[tlvu]

I found out how to get Thredds to handle the X-Forwarded-Proto and X-Forwarded-For http headers from Nginx properly. It's just one more line to add in the server.xml file !

In the unidata/thredds-docker:4.6.14 docker image, in file /usr/local/tomcat/conf/server.xml, there is only one line <Engine name="Catalina" defaultHost="localhost">, just add this line right below <Valve className="org.apache.catalina.valves.RemoteIpValve" protocolHeader="X-Forwarded-Proto" />. Doc for that Valve https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/catalina/valves/RemoteIpValve.html

I now have the original httpS proto in the URL displayed by Thredds and the original real client IP in my logs.

Below are the logs from 1 refresh in the browser:

$ docker exec thredds tail -f /usr/local/tomcat/logs/localhost_access_log.`date '+%Y-%m-%d'`.txt /usr/local/tomcat/content/thredds/logs/threddsServlet.log

==> /usr/local/tomcat/content/thredds/logs/threddsServlet.log <==
2019-09-20T17:22:07.990 +0000 [     15358][       2] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1"
2019-09-20T17:22:07.999 +0000 [     15367][       1] INFO  - threddsServlet - Remote host: 172.16.0.60 - Request: "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/catalog.html HTTP/1.0"
2019-09-20T17:22:08.109 +0000 [     15477][       2] INFO  - threddsServlet - Request Completed - 200 - 1603 - 126
2019-09-20T17:22:08.122 +0000 [     15490][       1] INFO  - threddsServlet - Request Completed - 200 - 3070 - 138
2019-09-20T17:22:08.213 +0000 [     15581][       3] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/ HTTP/1.1"
2019-09-20T17:22:08.223 +0000 [     15591][       3] INFO  - threddsServlet - Request Completed - 302 - -1 - 10
2019-09-20T17:22:08.231 +0000 [     15599][       4] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1"
2019-09-20T17:22:08.235 +0000 [     15603][       4] INFO  - threddsServlet - Request Completed - 200 - 1603 - 4
2019-09-20T17:22:08.269 +0000 [     15637][       5] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/threddsIcon.gif HTTP/1.1"
2019-09-20T17:22:08.284 +0000 [     15652][       6] INFO  - threddsServlet - Remote host: 172.21.0.1 - Request: "GET /twitcher/ows/proxy/thredds/folder.gif HTTP/1.1"
2019-09-20T17:22:08.296 +0000 [     15664][       5] INFO  - threddsServlet - Request Completed - 304 - -1 - 27
2019-09-20T17:22:08.297 +0000 [     15665][       6] INFO  - threddsServlet - Request Completed - 304 - -1 - 13

==> /usr/local/tomcat/logs/localhost_access_log.2019-09-20.txt <==
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1" 200 1603
172.21.0.22 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog/birdhouse/catalog.html HTTP/1.0" 200 3070
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds HTTP/1.1" 302 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/ HTTP/1.1" 302 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/catalog.html HTTP/1.1" 200 1603
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/threddsIcon.gif HTTP/1.1" 304 -
172.21.0.1 - - [20/Sep/2019:17:22:08 +0000] "GET /twitcher/ows/proxy/thredds/folder.gif HTTP/1.1" 304 -

The IP 172.16.0.60 is the original real client IP. 172.21.0.1 is the network gateway from docker-compose and 172.21.0.22 is the IP of the Nginx proxy container.

Not sure why Thredds/Tomcat recorded to be hit by the network gateway so much. I would have expected to see the original real client IP or the Nginx proxy IP at all the places we see the network gateway IP.

Also it's funny that Thredds logs have the good client IP while the Tomcat logs have the Nginx proxy IP.

Will submit a PR for the extra needed Valve line in server.xml for the docker image unidata/thredds-docker. This way I can just use the stock docker image without having to customize the server.xml file.

[lesserwhirls]

Excellent!

Have you ran across this post at https://serverfault.com/questions/514551/make-tomcat-use-x-real-ip?

It sounds like you could use

<Valve className="org.apache.catalina.valves.RemoteIpValve"
               protocolHeader="X-Forwarded-Proto"
               remoteIpHeader="X-Forwarded-For"
               requestAttributesEnabled="true"
               internalProxies="172\.21\.0\.1"  />

and that would take care of everything you needed, including the IP address in the tomcat logs.

[tlvu]

Have you ran across this post at https://serverfault.com/questions/514551/make-tomcat-use-x-real-ip?

It sounds like you could use

<Valve className="org.apache.catalina.valves.RemoteIpValve"
               protocolHeader="X-Forwarded-Proto"
               remoteIpHeader="X-Forwarded-For"
               requestAttributesEnabled="true"
               internalProxies="172\.21\.0\.1"  />

and that would take care of everything you needed, including the IP address in the tomcat logs.

No did not saw that post, thanks for the reference. That post seems to be applicable to older version of Tomcat.

requestAttributesEnabled do not exist anymore on that RemoteIpValve for Tomcat 8 and later. The default value for remoteIpHeader is already the wanted one so no need to set.

internalProxies probably a good idea but in the context of a full docker stack, the proxy is also running in a container so its IP is not known in advance which make it harder to set in the config file, but still possible.

The fact that the real client IP is not in the Tomcat logs is not too much annoying. The Thredds logs have all the important bits (date time, real client IP, http verb, url path, return code, content lenght).

The proposed minimally customized RemoteIpValve works well enough.

Also, from what I understand, X-Forwarded-For is the superset of X-Real-IP. X-Real-IP contains only 1 IP. X-Forwarded-For is actually a list of IPs so if the customer goes through 3 proxies before reaching Thredds, the X-Forwarded-For header received by Thredds would contain original client IP, proxy1 IP, proxy2 IP.

My PR has been declined by the way. Don't know if it's possible to appeal the decision. I've re-explained in the PR why I think the PR is generic and will contribute to the out of the box user experience with Thredds.