Don't use duplicate hostnames for prod docker containers

[cdrini]

Currently we set the container's hostname to the host machine's hostname. This means multiple containers have the same hostname, which might be causing some of our issue w.r.t to accessing the host machine from a container. Switch to something more unique.

Eg instead of ol-home0 -> ol-home0-solr_updater-1 . Effectively {hostname}-{service name}-{instance}.

Concerns:

• webserver logs use the hostname to write the logs to archive.org I think. Need to make sure that still writes to the same place.

Stakeholders

@cclauss

[cclauss]

Based on yesterday's discussion, I agree that giving each Docker container its own hostname and using DNS based on those hostnames is the best way to simplify our communications.

I would like to suggest that we use {service name}-{instance} instead of {hostname}-{service name}-{instance} to maximize our flexibility to move workloads in longer-term efforts such as #8043. For instance, we currently have both openlibrary-covers-1 and openlibrary-covers-2 running on the same host ol-covers0. This means that our covers service has a single point of failure -- if ol-covers0 fails then Open Library has no covers. If we leave the server name out of the Docker hostname then we free ourselves to move jobs to other hosts.

Therefore, I propose that the Docker hostname exactly match the Docker container name. So on localhost:
% docker ps | rev | cut -w -f1 | rev | sort

openlibrary-covers-2
openlibrary-db-1
openlibrary-infobase-1
openlibrary-memcached-1
openlibrary-solr-1
openlibrary-solr-updater-1
openlibrary-web-1

[cdrini]

I think for the same reason, we might want to not make them identical to the container name. Also, the container name will be the same for both web containers, even if on is on ol-web1 vs ol-web2. This might be the best scheme: docker-{hostname}-{service}-{instance} eg docker-ol-web1-web-1 or docker-ol-home0-infobase-1, etc.

[cdrini]

We want to know the host in eg Sentry, cause the host can sometimes cause bugs.

[cclauss]

We want to know the host in eg Sentry

Each Sentry error includes server_name which is fairly evenly divided
between ol-web1.us.archive.org and ol-web2.us.archive.org...

[cdrini]

I believe sentry gets server_name from the HOSTNAME ; server_name will change to be whatever we set hostname to be. We do sometimes make variations and sort of lazy A/B tests by putting different code on the web servers, so we do need to know which webserver handled a request.