v0.2.0: hairpin-proxy-etchosts-controller DaemonSet for /etc/hosts rewrites

Author: compumike

README.md

@@ -7,7 +7,7 @@ If you've had problems with ingress-nginx, cert-manager, LetsEncrypt ACME HTTP01
 ## One-line install
 
 ```shell
-kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.1.2/deploy.yml
+kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy.yml
 ```
 
 If you're using [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) and [cert-manager](https://github.com/jetstack/cert-manager), it will work out of the box. See detailed installation and testing instructions below.
@@ -72,7 +72,7 @@ The `dig` should show the external load balancer IP address. The first `curl` sh
 ### Step 1: Install hairpin-proxy in your Kubernetes cluster
 
 ```shell
-kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.1.2/deploy.yml
+kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy.yml
 ```
 
 If you're using `ingress-nginx`, this will work as-is.
@@ -119,3 +119,15 @@ This time, the first `dig` should show an internal service IP address (generally
 NOTE: CoreDNS is a cache, so even if you see the `rewrite` rules in Step 2, it will take another minute or two before the queries resolve correctly. Be patient. You may wish to `watch -n 1 dig subdomain.example.com` to see when this changeover happens.
 
 At this point, cert-manager's self-check will pass, and you'll get valid LetsEncrypt certificates within a few minutes.
+
+### Step 4: (Optional) Install hairpin-proxy-etchosts-controller DaemonSet
+
+Note that the CoreDNS rewrites above only cover access within containers, while the iptables rewrite applies to the Node itself. This mismatch causes a problem if your node itself needs to access something behind your ingress. An example is if you're hosting your own container registry with [trow](https://github.com/ContainerSolutions/trow) and it's behind the ingress. If you follow only steps 1-3 above, you'll experience image pull failures because the Docker daemon (running on the Node directly, not in a container) can't access your registry.
+
+To resolve this, we need to rewrite the DNS on the Node itself. The Node does not use CoreDNS, so we can instead rewrite `/etc/hosts` to point to the IP address of the `hairpin-proxy-haproxy` service. This runs as a DaemonSet, so that it can modify each Node's copy of `/etc/hosts`.
+
+To install this DaemonSet:
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/compumike/hairpin-proxy/v0.2.0/deploy-etchosts-daemonset.yml
+```

deploy-etchosts-daemonset.yml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    app: hairpin-proxy-etchosts-controller
+  name: hairpin-proxy-etchosts-controller
+  namespace: hairpin-proxy
+spec:
+  selector:
+    matchLabels:
+      app: hairpin-proxy-etchosts-controller
+  template:
+    metadata:
+      labels:
+        app: hairpin-proxy-etchosts-controller
+    spec:
+      serviceAccountName: hairpin-proxy-controller-sa
+      containers:
+        - image: compumike/hairpin-proxy-controller:0.2.0
+          name: main
+          command: ["/app/src/main.rb", "--etc-hosts", "/app/etchosts"]
+          volumeMounts:
+            - name: etchosts
+              mountPath: /app/etchosts
+          resources:
+            requests:
+              memory: "50Mi"
+              cpu: "10m"
+            limits:
+              memory: "100Mi"
+              cpu: "50m"
+      volumes:
+        - name: etchosts
+          hostPath:
+            path: /etc/hosts
+            type: File

deploy.yml

@@ -23,7 +23,7 @@ spec:
         app: hairpin-proxy-haproxy
     spec:
       containers:
-        - image: compumike/hairpin-proxy-haproxy:0.1.2
+        - image: compumike/hairpin-proxy-haproxy:0.2.0
           name: main
           resources:
             requests:
@@ -151,7 +151,7 @@ spec:
         runAsUser: 405
         runAsGroup: 65533
       containers:
-        - image: compumike/hairpin-proxy-controller:0.1.2
+        - image: compumike/hairpin-proxy-controller:0.2.0
           name: main
           resources:
             requests:

hairpin-proxy-controller/src/main.rb

@@ -3,6 +3,8 @@
 
 require "k8s-ruby"
 require "logger"
+require "optparse"
+require "socket"
 
 class HairpinProxyController
   COMMENT_LINE_SUFFIX = "# Added by hairpin-proxy"
@@ -32,7 +34,9 @@ def fetch_ingress_hosts
       end
     }.flatten
     all_tls_blocks = all_ingresses.map { |r| r.spec.tls }.flatten.compact
-    all_tls_blocks.map(&:hosts).flatten.compact.sort.uniq
+    hosts = all_tls_blocks.map(&:hosts).flatten.compact
+    hosts.filter! { |host| /\A[A-Za-z0-9.\-_]+\z/.match?(host) }
+    hosts.sort.uniq
   end
 
   def coredns_corefile_with_rewrite_rules(original_corefile, hosts)
@@ -68,10 +72,69 @@ def check_and_rewrite_coredns
     end
   end
 
+  def dns_rewrite_destination_ip_address
+    Addrinfo.ip(DNS_REWRITE_DESTINATION).ip_address
+  end
+
+  def etchosts_with_rewrite_rules(original_etchosts, hosts)
+    # Returns a String represeting the original /etc/hosts file, modified to include a rule for
+    # mapping *hosts to dns_rewrite_destination_ip_address. This handles kubelet and the node's Docker engine,
+    # which does not go through CoreDNS.
+    # This is an idempotent transformation because our rewrites are labeled with COMMENT_LINE_SUFFIX.
+
+    # Extract base configuration, without our hairpin-proxy rewrites
+    our_lines, original_lines = original_etchosts.strip.split("\n").partition { |line| line.strip.end_with?(COMMENT_LINE_SUFFIX) }
+
+    ip = dns_rewrite_destination_ip_address
+    hostlist = hosts.join(" ")
+    new_rewrite_line = "#{ip}\t#{hostlist} #{COMMENT_LINE_SUFFIX}"
+
+    if our_lines == [new_rewrite_line]
+      # Return early so that we're indifferent to the ordering of /etc/hosts lines.
+      return original_etchosts
+    end
+
+    (original_lines + [new_rewrite_line]).join("\n") + "\n"
+  end
+
+  def check_and_rewrite_etchosts(etchosts_path)
+    @log.info("Polling all Ingress resources and etchosts file at #{etchosts_path}...")
+    hosts = fetch_ingress_hosts
+
+    old_etchostsfile = File.read(etchosts_path)
+    new_etchostsfile = etchosts_with_rewrite_rules(old_etchostsfile, hosts)
+
+    if old_etchostsfile.strip != new_etchostsfile.strip
+      @log.info("/etc/hosts has changed! New contents:\n#{new_etchostsfile}\nWriting to #{etchosts_path}...")
+      File.write(etchosts_path, new_etchostsfile)
+    end
+  end
+
   def main_loop
+    etchosts_path = nil
+
+    OptionParser.new { |opts|
+      opts.on("--etc-hosts ETCHOSTSPATH", "Path to writable /etc/hosts file") do |h|
+        etchosts_path = h
+        raise "File #{etchosts_path} doesn't exist!" unless File.exist?(etchosts_path)
+        raise "File #{etchosts_path} isn't writable!" unless File.writable?(etchosts_path)
+      end
+    }.parse!
+
+    if etchosts_path && etchosts_path != ""
+      @log.info("Starting in /etc/hosts mutation mode on #{etchosts_path}. (Intended to be run as a DaemonSet: one instance per Node.)")
+    else
+      etchosts_path = nil
+      @log.info("Starting in CoreDNS mode. (Indended to be run as a Deployment: one instance per cluster.)")
+    end
+
     @log.info("Starting main_loop with #{POLL_INTERVAL}s polling interval.")
     loop do
-      check_and_rewrite_coredns
+      if etchosts_path.nil?
+        check_and_rewrite_coredns
+      else
+        check_and_rewrite_etchosts(etchosts_path)
+      end
 
       sleep(POLL_INTERVAL)
     end