[DEV] Add CoreDHCP to Helm

[rainest]

Short Description
Remove the existing dnsmasq Deployment from the chart and replace it with CoreDHCP, for OpenCHAMI/roadmap#50

Definition of Done

• The chart spawns a CoreDHCP instance.
• The chart spawns a tftpd service that CoreDHCP can use.
• CoreDHCP is able to talk to the chart's SMD instance.

Additional context
Ref #78 and #84 for equivalent work on the Podman side.

[rainest]

I'm not sure how much we considered the needs of the DHCP server for the original dnsmasq Deployment. It was running, but IDK if we ever had a proof of concept for anything talking to it.

The DHCP server will handle requests for hosts outside the Kubernetes network. Normal broadcast delivery will not work as such, and we'd need to forward traffic to it.

This is apparently how CSM handles DHCP also--it has a Kea DHCP instance exposing a LoadBalancer, with metallb BGP peering to node networks and forwarding rules to the LB address on the node network (see CSM's PXE troubleshooting and DHCP troubleshooting guides).

I'm unsure where all that gets configured for full CSM setups, but found a basic example of minimal configuration for such a setup.

I don't think there's any way to handle dynamic population of the server_id or router config or that dynamic handling would even be desirable. AFAIK these will need to be config parameters that we just trust you've set to the correct value. The V4 ID needs to match the spec.loadBalancerIP. I don't think any other in-cluster config cares about the V6 ID.

---

We oddly had an existing tftpd key, but weren't using it in any of the templates. It was added alongside dnsmasq and the dnsmasq built-in TFTP server configuration.

[rainest]

#87 provides a basic "it runs!" set of values and templates, with some caveats:

• The coresmd image is currently busted after some possibly incomplete file rearrangement upstream. The /coredhcp path in the image is a directory with a README.md; it's apparently supposed to get replaced with a binary built from some templated Go. I hulk smashed the (working) release binary from the repo into a local image build instead.
• SMD in the chart does not appear to serve TLS. I stuffed a fake cert into the SMD plugin config. The plugin appears to have connected over HTTP fine (it logged level=debug msg="EthernetInterfaces: map[]" prefix="plugins/coresmd" for my empty SMD instance with no errors).
• Config generation needs design work. The current YAML in YAML in YAML templates approach works, but is ugly as sin.

[synackd]

The coresmd image is currently busted after some possibly incomplete file rearrangement upstream. The /coredhcp path in the image is a directory with a README.md; it's apparently supposed to get replaced with a binary built from some templated Go. I hulk smashed the (working) release binary from the repo into a local image build instead.

Does the latest version v0.0.5 work for you? I examined it and the /coredhcp is a binary in this version.

[alexlovelltroy]

SMD in the chart does not appear to serve TLS. I stuffed a fake cert into the SMD plugin config. The plugin appears to have connected over HTTP fine (it logged level=debug msg="EthernetInterfaces: map[]" prefix="plugins/coresmd" for my empty SMD instance with no errors).

At this point, we have not enabled TLS at the SMD level and rely on the API gateway for TLS termination and signed tokens for authN/authZ. Having said that, we have the ACME pieces running and we could create and rotate TLS certificates for the microservices using that or we could protect them using an mTLS service mesh. This matters more for k8S deployments than it does in our podman deployments

Do you have a proposal for mTLS within k8s for SMD that doesn't preclude the current operations?

[alexlovelltroy]

I don't think there's any way to handle dynamic population of the server_id or router config or that dynamic handling would even be desirable. AFAIK these will need to be config parameters that we just trust you've set to the correct value. The V4 ID needs to match the spec.loadBalancerIP. I don't think any other in-cluster config cares about the V6 ID.

You're driving at the right stuff here. We may need to explore options outside of the standard k8s networking in order to get this to work reliably. I've never understood how networking would work to bring DHCP properly into a remote k8s cluster without complex and unpleasant VLAN incantations. The solution in CSM only works because of direct connections to the worker nodes and plenty of VLAN tagging.

[synackd]

The coresmd image is currently busted after some possibly incomplete file rearrangement upstream. The /coredhcp path in the image is a directory with a README.md; it's apparently supposed to get replaced with a binary built from some templated Go. I hulk smashed the (working) release binary from the repo into a local image build instead.

Does the latest version v0.0.5 work for you? I examined it and the /coredhcp is a binary in this version.

@rainest Ah, I found the issue. We were originally pushing coresmd as the container name and then started pushing coredhcp. This led to the former one not working while the latter did. We have deleted the coresmd container to eliminate confusion. Going forward, we should use ghcr.io/openchami/coredhcp as the CoreDHCP container that has the coresmd plugins built-in.

Thanks for reporting the issue!

[synackd]

I will update the quickstart docker-compose recipe to use the correct container.

[synackd]

The above PR also fixes an issue where 'permission denied' would be seen when binding to port 67. Fixed in coresmd v0.0.6.

[rainest]

TIL TFTP is one of those archaic protocols that doesn't much care for the realities of NAT and plays fun games with ports. Because it sends responses from arbitrary high ports instead of 69, there's no matching entry in the iptables CNI goop, and the responses get dropped.

I'm not exactly sure how CSM's existing TFTP handles this, since I don't see much of obvious interest in its Service definition. The BOS diagram doesn't go into NAT and the PXE overview doesn't cover much anything aside from the special switch routes.

On my KinD instance:

coredhcp-984f64465-8kk9s            1/1     Running     0               32m     10.244.0.42   kind-control-plane   <none>           <none>

is the internal single Pod IP. metallb is between the client and the Pod, but the problem is after that, within the cluster network, AFAICT.

From within the kind container, tcpdump will show inbound requests to the Pod from 10.244.0.1 (IDK enough about the CNI innards to properly say what this is), with attempted outbound traffic on random high ports that goes nowhere:

22:19:43.417657 veth84a235e5 In  IP 10.244.0.42.35051 > 10.244.0.1.11374: UDP, length 516
22:19:43.417690 veth84a235e5 Out IP 10.244.0.1 > 10.244.0.42: ICMP 10.244.0.1 udp port 11374 unreachable, length 552
22:19:43.822215 veth84a235e5 In  IP 10.244.0.42.47660 > 10.244.0.1.11374: UDP, length 516
22:19:43.822252 veth84a235e5 Out IP 10.244.0.1 > 10.244.0.42: ICMP 10.244.0.1 udp port 11374 unreachable, length 552
22:19:44.057596 veth84a235e5 In  IP 10.244.0.42.45425 > 10.244.0.1.11374: UDP, length 516
22:19:44.057647 veth84a235e5 Out IP 10.244.0.1 > 10.244.0.42: ICMP 10.244.0.1 udp port 11374 unreachable, length 552
22:19:44.426346 veth84a235e5 Out IP 10.244.0.1.11374 > 10.244.0.42.69: TFTP, length 27, RRQ "ipxe-x86_64.efi" netascii
22:19:44.439219 veth84a235e5 In  IP 10.244.0.42.48518 > 10.244.0.1.11374: UDP, length 516
22:19:44.439265 veth84a235e5 Out IP 10.244.0.1 > 10.244.0.42: ICMP 10.244.0.1 udp port 11374 unreachable, length 552
22:19:44.636699 veth84a235e5 In  IP 10.244.0.42.48515 > 10.244.0.1.11374: UDP, length 516
22:19:44.636741 veth84a235e5 Out IP 10.244.0.1 > 10.244.0.42: ICMP 10.244.0.1 udp port 11374 unreachable, length 552

Code state looks like what'd I'd expect: it successfully reads the file, but the rf.ReadFrom() call times out (I assume it's waiting for some sort of TFTP completed response from the client):

(dlv) p err
error(*net.OpError) *{
	Op: "read",
	Net: "udp",
	Source: net.Addr(*net.UDPAddr) *{
		IP: net.IP len: 16, cap: 16, [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
		Port: 11374,
		Zone: "",},
	Addr: net.Addr nil,
	Err: error(*internal/poll.DeadlineExceededError) *{},}
(dlv) p nbytes
512
(dlv) p raddr
"10.244.0.1"
(dlv) p filePath
"/tftpboot/ipxe-x86_64.efi"