Skip to content

Commit 6bb2959

Browse files
larskslarsks
committed
Configure bond0 interface
We want to use bonded interface pairs on these system. The nodes aren't yet wired for it, but setting this up now will allow us to refer to the `bond0` interface in e.g. VLAN configurations (and means we won't have to re-work those later). Because we're using OVNKubernetes, we can't use nmstate [1] to enact the configuration. The recommendation is to apply the configuration using a MachineConfig [2] resource, but this is complicated by the fact that our nodes don't all have the same interface names, and it's not possible to apply node-specific machineconfigs [3]. We work around this solution by: 1. Copying nmconnection files for *all hosts* to *every host*, but placing them in `/etc/mco` (just because that's a convenient available directory, it seems relatively topical, and it's not possible to create new directories using the `directories` section of an ignition config). 2. Installing a systemd unit that runs a shell script at boot that copies the host-specific configs from `/etc/mco` into `/etc/NetworkManager/system-connections`. [1]: https://docs.openshift.com/container-platform/4.10/networking/k8s_nmstate/k8s-nmstate-about-the-k8s-nmstate-operator.html [2]: https://docs.openshift.com/container-platform/4.10/post_installation_configuration/machine-configuration-tasks.html [3]: openshift/machine-config-operator#1720 x-branch: feature/bond0
1 parent 023c830 commit 6bb2959

14 files changed

+233
-0
lines changed

‎cluster-scope/overlays/nerc-ocp-infra/kustomization.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ resources:
88
- ../../bundles/acm
99
- ../../bundles/odf
1010
- clusterversion.yaml
11+
- machineconfigs/configure-bond0
1112

1213
patches:
1314
- path: oauths/cluster_patch.yaml

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/Makefile

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
MACHINECONFIGS = \
2+
configure-bond0.yaml
3+
4+
%.yaml: %.in.yaml
5+
yq -y --argjson config "$$(yq .spec.config $< | butane -d src)" '.spec.config |= $$config' $< > $@ || { rm -f $@; exit 1; }
6+
7+
all: $(MACHINECONFIGS)
8+
9+
clean:
10+
rm -f $(MACHINECONFIGS)

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Including file content in ignition configs is a pain, because it has to be base64 encoded. The `Makefile` in this directory uses [Butane][] to transpile `MachineConfig` resources, automatically including and encoding content from files.
2+
3+
To run the `Makefile`, you will need:
4+
5+
- [Butane][]
6+
- [yq][]
7+
- [jq][]
8+
9+
[yq]: https://kislyuk.github.io/yq/
10+
[jq]: https://stedolan.github.io/jq/
11+
[butane]: https://coreos.github.io/butane/

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/configure-bond0.in.yaml

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
apiVersion: machineconfiguration.openshift.io/v1
2+
kind: MachineConfig
3+
metadata:
4+
name: configure-bond0
5+
labels:
6+
machineconfiguration.openshift.io/role: master
7+
spec:
8+
config:
9+
variant: fcos
10+
11+
# butane version 1.3.0 generates ignition 3.2.0 configs
12+
version: 1.3.0
13+
14+
storage:
15+
files:
16+
- path: /etc/mco/ctl-0-nic1.nmconnection
17+
contents:
18+
local: ctl-0-nic1.nmconnection
19+
- path: /etc/mco/ctl-0-nic2.nmconnection
20+
contents:
21+
local: ctl-0-nic2.nmconnection
22+
- path: /etc/mco/ctl-1-nic1.nmconnection
23+
contents:
24+
local: ctl-1-nic1.nmconnection
25+
- path: /etc/mco/ctl-1-nic2.nmconnection
26+
contents:
27+
local: ctl-1-nic2.nmconnection
28+
- path: /etc/mco/ctl-2-nic1.nmconnection
29+
contents:
30+
local: ctl-2-nic1.nmconnection
31+
- path: /etc/mco/ctl-2-nic2.nmconnection
32+
contents:
33+
local: ctl-2-nic2.nmconnection
34+
- path: /etc/NetworkConfiguration/system-connections/bond0.nmconnection
35+
contents:
36+
local: bond0.nmconnection
37+
- path: /etc/mco/configure-bond0-ports.sh
38+
contents:
39+
local: configure-bond0-ports.sh
40+
mode: 0755
41+
systemd:
42+
units:
43+
- name: configure-bond0-ports.service
44+
enabled: true
45+
contents: |
46+
[Unit]
47+
Description = Apply node-specific network configuration for bond0
48+
Requires = NetworkManager.service
49+
Before = NetworkManager.service
50+
51+
[Service]
52+
Type = oneshot
53+
ExecStart = /etc/mco/configure-bond0-ports.sh
54+
55+
[Install]
56+
WantedBy = multi-user.service

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/configure-bond0.yaml

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
apiVersion: machineconfiguration.openshift.io/v1
2+
kind: MachineConfig
3+
metadata:
4+
name: configure-bond0
5+
labels:
6+
machineconfiguration.openshift.io/role: master
7+
spec:
8+
config:
9+
ignition:
10+
version: 3.2.0
11+
storage:
12+
files:
13+
- path: /etc/mco/ctl-0-nic1.nmconnection
14+
contents:
15+
source: data:,%5Bconnection%5D%0Aid%3Dens6f0%0Atype%3Dethernet%0Ainterface-name%3Dens6f0%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
16+
- path: /etc/mco/ctl-0-nic2.nmconnection
17+
contents:
18+
source: data:,%5Bconnection%5D%0Aid%3Dens6f1%0Atype%3Dethernet%0Ainterface-name%3Dens6f1%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
19+
- path: /etc/mco/ctl-1-nic1.nmconnection
20+
contents:
21+
source: data:,%5Bconnection%5D%0Aid%3Dens5f0%0Atype%3Dethernet%0Ainterface-name%3Dens5f0%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
22+
- path: /etc/mco/ctl-1-nic2.nmconnection
23+
contents:
24+
source: data:,%5Bconnection%5D%0Aid%3Dens5f1%0Atype%3Dethernet%0Ainterface-name%3Dens5f1%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
25+
- path: /etc/mco/ctl-2-nic1.nmconnection
26+
contents:
27+
source: data:,%5Bconnection%5D%0Aid%3Dens5f0%0Atype%3Dethernet%0Ainterface-name%3Dens5f0%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
28+
- path: /etc/mco/ctl-2-nic2.nmconnection
29+
contents:
30+
source: data:,%5Bconnection%5D%0Aid%3Dens5f1%0Atype%3Dethernet%0Ainterface-name%3Dens5f1%0Amaster%3Dbond0%0Aslave-type%3Dbond%0Aautoconnect%3Dtrue%0A%0A%5Bethernet%5D%0Amtu%3D9000%0A
31+
- path: /etc/NetworkConfiguration/system-connections/bond0.nmconnection
32+
contents:
33+
compression: gzip
34+
source: data:;base64,H4sIAAAAAAAC/0zMwQqDMAzG8XvepaOCjO2QJ5EeavsNCzaRmgp7++EQtlvgn+83JRVBsqISqGSeVbIne2/4nlTE0F4xwUmsuHLspteOrXXQD7n9Nbev8cDOA9EEW9AEFqha56f3nmg6tUBVM3iOa5QE1xrVUqoKD+P5UrZjDFRhi2Y+bcpL2pyVCu3Gj/voPX0CAAD//04u+Q/FAAAA
35+
- path: /etc/mco/configure-bond0-ports.sh
36+
contents:
37+
compression: gzip
38+
source: data:;base64,H4sIAAAAAAAC/5SPvU4DMRCE+32KIaRCmFVoU1Eg0SQU0EUpfPYetwpen2yLHxHeHWEhIbqw5Y7m+zTnZzyo8eDrRJQOUQvcDJYWeCvtNZfDxpt/ksL1vTZJLmQzCU2zVRe1+uFZIqWXEyssVi+uLP2+/utiojEXzL5N0J92CpmXd/cPj9ubza0zDR+ry+vPP5o1YiYA0BG7HdyIZUfs92u0Saxn3xdmLHq0OHUSjkfImzasOmRUitmEvgIAAP//QY2VpFoBAAA=
39+
mode: 493
40+
systemd:
41+
units:
42+
- contents: '[Unit]
43+
44+
Description = Apply node-specific network configuration for bond0
45+
46+
Requires = NetworkManager.service
47+
48+
Before = NetworkManager.service
49+
50+
51+
[Service]
52+
53+
Type = oneshot
54+
55+
ExecStart = /etc/mco/configure-bond0-ports.sh
56+
57+
58+
[Install]
59+
60+
WantedBy = multi-user.service
61+
62+
'
63+
enabled: true
64+
name: configure-bond0-ports.service

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/kustomization.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
resources:
4+
- configure-bond0.yaml

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/bond0.nmconnection

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[connection]
2+
id=bond0
3+
type=bond
4+
interface-name=bond0
5+
autoconnect=true
6+
connection.autoconnect-slaves=1
7+
8+
[ethernet]
9+
mtu=9000
10+
11+
[bond]
12+
mode=balance-rr
13+
miimon=140
14+
15+
[ipv4]
16+
method=auto
17+
dhcp-timeout=86400

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/configure-bond0-ports.sh

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/bin/bash
2+
3+
mkdir -p /etc/NetworkManager/system-connections-disabled
4+
mv /etc/NetworkManager/system-connections/ens*.nmconnection /etc/NetworkManager/system-connections-disabled/
5+
6+
for path in /etc/mco/$HOSTNAME-nic{1,2}.nmconnection; do
7+
if [[ -f $path ]]; then
8+
cp "$path" /etc/NetworkManager/system-connections/ || exit 1
9+
fi
10+
done

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-0-nic1.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens6f0
3+
type=ethernet
4+
interface-name=ens6f0
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-0-nic2.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens6f1
3+
type=ethernet
4+
interface-name=ens6f1
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-1-nic1.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens5f0
3+
type=ethernet
4+
interface-name=ens5f0
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-1-nic2.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens5f1
3+
type=ethernet
4+
interface-name=ens5f1
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-2-nic1.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens5f0
3+
type=ethernet
4+
interface-name=ens5f0
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

‎cluster-scope/overlays/nerc-ocp-infra/machineconfigs/configure-bond0/src/ctl-2-nic2.nmconnection

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[connection]
2+
id=ens5f1
3+
type=ethernet
4+
interface-name=ens5f1
5+
master=bond0
6+
slave-type=bond
7+
autoconnect=true
8+
9+
[ethernet]
10+
mtu=9000

0 commit comments

Comments
 (0)