Skip to content

feat(routing): implemented back iptables filtering #360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

feat(routing): implemented back iptables filtering #360

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5dcf7d1
feat(routing): implemented back iptables filtering
lukas8219 Mar 24, 2025
3285016
added iptables binary in base image
lukas8219 Mar 24, 2025
cf5e94c
changed to TableFilter in wrong NAT
lukas8219 Mar 24, 2025
52d1c5f
Adjusted for requested changes
lukas8219 Mar 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,13 @@
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

FROM golang:1.24.1-alpine3.21 AS builder
FROM golang:1.24.1-alpine3.21 AS base

RUN apk add --no-cache clang llvm bpftool libbpf-dev
ENV CGO_ENABLED=0
ENV GOOS=linux

FROM base AS builder
WORKDIR /app

COPY go.mod go.sum ./
Expand All @@ -42,6 +45,7 @@ RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o gke-metadata-server \
github.com/matheuscscp/gke-metadata-server

FROM alpine:3.21
RUN apk add --no-cache iptables

COPY --from=builder /app/gke-metadata-server .

Expand Down
4 changes: 4 additions & 0 deletions api/api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,11 @@ const (
RoutingModeDefault = RoutingModeBPF
RoutingModeBPF = "eBPF"
RoutingModeLoopback = "Loopback"
RoutingModeIPTables = "IPTables"
Copy link
Collaborator Author

@lukas8219 lukas8219 Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should do it in lowercase?

Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think uppercase like this is fine 👍


GKEAnnotationServiceAccount = GroupGKE + "/gcp-service-account"
GKELabelNodeEnabled = GroupGKE + "/gke-metadata-server-enabled"

GKEMetadataServerAddressDefault = "169.254.169.254"
GKEMetadataServerPortDefault = 80
Comment on lines +43 to +44
Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
GKEMetadataServerAddressDefault = "169.254.169.254"
GKEMetadataServerPortDefault = 80
GKEMetadataServerDefaultAddress = "169.254.169.254"
GKEMetadataServerDefaultPort = 80

Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one, please check if we have these values anywhere else in the code and use these constants 🙏

)
3 changes: 2 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@ require (
github.com/sirupsen/logrus v1.9.3
github.com/spf13/pflag v1.0.6
github.com/stretchr/testify v1.10.0
github.com/vishvananda/netlink v1.3.0
github.com/vishvananda/netlink v1.3.1-0.20250206174618-62fb240731fa
Copy link
Collaborator Author

@lukas8219 lukas8219 Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is needed but I think i ran go mod tidy instead of go mod download ?

Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go mod tidy is the right one, go mod download is for dockerfile caching magic. Super weird this tag changing like this, their latest is still 1.3.0: https://github.com/vishvananda/netlink/releases/tag/v1.3.0

But the CI failed it ran make tidy (which runs go mod tidy) and found differences. I think it corrected the go.mod file back to simply 1.3.0, so I think there's something wrong with your environment modifying the go.mod file like this?

Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was also a diff on the file internal/iptables/iptables.go, this one is probably because of go fmt. Are you using format on save? I use VSCode format on save for Go.

golang.org/x/oauth2 v0.28.0
google.golang.org/api v0.226.0
k8s.io/api v0.32.3
k8s.io/apimachinery v0.32.3
k8s.io/client-go v0.32.3
k8s.io/klog/v2 v2.130.1
k8s.io/kubernetes v1.32.3
sigs.k8s.io/controller-runtime v0.20.3
)

Expand Down
8 changes: 6 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lithammer/dedent v1.1.0 h1:VNzHMVCBNG1j0fh3OrsFRkVUwStdDArbgBWoPAffktY=
github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mdlayher/netlink v1.7.2 h1:/UtM3ofJap7Vl4QWCPDGXY8d3GIY2UGSDbK+QWmY8/g=
Expand Down Expand Up @@ -170,8 +172,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/vishvananda/netlink v1.3.0 h1:X7l42GfcV4S6E4vHTsw48qbrV+9PVojNfIhZcwQdrZk=
github.com/vishvananda/netlink v1.3.0/go.mod h1:i6NetklAujEcC6fK0JPjT8qSwWyO0HLn4UKG+hGqeJs=
github.com/vishvananda/netlink v1.3.1-0.20250206174618-62fb240731fa h1:iAhToRwOrdk+pKzclvLM7nKZhsg8f7dVrgkFccDUbUw=
github.com/vishvananda/netlink v1.3.1-0.20250206174618-62fb240731fa/go.mod h1:i6NetklAujEcC6fK0JPjT8qSwWyO0HLn4UKG+hGqeJs=
github.com/vishvananda/netns v0.0.4 h1:Oeaw1EM2JMxD51g9uhtC0D7erkIjgmj8+JZc26m1YX8=
github.com/vishvananda/netns v0.0.4/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
Expand Down Expand Up @@ -286,6 +288,8 @@ k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f h1:GA7//TjRY9yWGy1poLzYYJJ4JRdzg3+O6e8I+e+8T5Y=
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f/go.mod h1:R/HEjbvWI0qdfb8viZUeVZm0X6IZnxAydC7YU42CMw4=
k8s.io/kubernetes v1.32.3 h1:2A58BlNME8NwsMawmnM6InYo3Jf35Nw5G79q46kXwoA=
k8s.io/kubernetes v1.32.3/go.mod h1:GvhiBeolvSRzBpFlgM0z/Bbu3Oxs9w3P6XfEgYaMi8k=
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 h1:M3sRQVHv7vB20Xc2ybTt7ODCeFj6JSWYFzOFnYeS6Ro=
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
sigs.k8s.io/controller-runtime v0.20.3 h1:I6Ln8JfQjHH7JbtCD2HCYHoIzajoRxPNuvhvcDbZgkI=
Expand Down
80 changes: 80 additions & 0 deletions internal/iptables/iptables.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
package iptables

import (
"errors"
"fmt"
"net/netip"
"strconv"

"github.com/matheuscscp/gke-metadata-server/api"
Copy link
Owner

@matheuscscp matheuscscp Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We put internal imports in a separate import block between the standard libraries and third party imports

Suggested change
"github.com/matheuscscp/gke-metadata-server/api"
"github.com/matheuscscp/gke-metadata-server/api"

"k8s.io/kubernetes/pkg/util/iptables"
"k8s.io/utils/exec"
)

func LoadAndAttach(emulatorAddr netip.Addr, emulatorPort int) func() (func() error, error) {
return func() (func() error, error) {
// Create the following iptables rules:
// iptables -t nat -A OUTPUT -d 169.254.169.254 -p tcp --dport 80 -j DNAT --to-destination <emulatorAddr>
// iptables -A FORWARD -d <emulatorIP> -p tcp --dport <emulatorPort> -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

// This rule essentially rewrites the destination of packets targeting the
// GKE metadata server with the ip:port address of the emulator, i.e. it
// effectively modifies the destination fields of matching packets.
ipTables := iptables.New(exec.New(), iptables.ProtocolIPv4)

// match conditions
natTableArgs := [...]string{
"-d", api.GKEMetadataServerAddressDefault,
"-p", "tcp",
"--dport", strconv.Itoa(api.GKEMetadataServerPortDefault),

// action taken
"-j", "DNAT",
"--to-destination", emulatorAddr.String(),
}

filterTableArgs := [...]string{
"-d", emulatorAddr.String(),
"-p", "tcp",
"--dport", strconv.Itoa(emulatorPort),
"-m", "state", "--state", "NEW,ESTABLISHED,RELATED", // new or established connections
// action taken
"-j", "ACCEPT",
}

close := func() error {
err1 := ipTables.DeleteRule(
iptables.Table(iptables.TableNAT),
iptables.ChainOutput,
natTableArgs[:]...,
)
err2 := ipTables.DeleteRule(
iptables.Table(iptables.TableNAT),
iptables.ChainForward,
filterTableArgs[:]...,
)
return errors.Join(err1, err2)
}

_, err := ipTables.EnsureRule(
iptables.Append,
iptables.TableNAT, // NAT rules are applied before routing
iptables.ChainOutput, // output chain is for locally generated traffic
natTableArgs[:]...,
)
if err != nil {
return nil, fmt.Errorf("error adding DNAT rule: %w", err)
}

// This rule ensures that packets destined to the emulator IP and port
// are accepted to be forwarded by the host, i.e. prevents the host from
// dropping matching packets.
_, err = ipTables.EnsureRule(
iptables.Append,
iptables.TableFilter, // filter table is for access control (should packets be forwarded or dropped?)
iptables.ChainForward, // forward chain is for packets that are being routed, i.e. not destined to the local host
filterTableArgs[:]...,
)
return close, err
}
}
3 changes: 3 additions & 0 deletions internal/routing/routing.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"net/netip"

"github.com/matheuscscp/gke-metadata-server/api"
"github.com/matheuscscp/gke-metadata-server/internal/iptables"
"github.com/matheuscscp/gke-metadata-server/internal/loopback"
"github.com/matheuscscp/gke-metadata-server/internal/redirect"

Expand All @@ -44,6 +45,8 @@ func LoadAndAttach(node *corev1.Node, emulatorIP netip.Addr, emulatorPort int) (
loadAndAttach = redirect.LoadAndAttach(emulatorIP, emulatorPort)
case api.RoutingModeLoopback:
loadAndAttach = loopback.LoadAndAttach
case api.RoutingModeIPTables:
loadAndAttach = iptables.LoadAndAttach(emulatorIP, emulatorPort)
default:
return "", nil, fmt.Errorf("invalid routing mode: %s", mode)
}
Expand Down
Loading