Based on the ideas discussed in the issues [1] and PRs [2][3][6], it's been decided to switch from using an L4 reverse proxy to an L7 forward proxy to properly block peer network traffic, without the need to use external tools. The design aims to implement only the minimal required features that satisfy blocking incoming and outgoing peer traffic. Complicated features such as packet reordering, packet delivery delay, etc. to a future container-based solution. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and network traffic in scenario (b) is not blocked at all. We introduce an L7 forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the L4 reverse proxy, as the L7 forward proxy holds the information of the destination, we can block all incoming and outgoing traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy --- B ^ newly introduced ``` The main subtasks are - redesigned as an L7 forward proxy - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to bypass the limitation of `http.ProxyFromEnvironment` - implement an L7 forward proxy Known limitations are - Doesn't support unix socket, as L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for peer communication, only a few of the e2e test cases use unix sockets as the majority of e2e test cases use HTTP/HTTPS. It's been discussed and decided that it is ok for now without the unix socket support. - it's L7 so we need to send a perfectly crafted HTTP request - doesn’t support reordering, dropping, or manipulating packets on the fly - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) [6] etcd-io#17938 Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow. Signed-off-by: Siyuan Zhang <sizhang@google.com> Signed-off-by: Iván Valdés Castillo <iv@nvald.es> Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>

Commits

Commits on Sep 26, 2024

Remove PauseAccept of the reverse proxy from the e2e test

Remove DelayAccept of the reverse proxy from the e2e test

Remove LatencyAccept of the reverse proxy from the e2e test

Remove PauseTx and PauseRx of the reverse proxy from the e2e test

Fix Blackhole implementation for e2e tests