Skip to content

Commit a143e3c

Browse files
authored
Merge pull request #419 from ekoops/ekoops/k8s-polykube
pcn-loadbalancer-rp MULTI port mode and pcn-k8sdispatcher
2 parents 577bf22 + 76616a0 commit a143e3c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+7480
-217
lines changed

AUTHORS

+1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ off on commits in the Polycube repository:
1010
Gianluca Scopelliti [email protected]
1111
Giuseppe Ognibebe [email protected]
1212
Jianwen Pi [email protected]
13+
Leonardo Di Giovanna [email protected]
1314
Matteo Bertrone [email protected]
1415
Mauricio Vásquez Bernal [email protected]
1516
Nico Caprioli [email protected]
Loading
Loading
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# K8dispatcher
2+
3+
The ``pcn-k8sdispatcher`` service is specifically designed as part of our Kubernetes networking solution (please see [polykube](https://github.com/polycube-network/polykube) to get more information about it). The service provides an eBPF implementation of a custom NAT: it performs different actions depending on the type and on the direction of the traffic.
4+
5+
For Egress Traffic, the following flow chart can be used to explain the functioning of the service:
6+
7+
![K8sdispatcher egress flow chart](egress.png)
8+
9+
The Egress Traffic is the traffic generated by Pods and directed to the external world.
10+
This traffic can be generated by an internal Pod that wants to contact the external world or as a response to an
11+
external world request. For this traffic, the service maintains an egress session table containing information
12+
about the active egress sessions. The first time a Pod wants to contact the external world, no active egress session
13+
will be present in the table: in this scenario, the service performs SNAT, replacing the address of the Pod
14+
with the address of the node, and creates entries in the ingress and egress session table accordingly.
15+
If the outgoing traffic is generated as a response to an external request, it can only be originated as a response to
16+
a request made to a NodePort Service. For traffic related to NodePort Services with a CLUSTER ExternalTrafficPolicy,
17+
if an egress session table hit happens, the destination IP address and port are replaced accordingly to the session data.
18+
The traffic related to NodePort Services with a LOCAL ExternalTrafficPolicy is forwarded as it is to the next cube.
19+
20+
For Ingress Traffic, the following flow chart can be used to explain the functioning of the service:
21+
22+
![K8sdispatcher ingress flow chart](ingress.png)
23+
24+
The Ingress Traffic can be differentiated in traffic directed to the host (either directly or because it needs VxLAN
25+
processing) and traffic directed to Pods. The traffic directed to Pods can be the traffic generated by an external host
26+
trying to contact a NodePort service or the return traffic generated by an external host providing a response to an
27+
internal Pod request. The service uses an ingress session table containing all the active ingress sessions.
28+
If a session table hit happens, the service apply NAT according to the session data. If no session table entry is
29+
associated with the incoming packet, the service tries to determine if a NodePort rule matches the packet
30+
characteristics. In case of no NodePort rule matching, the packet is sent to the Linux stack for further processing.
31+
In case of NodePort rule matching, different actions are applied according to the ExternalTrafficPolicy of the
32+
Kubernetes NodePort Service associated to the rule. If the policy is LOCAL, the traffic is allowed to reach only
33+
backend Pods located on the current node: in this case the packet can proceed towards the Pod without modifications.
34+
In case the policy is CLUSTER, the packet can also reach backend Pods located on other nodes: since later in
35+
the chain the packet will be processed by a load balancer and the return packet will have to transit through
36+
the same load balancer, SNAT is applied by replacing the source IP address with a specific reserved address belonging
37+
to the Pod CIDR of the node on which the k8sdispatcher is deployed. In this way the two nodes (the one that
38+
receives the request and the one running the selected backend Pod) will exchange the packets of the flow over
39+
the VxLAN interconnect. In this latter case, corresponding session entries are stored into the ingress and egress
40+
sessions tables.

Documentation/services/pcn-loadbalancer-rp/loadbalancer-rp.md

+10-5
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,17 @@
22

33

44
This service implements a ``Reverse Proxy Load Balancer``.
5-
According to the algorithm, incoming IP packets are delivered to the real servers by replacing their IP destination address with the one of the real server, chosen by the load balancing logic. Hence, IP address rewriting is performed in both directions, for traffic coming from the Internet and the reverse.
6-
Packet are hashed to determine which is the correct backend; the hashing function guarantees that all packets belonging to the same TCP/UDP session will always be terminated to the same backend server.
5+
According to the algorithm, incoming IP packets are delivered to the real servers by replacing their IP destination address with the one of the real server, chosen by the load balancing logic. Hence, IP address rewriting is performed in both directions, for traffic coming from the Internet and the reverse. Packet are hashed to determine which is the correct backend; the hashing function guarantees that all packets belonging to the same TCP/UDP session will always be terminated to the same backend server.
76

8-
Unknown packets (e.g., ARP; IPv6) are simply forwarded as they are.
7+
This service supports two different port types (FRONTEND and BACKEND) and two different port modes (SINGLE and MULTI). Depending on the port mode, the cube can have one or more FRONTEND ports: in SINGLE port mode (which is the default one), only a single FRONTEND port is supported, whereas multiple FRONTEND ports are supported in MULTI port mode. The MULTI port mode is specifically designed in order to allow the service to work properly as part of our Kubernetes networking solution (please see [polykube](https://github.com/polycube-network/polykube) to get more information about it). Regardless the port mode, only a BACKEND port is supported.
98

9+
If a packet coming from a FRONTEND port is directed to a service, DNAT is performed on it; the corresponding reverse natting operation is performed for packets coming from backends and on the way back to clients. ARP packets are forwarded as they are (in MULTI port mode, if the packet is an ARP request from the BACKEND port, it is forwarded to the right FRONTEND port). Unknown packets (e.g., IPv6) are simply forwarded as they are (in MULTI port mode, if the packet comes from the BACKEND port, it is flooded to all the FRONTEND ports).
1010

1111
## Features
1212

1313

14+
- Support for different port modes (SINGLE and MULTI)
15+
- Support for multiple frontend ports (in MULTI port mode)
1416
- Support for multiple virtual services (with multiple ``vip:protocol:port`` tuples)
1517
- Support for ICMP Echo Request, hence enabling to ``ping`` virtual servers
1618
- Session affinity: a TCP session is always terminated to the same backend server even in case the number of backend servers changes at run-time (e.g., a new backend is added)
@@ -21,8 +23,11 @@ Unknown packets (e.g., ARP; IPv6) are simply forwarded as they are.
2123

2224
## Limitations
2325

26+
- In SINGLE port mode, only two ports are supported (a FRONTEND and a BACKEND port)
27+
- In MULTI port mode, multiple FRONTEND port are supported but only a single BACKEND port can exists
28+
- In MULTI port mode, an IPv4 address must be configured on FRONTEND port creation in order to allow packets to flow back to the frontend clients
29+
- In MULTI port mode, the supported topology is the one leveraged in the [polykube](https://github.com/polycube-network/polykube) Kubernetes networking solution
2430

25-
- Supports only two interfaces
2631

2732
## How to use
2833

@@ -38,7 +43,7 @@ Each backend supports a ``weight`` that determines how incoming sessions are dis
3843

3944
A set of ``virtual services``, which are specified by a Virtual IP address, protocol and a port (``vip:protocol:port``), are mapped to a given set of ``backend services``, actually running on multiple real servers.
4045

41-
Hence, this service exports two network interfaces:
46+
Hence, in SINGLE port mode, this service exports two network interfaces:
4247
- Frontend port: connects the LB to the clients that connect to the virtual service, likely running on the public Internet
4348
- Backend port: connects the LB to to backend servers
4449

scripts/install.sh

+4-2
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,8 @@ if [ "$MODE" == "pcn-iptables" ]; then
125125
-DENABLE_SERVICE_SIMPLEFORWARDER=OFF \
126126
-DENABLE_SERVICE_TRANSPARENTHELLOWORLD=OFF \
127127
-DENABLE_SERVICE_SYNFLOOD=OFF \
128-
-DENABLE_SERVICE_PACKETCAPTURE=OFF
128+
-DENABLE_SERVICE_PACKETCAPTURE=OFF \
129+
-DENABLE_SERVICE_K8SDISPATCHER=OFF
129130
elif [ "$MODE" == "pcn-k8s" ]; then
130131
cmake .. -DENABLE_SERVICE_BRIDGE=OFF \
131132
-DENABLE_SERVICE_DDOSMITIGATOR=ON \
@@ -143,7 +144,8 @@ elif [ "$MODE" == "pcn-k8s" ]; then
143144
-DENABLE_SERVICE_SIMPLEFORWARDER=OFF \
144145
-DENABLE_SERVICE_TRANSPARENTHELLOWORLD=OFF \
145146
-DENABLE_SERVICE_SYNFLOOD=OFF \
146-
-DENABLE_SERVICE_PACKETCAPTURE=ON
147+
-DENABLE_SERVICE_PACKETCAPTURE=ON \
148+
-DENABLE_SERVICE_K8SDISPATCHER=OFF
147149
else
148150
cmake .. -DENABLE_PCN_IPTABLES=ON
149151
fi

src/services/CMakeLists.txt

+1
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ add_service(transparenthelloworld pcn-transparent-helloworld)
3737
add_service(synflood pcn-synflood)
3838
add_service(packetcapture pcn-packetcapture)
3939
add_service(dynmon pcn-dynmon)
40+
add_service(k8sdispatcher pcn-k8sdispatcher)
4041

4142
# save string to create code that load the services
4243
SET_PROPERTY(GLOBAL PROPERTY LOAD_SERVICES_ ${LOAD_SERVICES})
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Swagger Codegen Ignore
2+
# Generated by swagger-codegen https://github.com/swagger-api/swagger-codegen
3+
4+
# Use this file to prevent files from being overwritten by the generator.
5+
6+
.swagger-codegen-ignore
7+
8+
src/*.cpp
9+
src/*.h
10+
11+
!src/*Interface.h
12+
!src/*JsonObject.h
13+
!src/*JsonObject.cpp
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
cmake_minimum_required (VERSION 3.2)
2+
3+
set (CMAKE_CXX_STANDARD 11)
4+
5+
add_subdirectory(src)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
module k8sdispatcher {
2+
yang-version 1.1;
3+
namespace "http://polycube.network/k8sdispatcher";
4+
prefix "k8sdispatcher";
5+
6+
import polycube-base { prefix "polycube-base"; }
7+
import polycube-standard-base { prefix "polycube-standard-base"; }
8+
9+
import ietf-inet-types { prefix "inet"; }
10+
11+
organization "Polycube open source project";
12+
description "YANG data model for the Polycube K8s Dispatcher";
13+
14+
polycube-base:service-description "K8s Dispatcher Service";
15+
polycube-base:service-version "2.0.0";
16+
polycube-base:service-name "k8sdispatcher";
17+
polycube-base:service-min-kernel-version "4.14.0";
18+
19+
typedef l4-proto {
20+
type enumeration {
21+
enum "TCP" {
22+
value 6;
23+
description "The TCP protocol type";
24+
}
25+
enum "UDP" {
26+
value 17;
27+
description "The UDP protocol type";
28+
}
29+
enum "ICMP" {
30+
value 1;
31+
description "The ICMP protocol type";
32+
}
33+
}
34+
description "L4 protocol";
35+
}
36+
37+
uses "polycube-standard-base:standard-base-yang-module" {
38+
augment ports {
39+
leaf type {
40+
type enumeration {
41+
enum BACKEND { description "Port connected to the internal CNI topology"; }
42+
enum FRONTEND { description "Port connected to the node NIC"; }
43+
}
44+
description "Type of the K8s Dispatcher cube port (e.g. BACKEND or FRONTEND)";
45+
mandatory true;
46+
polycube-base:init-only-config;
47+
}
48+
leaf ip {
49+
type inet:ipv4-address;
50+
description "IP address of the node interface (only for FRONTEND port)";
51+
polycube-base:cli-example "10.10.1.1";
52+
polycube-base:init-only-config;
53+
}
54+
}
55+
}
56+
57+
leaf internal-src-ip {
58+
type inet:ipv4-address;
59+
description "Internal source IP address used for natting incoming packets directed to Kubernetes Services with a CLUSTER external traffic policy";
60+
mandatory true;
61+
polycube-base:cli-example "10.10.1.1";
62+
polycube-base:init-only-config;
63+
}
64+
65+
leaf nodeport-range {
66+
type string;
67+
description "Port range used for NodePort Services";
68+
default "30000-32767";
69+
polycube-base:cli-example "30000-32767";
70+
}
71+
72+
list session-rule {
73+
key "direction src-ip dst-ip src-port dst-port proto";
74+
description "Session entry related to a specific traffic direction";
75+
config false;
76+
77+
leaf direction {
78+
type enumeration {
79+
enum INGRESS {
80+
description "Direction of traffic going from the internal topology to the external world";
81+
}
82+
enum EGRESS {
83+
description "Direction of traffic going from the external world to the internal CNI topology";
84+
}
85+
}
86+
description "Session entry direction (e.g. INGRESS or EGRESS)";
87+
}
88+
leaf src-ip {
89+
type inet:ipv4-address;
90+
description "Session entry source IP address";
91+
}
92+
leaf dst-ip {
93+
type inet:ipv4-address;
94+
description "Session entry destination IP address";
95+
}
96+
leaf src-port {
97+
type inet:port-number;
98+
description "Session entry source L4 port number";
99+
}
100+
leaf dst-port {
101+
type inet:port-number;
102+
description "Session entry destination L4 port number";
103+
}
104+
leaf proto {
105+
type l4-proto;
106+
description "Session entry L4 protocol";
107+
polycube-base:cli-example "TCP, UDP, ICMP";
108+
}
109+
110+
leaf new-ip {
111+
type inet:ipv4-address;
112+
description "Translated IP address";
113+
config false;
114+
}
115+
leaf new-port {
116+
type inet:port-number;
117+
description "Translated L4 port number";
118+
config false;
119+
}
120+
leaf operation {
121+
type enumeration {
122+
enum XLATE_SRC { description "The source IP and port are replaced"; }
123+
enum XLATE_DST { description "The destination IP and port are replaced"; }
124+
}
125+
description "Operation applied on the original packet";
126+
config false;
127+
}
128+
leaf originating-rule {
129+
type enumeration {
130+
enum POD_TO_EXT {
131+
description "Traffic related to communication between a Pod and the external world";
132+
}
133+
enum NODEPORT_CLUSTER {
134+
description "Traffic related to communication involving a NodePort Service with having a CLUSTER external traffic policy";
135+
}
136+
}
137+
description "Rule originating the session entry";
138+
config false;
139+
}
140+
}
141+
142+
list nodeport-rule {
143+
key "nodeport-port proto";
144+
description "NodePort rule associated with a Kubernetes NodePort Service";
145+
146+
leaf nodeport-port {
147+
type inet:port-number;
148+
description "NodePort rule nodeport port number";
149+
polycube-base:cli-example "30500";
150+
}
151+
leaf proto {
152+
type l4-proto;
153+
description "NodePort rule L4 protocol";
154+
polycube-base:cli-example "TCP, UDP, ICMP";
155+
}
156+
157+
leaf external-traffic-policy {
158+
type enumeration {
159+
enum LOCAL { description "Incoming traffic is allowed to be served only by local backends"; }
160+
enum CLUSTER { description "Incoming traffic is allowed to be served by any backend of the cluster"; }
161+
}
162+
default CLUSTER;
163+
description "The external traffic policy of the Kubernetes NodePort Service";
164+
}
165+
leaf rule-name {
166+
type string;
167+
description "An optional name for the NodePort rule";
168+
polycube-base:cli-example "my-nodeport-rule";
169+
polycube-base:init-only-config;
170+
}
171+
}
172+
}
173+
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
include(${PROJECT_SOURCE_DIR}/cmake/LoadFileAsVariable.cmake)
2+
3+
aux_source_directory(serializer SERIALIZER_SOURCES)
4+
aux_source_directory(api API_SOURCES)
5+
aux_source_directory(base BASE_SOURCES)
6+
7+
include_directories(serializer)
8+
9+
if (NOT DEFINED POLYCUBE_STANDALONE_SERVICE OR POLYCUBE_STANDALONE_SERVICE)
10+
find_package(PkgConfig REQUIRED)
11+
pkg_check_modules(POLYCUBE libpolycube)
12+
include_directories(${POLYCUBE_INCLUDE_DIRS})
13+
endif (NOT DEFINED POLYCUBE_STANDALONE_SERVICE OR POLYCUBE_STANDALONE_SERVICE)
14+
15+
# Needed to load files as variables
16+
include_directories(${CMAKE_CURRENT_BINARY_DIR})
17+
18+
add_library(pcn-k8sdispatcher SHARED
19+
${SERIALIZER_SOURCES}
20+
${API_SOURCES}
21+
${BASE_SOURCES}
22+
K8sdispatcher.cpp
23+
NodeportRule.cpp
24+
Ports.cpp
25+
SessionRule.cpp
26+
K8sdispatcher-lib.cpp
27+
Utils.cpp)
28+
29+
# load ebpf datapath code a variable
30+
load_file_as_variable(pcn-k8sdispatcher
31+
K8sdispatcher_dp.c
32+
k8sdispatcher_code)
33+
34+
# load datamodel in a variable
35+
load_file_as_variable(pcn-k8sdispatcher
36+
../datamodel/k8sdispatcher.yang
37+
k8sdispatcher_datamodel)
38+
39+
target_link_libraries(pcn-k8sdispatcher ${POLYCUBE_LIBRARIES})
40+
41+
# Specify shared library install directory
42+
43+
set(CMAKE_INSTALL_LIBDIR /usr/lib)
44+
45+
install(
46+
TARGETS
47+
pcn-k8sdispatcher
48+
DESTINATION
49+
"${CMAKE_INSTALL_LIBDIR}"
50+
)

0 commit comments

Comments
 (0)