Implement BGP #156
sebiklamar
started this conversation in
Ideas
Replies: 1 comment
-
@sebiklamar Thanks for the info! This is a goldmine once I start testing it out. I've been way too busy these last months, but I think I'll be able to look at BGP with Cilium and UniFi soon™ Other resources I have for this are |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi Vegard,
as I saw you having BGP support on your roadmap list, I want to share my documentation and experience of my BGP w/ Opnsense including
EgressGateway
PoC.What's in the pocket
Ingress
(IngressGateway
) with any IP you want, i.e. you don't need to have a matching interface for that. Just name an IP to your liking in theio.cilium/lb-ipam-ips
annotation as usually and cilium will add it to the BGP routing.L2
routing/reachability of your BGP-managed IPs because your default gateway will do a RA for your hosts still residing in the same L2 net (your192.168.1.0/24
subnet), i.e. you can switch of yourL2Announcement
code. I didn't further investigate this b/c this is not a relevant scenario in my environment (k8s nodes in a separate server VLAN).Egress
(EgressGateway
) with BGP iff you have anIngress
defined in parallel. Due to the limitation mentioned in the next (4.) item there's no BGP advertisement for theEgress
. Hence, you need to define anIngress
in parallel (counter-measure 1) and you need to ensure theEgress
is set up for the same node as the pod using theIngress
is running (counter-measure 2) , i.e. you will need to pin (nodeSelector
) both theDeployment
and theCiliumEgressGatewayPolicy
.NB: For
Ingress
-only scenario (1.) no pinning is needed at all, i.e. you will have full flexibility.Egress
-only scenario (without anyIngress
for the same IP). Would need Cilium Enterprise Ed., cf. Cilium blog post for Cilium Enterprise 1.15.EgressGateway
feature, you would need to live withoutCiliumEndpointSlice
feature (cf. cilium issue 24833).CiliumEndpointSlice
is currently enabled in your environment.How-to
TLDR
For seeing all pieces in action see in my evolving homelab repo based on your setup
openssh
app forEgressGateway
(andIngressGateway
) example.additional-values.yaml
file for cilium installation, cf. details below.base
andenvs/dev
folders contain the Resource definitions for BGPLinks
BGPPeeringPolicy
), though good opnsense docu: https://baremetalblog.com/posts/tech/2024-03-12-cilium-bgp-and-you/EgressGateway
: https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/opnsense guide
See the 3rd link listed above which is having good screenshot documentation on opnsense setup.
My documented install notes from Obsidian:
os-frr
package (from Plugins tab)tcp/179
on interfaceK8S
Warning message during install
Configuration
Routing > General
Routing > BGP
General
Neigbours
k8s guide
Ensure BGP is enabled in cilium
For also having
Egress
feature available, ciliumvalues.yaml
file needs to be at least:FYI: I currently maintain a base
values.yaml
and an env.-specificadditional-values.yaml
file. Though, I didn't manage yet to maintain thevalues.yaml
file only once in thebase
folder with only patches being applied in the environments as overlay.Resource Definitions
Overview
base/CiliumBGPAdvertisements
: Defines prefixes that are injected into the BGP routing table.base/CiliumBGPPeerConfig
: A common set of BGP peering setting. It can be used across multiple peers.<env>/CiliumBGPClusterConfig
: Defines BGP instances and peer configurations that are applied to multiple nodes, specific to an<environment>
.CiliumBGPNodeConfigOverride
: Defines node-specific BGP configuration to provide a finer control, not used.<env>/CiliumBGPClusterConfig
environment-specific
ASNs
Just pick an ASN ID from the private AS number range 64.512 – 65.534 (16b) or from the newer 32b range 4.200.000.000 - 4.294.967.294.
Most documented setups (incl. mine at the moment) use different ASNs for the router (e.g. opnsense) and partner (k8s cluster), although, AFAIK, the same ASNs can be used (it's the same party/system "Vegard's system/environment").
Cheat Sheet
Have fun with a modern routing setup with certainly more robust connectivity (instead of the
k8sClientRateLimit
hack for L2), and true source IP.HTH -- Sebastian
Edit: Added cilium folders links in TLDR section as openssh only covers Egress and not BGP and links.
Beta Was this translation helpful? Give feedback.
All reactions