Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable podCIDR allocation from control-plane when using calico #10639

Merged
merged 2 commits into from
Dec 12, 2023
+18 −6
Merged

Disable podCIDR allocation from control-plane when using calico #10639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7239349
Disable control plane allocating podCIDR for nodes when using calico
VannTen Nov 21, 2023
be555fd
Don't default to kube_network_node_prefix for calico_pool_blocksize
VannTen Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions roles/kubernetes/control-plane/meta/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ dependencies:
when:
- etcd_deployment_type == "kubeadm"
- not (ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk", "ClearLinux"] or is_fedora_coreos)
- role: network_plugin/calico_defaults
4 changes: 4 additions & 0 deletions roles/kubernetes/control-plane/templates/kubeadm-config.v1beta3.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,11 +292,15 @@ controllerManager:
cluster-cidr: "{{ kube_pods_subnet }}{{ ',' + kube_pods_subnet_ipv6 if enable_dual_stack_networks else '' }}"
{% endif %}
service-cluster-ip-range: "{{ kube_service_addresses }}{{ ',' + kube_service_addresses_ipv6 if enable_dual_stack_networks else '' }}"
{% if kube_network_plugin is defined and kube_network_plugin == "calico" and not calico_ipam_host_local %}
allocate-node-cidrs: "false"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? The default value is true. I believe it won't affect the operation of calico-ipam, as calico-ipam does not allocate addresses from node.PodCIDR; it allocates from blocks it creates on its own. These two are not correlated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not affect calico, it affects kube-controller-manager.
The current kubespray defaults are kube_network_node_prefix = 24 and calico_pool_blocksize = 26.
We faced the following situation during upgrade, ( described in projectcalico/calico#7722).
While calico is fine, the controller-manager thinks it has no more cidrs to allocate, and give the node the CIDRNotAvailable condition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyclinder @VannTen This would be equally applicable when using Cilium (when Cilium IPAM mode is not k8s host scope), right? If you confirm I'll make a MR to add a variable like kube_allocate_node_cidrs here, and I guess in v1beta4 too.

{% else %}
{% if enable_dual_stack_networks %}
node-cidr-mask-size-ipv4: "{{ kube_network_node_prefix }}"
node-cidr-mask-size-ipv6: "{{ kube_network_node_prefix_ipv6 }}"
{% else %}
node-cidr-mask-size: "{{ kube_network_node_prefix }}"
{% endif %}
{% endif %}
profiling: "{{ kube_profiling }}"
terminated-pod-gc-threshold: "{{ kube_controller_terminated_pod_gc_threshold }}"
Expand Down
3 changes: 3 additions & 0 deletions roles/network_plugin/calico/meta/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
---
dependencies:
- role: network_plugin/calico_defaults
2 changes: 1 addition & 1 deletion roles/network_plugin/calico/tasks/check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@
- name: "Check if inventory match current cluster configuration"
assert:
that:
- calico_pool_conf.spec.blockSize | int == (calico_pool_blocksize | default(kube_network_node_prefix) | int)
- calico_pool_conf.spec.blockSize | int == calico_pool_blocksize | int
- calico_pool_conf.spec.cidr == (calico_pool_cidr | default(kube_pods_subnet))
- not calico_pool_conf.spec.ipipMode is defined or calico_pool_conf.spec.ipipMode == calico_ipip_mode
- not calico_pool_conf.spec.vxlanMode is defined or calico_pool_conf.spec.vxlanMode == calico_vxlan_mode
Expand Down
4 changes: 2 additions & 2 deletions roles/network_plugin/calico/tasks/install.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@
"name": "{{ calico_pool_name }}",
},
"spec": {
"blockSize": {{ calico_pool_blocksize | default(kube_network_node_prefix) }},
"blockSize": {{ calico_pool_blocksize }},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default(26) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's already in defaults/main.yml, I'd rather avoid duplicating.

"cidr": "{{ calico_pool_cidr | default(kube_pods_subnet) }}",
"ipipMode": "{{ calico_ipip_mode }}",
"vxlanMode": "{{ calico_vxlan_mode }}",
Expand Down Expand Up @@ -274,7 +274,7 @@
"name": "{{ calico_pool_name }}-ipv6",
},
"spec": {
"blockSize": {{ calico_pool_blocksize_ipv6 | default(kube_network_node_prefix_ipv6) }},
"blockSize": {{ calico_pool_blocksize_ipv6 }},
"cidr": "{{ calico_pool_cidr_ipv6 | default(kube_pods_subnet_ipv6) }}",
"ipipMode": "{{ calico_ipip_mode_ipv6 }}",
"vxlanMode": "{{ calico_vxlan_mode_ipv6 }}",
Expand Down
2 changes: 1 addition & 1 deletion roles/network_plugin/calico/templates/calico-config.yml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ data:
"etcd_key_file": "{{ calico_cert_dir }}/key.pem",
"etcd_ca_cert_file": "{{ calico_cert_dir }}/ca_cert.crt",
{% endif %}
{% if calico_ipam_host_local is defined %}
{% if calico_ipam_host_local %}
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"
Expand Down
8 changes: 6 additions & 2 deletions ...s/network_plugin/calico/defaults/main.yml → ..._plugin/calico_defaults/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@ calico_vxlan_mode: Always # valid values are 'Always', 'Never' and 'CrossSubnet
calico_cni_pool: true
calico_cni_pool_ipv6: true

# add default ippool blockSize (defaults kube_network_node_prefix)
# add default ippool blockSize
calico_pool_blocksize: 26

# Calico doesn't support ipip tunneling for the IPv6.
calico_ipip_mode_ipv6: Never
calico_vxlan_mode_ipv6: Never

# add default ipv6 ippool blockSize (defaults kube_network_node_prefix_ipv6)
# add default ipv6 ippool blockSize
calico_pool_blocksize_ipv6: 122

# Calico network backend can be 'bird', 'vxlan' and 'none'
Expand Down Expand Up @@ -161,6 +161,10 @@ calico_ipam_autoallocateblocks: true
# Calico IPAM maxBlocksPerHost, default 0
calico_ipam_maxblocksperhost: 0

# Calico host local IPAM (use node .spec.podCIDR)

calico_ipam_host_local: false

# Calico apiserver (only with kdd)
calico_apiserver_enabled: false

Expand Down