[Calico] Lack of cluster resources - TestNSE_Composition is unstable

[glazychev-art]

Description

When we add TestRunFeatureSuiteCalico, we can see that TestNSE_Composition doesn't work properly.
The first Request (from the init-container) is usually fine, but the following Requests (refreshes) fail. Since the datapath has already been done, it does not affect the ping.
But it affects subsequent tests: since new Requests (from cmd-nsc) can't complete successfully, we don't have a Connection that will be closed when the test is completed (but actually Connection exists because of the init-container)

Logs

Pay attention to TestNSE_Composition and TestSelect_Forwarder.
Logs from cmd-nsc TestSelect_Forwarder:

...
Mar 25 14:57:26.176�[37m [TRAC] [id:alpine-0] [type:networkService] �[0m(1.1)   request={"connection":{"id":"alpine-0","network_service":"nse-composition","mechanism"...
...

Calico logs(5).zip

There is a guess that this is due to insufficient resources on github host-machine

[glazychev-art]

GitHub uses Microsoft Azure Standard_DS2_v2 virtual machines as runners - https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
They have 2-core CPU. And this is not enough for nse-composition test, because:

• Сalico-VPP uses about 250m per node = 750m per kind-cluster
• nsmgr requests 200m per worker-node = 400m https://github.com/networkservicemesh/deployments-k8s/blob/main/apps/nsmgr/nsmgr.yaml#L59
• nse-composition requests 150m for each endpoint = 600m - https://github.com/networkservicemesh/deployments-k8s/blob/main/examples/features/nse-composition/passthrough-1.yaml#L44
• registry requests 100m - https://github.com/networkservicemesh/deployments-k8s/blob/main/examples/features/nse-composition/passthrough-1.yaml#L44
  ...

In total, we already have 1850m, and that's not all pods.

Typical resource consumption:

NAMESPACE              POD                                          NAME                      CPU(cores)   MEMORY(bytes)
calico-apiserver       calico-apiserver-6d6569994f-fg842            calico-apiserver          6m           33Mi
calico-apiserver       calico-apiserver-6d6569994f-lkn9t            calico-apiserver          4m           27Mi
calico-system          calico-kube-controllers-67f85d7449-26cn9     calico-kube-controllers   3m           19Mi
calico-system          calico-node-28hpm                            calico-node               16m          77Mi
calico-system          calico-node-c8nq6                            calico-node               21m          85Mi
calico-system          calico-node-tm6x8                            calico-node               20m          81Mi
calico-system          calico-typha-5ccbc59d8b-7n8zs                calico-typha              2m           25Mi
calico-system          calico-typha-5ccbc59d8b-sdkwl                calico-typha              2m           22Mi
calico-vpp-dataplane   calico-vpp-node-56nqz                        agent                     2m           33Mi
calico-vpp-dataplane   calico-vpp-node-56nqz                        vpp                       212m         642Mi
calico-vpp-dataplane   calico-vpp-node-9j7kv                        agent                     2m           40Mi
calico-vpp-dataplane   calico-vpp-node-9j7kv                        vpp                       214m         648Mi
calico-vpp-dataplane   calico-vpp-node-cdk26                        agent                     2m           34Mi
calico-vpp-dataplane   calico-vpp-node-cdk26                        vpp                       93m          602Mi
kube-system            coredns-64897985d-7vslh                      coredns                   3m           18Mi
kube-system            coredns-64897985d-rntkb                      coredns                   3m           14Mi
kube-system            etcd-kind-control-plane                      etcd                      24m          51Mi
kube-system            kube-apiserver-kind-control-plane            kube-apiserver            91m          475Mi
kube-system            kube-controller-manager-kind-control-plane   kube-controller-manager   10m          57Mi
kube-system            kube-proxy-bjddv                             kube-proxy                1m           14Mi
kube-system            kube-proxy-vgrl5                             kube-proxy                1m           14Mi
kube-system            kube-proxy-xp52v                             kube-proxy                1m           16Mi
kube-system            kube-scheduler-kind-control-plane            kube-scheduler            3m           21Mi
kube-system            metrics-server-7cf8b65d65-szgbd              metrics-server            5m           20Mi
local-path-storage     local-path-provisioner-5ddd94ff66-sfmfn      local-path-provisioner    2m           11Mi
ns-m8h9j               alpine                                       alpine                    0m           0Mi
ns-m8h9j               alpine                                       cmd-nsc                   19m          8Mi
ns-m8h9j               alpine                                       coredns                   1m           2Mi
ns-m8h9j               nse-firewall-vpp-588ff74db5-bcx6x            nse                       89m          245Mi
ns-m8h9j               nse-kernel-578f48f89c-zlw7v                  nginx                     0m           2Mi
ns-m8h9j               nse-kernel-578f48f89c-zlw7v                  nse                       2m           8Mi
ns-m8h9j               nse-passthrough-1-7c87d76bcd-542sf           nse                       81m          233Mi
ns-m8h9j               nse-passthrough-2-77fb784bb7-6rfn5           nse                       83m          240Mi
ns-m8h9j               nse-passthrough-3-65b5d75896-nrq4r           nse                       81m          235Mi
nsm-system             admission-webhook-k8s-787ffbf7b7-rx4pm       admission-webhook-k8s     1m           10Mi
nsm-system             forwarder-vpp-nxrp4                          forwarder-vpp             60m          23Mi
nsm-system             forwarder-vpp-q9dj2                          forwarder-vpp             67m          20Mi
nsm-system             nsmgr-kxtz7                                  exclude-prefixes          1m           11Mi
nsm-system             nsmgr-kxtz7                                  nsmgr                     49m          26Mi
nsm-system             nsmgr-pmrcv                                  exclude-prefixes          1m           10Mi
nsm-system             nsmgr-pmrcv                                  nsmgr                     67m          28Mi
nsm-system             registry-k8s-5dbfd4bbfb-xld6v                registry                  12m          14Mi
spire                  spire-agent-2clqc                            spire-agent               53m          36Mi
spire                  spire-agent-c5hbf                            spire-agent               41m          37Mi
spire                  spire-server-0                               k8s-workload-registrar    1m           14Mi
spire                  spire-server-0                               spire-server              7m           27Mi
tigera-operator        tigera-operator-b876f5799-5dfw4              tigera-operator           3m           36Mi

So currently it is not possible to run TestNSE_Composition with Calico