Connection Issues with VPC CNI Network Policy Enforcement

[janavenkat]

What happened:

We recently enabled network policy in AWS VPC CNI. After enabling it, we started experiencing intermittent connectivity issues when EKS pods attempt to connect to RabbitMQ hosted behind VPC peering. The issue occurs randomly, even though we do not have any deny network policies—only allow_all ingress and allow_all egress policies are in place.

Disabling network policy enforcement resolves the issue, while re-enabling it causes the problem to return. The network-policy-agent logs indicate a possible reversal in connection direction, suggesting a potential issue with conntrack.

We managed to get the DENY logs by enabling enablePolicyEventLogs

Also we're using Otterize to automate our network polices.

Attach logs

 {
       "@timestamp": "2025-02-04 10:36:49.034",
       "@message": "Node: REDACTED.compute.internal; SIP: <INTERNAL_IP>; SPORT: 53912; DIP: <EXTERNAL_IP>; DPORT: 5671; PROTOCOL: TCP; PolicyVerdict: ACCEPT"
     },
     {
       "@timestamp": "2025-02-04 10:36:48.986",
       "@message": "Node: REDACTED.compute.internal; SIP: <EXTERNAL_IP>; SPORT: 5671; DIP: <INTERNAL_IP>; DPORT: 53912; PROTOCOL: TCP; PolicyVerdict: DENY"
     }
}

What you expected to happen:

We shouldn't expect any denied connections, as we're allowing all ingress and egress rules

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.29
• CNI Version: 1.19.0
• OS (e.g: cat /etc/os-release): Amazon linux 2
• Kernel (e.g. uname -a): 5.10.233-233.887.amzn2.x86_64

[orsenthil]

@janavenkat - this might be related to #345 please follow that issue.

[janavenkat]

@orsenthil Thank you and the fix seems merged. When we can expect the release in EKS Addons? but for now, I can only see version 1.19.2 available.

[orsenthil]

EKS Addons release is being made available, AWS docs will be updated. once the addons is available in all regions.

[Pavani-Panakanti]

@janavenkat Can you collect full logs with latest 1.19.3 version using this script https://github.com/awslabs/amazon-eks-ami/blob/main/log-collector-script/linux/README.md and send the node log bundle to "k8s-awscni-triage@amazon.com". That will help us further debug the issue. Thanks

[janavenkat]

@janavenkat Can you collect full logs with latest 1.19.3 version using this script https://github.com/awslabs/amazon-eks-ami/blob/main/log-collector-script/linux/README.md and send the node log bundle to "k8s-awscni-triage@amazon.com". That will help us further debug the issue. Thanks

I will send as soon as possible. So, you need these logs from the node where the connection gets blocked? The reason for asking while performing testing on new nodes is that they may scale up and the script will not run.

[janavenkat]

@janavenkat Can you collect full logs with latest 1.19.3 version using this script https://github.com/awslabs/amazon-eks-ami/blob/main/log-collector-script/linux/README.md and send the node log bundle to "k8s-awscni-triage@amazon.com". That will help us further debug the issue. Thanks

Sent the logs

[Pavani-Panakanti]

@janavenkat Yes we need the logs from the node where connection is blocked and also we should have network policy event logs enabled which will tell us about the DENY's. Thanks

[janavenkat]

@janavenkat Yes we need the logs from the node where connection is blocked and also we should have network policy event logs enabled which will tell us about the DENY's. Thanks

I sent the logs from the node where the connection is blocked.

[janavenkat]

@janavenkat Yes we need the logs from the node where connection is blocked and also we should have network policy event logs enabled which will tell us about the DENY's. Thanks

Yesterday, I sent the logs, and it appears that you have downloaded them. Please let me know if you require any additional information.

[janavenkat]

@janavenkat Yes we need the logs from the node where connection is blocked and also we should have network policy event logs enabled which will tell us about the DENY's. Thanks

@Pavani-Panakanti are there any updates? Thank you.

[Pavani-Panakanti]

@janavenkat We have identified an issue and are working on the fix. It should be available in the next release

[aquam8]

Can you please confirm that a fix is in progress for this issue? AFAIK it's still an issue i'm experiencing. Thanks!!