include sessionCookieEnabled when syncing routes

[rustyjux]

Description

Fixes APS-2743 - gwa scheduler is not using the route tag aps.route.session.cookie.enabled
Includes sessionCookieEnabled when examining the existing routes in OCP on the cluster being synced

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Documentation (non-breaking change with enhancements to documentation)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• I have checked that unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[ikethecoder]

remove code block

[rustyjux]

Ready for review. Changes worked as expected in dev and are currently deployed there.

[ikethecoder]

Are you sure it is OK to remove the raising of the SystemExit exception? I think I had to include this otherwise the SIGTERM signal is ignored if the process is in the middle of that code block. So could take longer to terminate the Pod.

[rustyjux]

My bad - I had removed earlier when trying to resolve errors that came up when deleting a route that doesn't exist (until I saw and modeled off microservices/kubeApi/tests/routers/test_del_route.py) and should have restored them.

More generally, I should have reviewed all my changes in the commit. Thanks for the catch

[rustyjux]

With the most recent changes in kube_api in dev (running in Gold which is in standby mode), I observed a route needing syncing being deleted and then re-inserted on the next scheduler run.

Detailed observations:

• there was an existing route not-sticky-ias9df9-dev which had the aps.route.session.cookie.enabled tag (so actually was sticky). was previously synced to both clsuters
• I removed the aps.route.session.cookie.enabled tag on the route in the gw config
• the update of the route in GoldDR is immediate and fine
• the first time sync occurs, the route is deleted in Gold and is missing - I actually observed in Route in OCP the route being absent

[2024-10-18 18:24:32,249] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:24:32,249] DEBUG routes.verify_and_create_routes: delete batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'dataPlane': 'konghd-kong-proxy', 'sessionCookieEnabled': True}]
[2024-10-18 18:24:32,250] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018182432-4d18599f5c
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:24:33,235] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.3542253514)
[2024-10-18 18:24:33,236] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:24:33,946] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:24:33,946] DEBUG routes.verify_and_create_routes: Deleting 1 routes
[2024-10-18 18:24:34,848] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Deleted route wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca
[2024-10-18 18:24:34,848] INFO h11_impl. send: 10.97.8.1:43580 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200

• on the next sync round, the route is recreated

[2024-10-18 18:40:47,239] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:40:47,239] DEBUG routes.verify_and_create_routes: delete batch: []
[2024-10-18 18:40:47,240] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018184047-2e8e88f2aa
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:40:48,236] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.)
[2024-10-18 18:40:48,237] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:40:48,841] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:40:48,841] INFO h11_impl. send: 10.97.8.1:19364 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200

• update gw config to restore the aps.route.session.cookie.enabled tag
• next sync cycle the route is deleted again

[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': True, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: delete batch: [{'name': 'wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'dataPlane': 'konghd-kong-proxy', 'sessionCookieEnabled': False}]
[2024-10-18 18:57:02,236] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241018185702-23d00a2cbe
[2024-10-18 18:57:03,136] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-18 18:57:03,136] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.3544619441)
[2024-10-18 18:57:03,136] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-18 18:57:03,935] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-18 18:57:03,935] DEBUG routes.verify_and_create_routes: Deleting 1 routes
[2024-10-18 18:57:04,833] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Deleted route wild-ns-gw-41c0c-not-sticky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca
[2024-10-18 18:57:04,833] INFO h11_impl. send: 10.97.8.1:62568 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200

So why is delete now happening after insert? Presumably this was not always the behaviour, though it does come second in routes and was always based on route name

gwa-api/microservices/kubeApi/routers/routes.py

Line 217 in eaa8ea6

kubectl_delete('route', route["name"])

I have set kube api back to dev branch in versions to avoid creating more issues

[ikethecoder]

Hmm... I think the reason is that for deletion, it really just needs to know whether the name is no longer in the source_routes. The name is derived from the selectTag and host so this insert+delete issue would not have happened (dataPlane doesn't change). When you added sessionCookieEnabled then differences get added into the delete list incorrectly. To fix, I would just compare the name for building the delete_batch.

[rustyjux]

Aha thanks for digging in there. That's a tricky side effect from changing in_list() which is used in the comparison. I updated tests to reflect where deletes should and should not be happening.

Newest code works as expected when a sync update is needed in DR:

[2024-10-21 23:50:50,966] DEBUG routes.verify_and_create_routes: insert batch: [{'name': 'wild-ns-gw-41c0c-sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'selectTag': 'ns.gw-41c0c', 'host': 'sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca', 'sessionCookieEnabled': False, 'dataPlane': 'konghd-kong-proxy'}]
[2024-10-21 23:50:50,966] DEBUG routes.verify_and_create_routes: delete batch: []
[2024-10-21 23:50:50,968] DEBUG routes.verify_and_create_routes: Creating 1 routes - tmp /tmp/sync/20241021235050-dfbc437d7c
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] ['sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca'] No override applied {}
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 matched ssl cert data-api.tls
[2024-10-21 23:50:51,956] DEBUG ocp_routes.prepare_apply_routes: [ns.gw-41c0c] Route A 001 wild-ns-gw-41c0c-sticky-icky-ias9df9-dev-api-gov-bc-ca.dev.api.gov.bc.ca (ver.2834629484)
[2024-10-21 23:50:51,956] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Prepared 1 routes
[2024-10-21 23:50:52,585] DEBUG routes.verify_and_create_routes: [gw-41c0c] - Applied 1 routes
[2024-10-21 23:50:52,585] INFO h11_impl. send: 10.95.10.1:14778 - "POST /namespaces/gw-41c0c/routes/sync HTTP/1.1" 200

[rustyjux]

Ugh probably should have squash merged. Could do a hard reset or we live with it