Skip to content

Commit

Permalink
Deduplicate status messages (rancher#2859)
Browse files Browse the repository at this point in the history
Prevents fleet from crashing due to resources exceeding etcd's
configured size limit.

Deduplicate messages should only be necessary for edge cases which are
not officially supported by fleet but result in ever increasing message
sizes.

This is due to the messages being copied from one resource to another
and back again. Every resource adds its status to the message. This only
happens if a cluster group is deployed by a GitRepo, which results in a
bundle containing a cluster group. This bundle can only become ready if
the cluster group is ready, but if the cluster group points to the
cluster of the bundle that deployed the cluster group, this cannot ever
happen. The user is expected to fix this situation but deduplicating the
messages prevents the message from growing up to the point where etcd's
limit is reached and fleet crashes.

Deduplicating the messages also has the effect of not changing the
status of resources frequently, which results in less controllers being
triggered.
  • Loading branch information
p-se committed Oct 31, 2024
1 parent 615f45c commit 030e841
Showing 1 changed file with 25 additions and 1 deletion.
26 changes: 25 additions & 1 deletion internal/helmdeployer/kustomize/kstatus.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package kustomize

import (
"strings"

"github.com/rancher/fleet/internal/cmd/agent/deployer/data"
"github.com/rancher/fleet/internal/cmd/agent/deployer/summary"
fleetv1 "github.com/rancher/fleet/pkg/apis/fleet.cattle.io/v1alpha1/summary"
Expand Down Expand Up @@ -31,7 +33,29 @@ func KStatusSummarizer(obj data.Object, conditions []summary.Condition, summary
}

if result.Message != "" {
summary.Message = append(summary.Message, result.Message)
// Deduplicate messages here. This should only be necessary for edge
// cases which are not officially supported but result in ever
// increasing message sizes. This is due to the messages being copied
// from one resource to another and back again. Every resource adds its
// status to the message. This only happens if a cluster group is
// deployed by a GitRepo, which results in a bundle containing a cluster
// group. This bundle can only ever become ready if the cluster group is
// ready, but if the cluster group points to the cluster of the bundle,
// this cannot ever happen. This is an edge case and should be fixed by
// the user. Deduplicating the messages prevents the message from
// growing up to the point where etcd's limit is reached and fleet
// crashes.
messages := make(map[string]bool)
var resultMessages []string
for _, message := range strings.Split(result.Message, ";") {
if _, ok := messages[message]; ok {
continue
}
messages[message] = true
resultMessages = append(resultMessages, message)
}

summary.Message = append(summary.Message, strings.Join(resultMessages, ";"))
}

return summary
Expand Down

0 comments on commit 030e841

Please sign in to comment.