Skip to content

fix(txt-registry): skip creation of already-existing TXT records (#4914) #5459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

u-kai
Copy link
Contributor

@u-kai u-kai commented May 24, 2025

What does it do ?

Adds a cache-based filter to TXTRegistry so that External-DNS no longer
tries to recreate TXT records that already exist in the provider.
Fixes #4914

Motivation

When only the data record (A/CNAME) is removed, External-DNS next sync loop
recreates that record and also tries to create the unchanged TXT record,
which providers like Route53 reject with “record already exists”.
Caching existing TXT entries and skipping them in ApplyChanges() prevents
this failure.

More

  • Yes, this PR title follows Conventional Commits
  • Yes, I added unit tests
  • Yes, I updated end user documentation accordingly

@k8s-ci-robot k8s-ci-robot requested a review from mloiseleur May 24, 2025 09:43
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 24, 2025
@k8s-ci-robot k8s-ci-robot requested a review from szuecs May 24, 2025 09:43
@k8s-ci-robot
Copy link
Contributor

Welcome @u-kai!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 24, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @u-kai. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 24, 2025
@mloiseleur
Copy link
Collaborator

mloiseleur commented May 24, 2025

Thanks.
I'm not sure of this approach 🤔 It may increase memory usage on big users.
/ok-to-test
cc @szuecs

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 24, 2025
@mloiseleur
Copy link
Collaborator

mloiseleur commented May 24, 2025

What happens the txt record has also been deleted ?
It will still be in this cache, so not re-created ?

@u-kai
Copy link
Contributor Author

u-kai commented May 24, 2025

@mloiseleur
Thanks for pointing that out — you’re right, the current patch keeps the TXT snapshot alive across the entire reconcile loop.
Apologies — I overlooked this in the first draft.

Proposed update

func (r *TXTRegistry) ApplyChanges(ctx context.Context, c *plan.Changes) error {
    ...

    if err := r.provider.ApplyChanges(ctx, c); err != nil {
        return err
    }

    // ✨ NEW: drop the snapshot so the next Records() call rebuilds it
    r.existingTXT = txtSnapshot{}  // formerly existingTXT

    return nil
}
  • Memory – The snapshot now lives for just one reconciliation pass.
    In my understanding, GC might run a little later than before, but the peak heap usage right after r.provider.Records() when the full slice of endpoints is allocated — remains exactly the same.
    If I’m mistaken, please let me know. 🙇

  • Correctness – If a TXT record is deleted between loops, the next Records() call rebuilds the snapshot without it, so the data record will be recreated.

Let me know if this per-loop cleanup addresses your concerns; I can push the change right away. 🙇‍♂️

@u-kai

This comment was marked as off-topic.

@mloiseleur

This comment was marked as off-topic.

@u-kai

This comment was marked as off-topic.

@ivankatliarchuk
Copy link
Contributor

ivankatliarchuk commented May 25, 2025

There are concerns Regarding TXT Caching as a Solution

I'm unclear how caching will resolve the underlying issue. From my perspective, this looks like a bug in either the CRD source, the TXT registry, or how records and plans are being handled. Instead of identifying and fixing that root cause, the current proposal is to introduce a new feature – TXT caching. The issue missing how-to reproduce, what the arguments are and version for reported bug is ~0.13, when latest version is 0.17, which means, the issue may no longer even exist.

The approach could actually worsen the situation for TXT records. Implementing a cache on a stateless service means the problem is likely to reappear during restarts or new service startups.

This proposed caching mechanism isn't a fix. We need to find the root cause of the problem. The expectation should be to reproduce the issue in a controlled environment and then try to pinpoint where the bug lies.

Relevant issue #4998

It appears there's a bug within either the CRD source, the TXT registry, or the mechanism for pulling records for a specific provider and handling the plan. Instead of addressing this underlying bug, the decision has been made to introduce a new feature: TXT caching.

@ivankatliarchuk
Copy link
Contributor

TXT registry already have a cache

recordsCache []*endpoint.Endpoint
and local endpoints and TXT records cache https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L138-L141

This is a high-impact issue, and without being able to reproduce it, it's tough to pinpoint the exact cause. It's even possible that upsert-only failed, but everything works fine when the sync flag is set.

I propose to start with:

  • Adding a unit test that specifically shows the issue where the record is deleted but the registry record persists.
  • Reproducing the actual bug on a local cluster using the Route53 provider and the latest external-dns.
  • Implementing the fix once the root cause is identified.

First two could be done in any order, but with focus on reproducing the problem and providing a unit test, that reveals the bug.

This is currently just an assumption, and without reproducing it, it's hard to be certain. We need to confirm that if a TXT registry record exists without a corresponding managed record (like A, AAAA, or CNAME), we either:

  • Mark the TXT registry record for deletion (though this won't work with upsert-only) and then apply the new changes, OR
  • Leave the TXT registry record untouched and only update the actual record.

@ivankatliarchuk
Copy link
Contributor

Fixes #5340

@ivankatliarchuk
Copy link
Contributor

ivankatliarchuk commented May 25, 2025

Fixes #5003

^ this issue is related, but not sure where scope of the fix will cover it

@u-kai
Copy link
Contributor Author

u-kai commented May 25, 2025

@ivankatliarchuk

Thank you for your feedback.

Initially, I approached this by introducing a caching mechanism.
However, based on mloiseleur suggestion, I revised the implementation to create a temporary data structure that exists only between the Records and ApplyChanges phases.
This structure functions similarly to a filter: it identifies and excludes any TXT records that already exist, preventing their re-creation. Once ApplyChanges is executed, this temporary structure is cleaned up.

The primary issue I'm addressing is ensuring that ExternalDNS can recreate an A record if it was accidentally deleted. Currently, when attempting to create an A record, ExternalDNS also tries to create a corresponding TXT record via the TXTRegistry.
If the TXT record already exists, providers like Route53 return an error, causing the entire operation to fail.

To resolve this, I implemented a mechanism to store existing TXT records during the Records phase. Before ApplyChanges attempts to create new records, it checks against this stored data to ensure it doesn't attempt to recreate existing TXT records, thereby avoiding errors.

Regarding recordsCache, I understand now that it's intended for non-TXT records and is scoped differently than my current change.
In contrast, my approach is limited to TXT records only, and the snapshot is used strictly within a single reconciliation loop to avoid stale state.
So while the naming may appear similar, the intention and scope are different.

I’ve pushed the updated implementation based on this approach.

Please let me know if I misunderstood any part of your suggestion — happy to revise if needed!

@u-kai
Copy link
Contributor Author

u-kai commented May 25, 2025

Hi @mloiseleur, I mentioned earlier that I would wait for your confirmation before pushing the changes, but I received a detailed comment from @ivankatliarchuk and pushed the revised implementation to help clarify the current behavior.

The latest code uses a temporary structure between Records and ApplyChanges instead of caching across reconciliation loops, as previously discussed.

Let me know if this direction still looks good to you — I’m happy to adjust if needed!

@ivankatliarchuk
Copy link
Contributor

Have you managed to validate that issue is still present or it's related to specific flags

  • upsert-only
  • sync
    ?

@ivankatliarchuk
Copy link
Contributor

relates: #3977

@u-kai
Copy link
Contributor Author

u-kai commented May 25, 2025

@ivankatliarchuk

Yes, I was able to reproduce the issue with both --policy=sync and --policy=upsert-only.
For clarity, I'm using hoge.com here as a placeholder — the actual domain has been replaced.

Environment

  • branch: master
  • command:
     external-dns \
    --provider aws \
    --registry txt \
    --txt-owner-id demo \
    --policy sync \ # and upsert-only
    --log-level debug \
    --source ingress
    
  • ingress resource
    apiVersion: networking.k8s.io/v1
     kind: Ingress
    metadata:
      name: demo-wild
      annotations:
        external-dns.alpha.kubernetes.io/aws-weight: "100"
        external-dns.alpha.kubernetes.io/hostname: api.hoge.com
        external-dns.alpha.kubernetes.io/set-identifier: demo
    spec:
      rules:
        - host: "api.hoge.com"
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: demo-service
                    port:
                      number: 80

Steps to Reproduce

  1. Deploy ExternalDNS.
  2. Deploy the above Ingress and let ExternalDNS create the records.
  3. Manually delete the A record for api.hoge.com from Route53.

Wait for the next reconciliation loop.

Result

On the next loop, ExternalDNS tries to recreate the A record and the associated TXT record.
However, since the TXT record still exists, Route 53 throws an error due to the attempted re-creation of an already-existing record.

DEBU[0001] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0001] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0001] Adding a-api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
INFO[0001] Desired change: CREATE a-api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0001] Desired change: CREATE api.hoge.com A   profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0001] Desired change: CREATE api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
...
DEBU[0061] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0061] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0061] Adding a-api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
INFO[0061] Desired change: CREATE a-api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0061] Desired change: CREATE api.hoge.com A   profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0061] Desired change: CREATE api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
ERRO[0061] Failure in zone hoge.com. when submitting change batch: operation error Route 53: ChangeResourceRecordSets, https response error StatusCode: 400, RequestID: 08d2eafe-c886-46d3-bd49-c5f905a7d3ee, InvalidChangeBatch: [Tried to create resource record set [name='a-api.hoge.com.', type='TXT', set-identifier='demo'] but it already exists, Tried to create resource record set [name='api.hoge.com.', type='TXT', set-identifier='demo'] but it already exists]  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
ERRO[0062] Failed to do run once: soft error
failed to submit all changes for the following zones: [/hostedzone/ZONEIDXXXXXXXXX]

@u-kai
Copy link
Contributor Author

u-kai commented May 25, 2025

@ivankatliarchuk
Thanks a lot for the review and helpful suggestion!
I've reflected the changes accordingly.

I'm currently reviewing the test cases and will follow up once everything is ready.

registry/txt.go Outdated
Comment on lines 321 to 328
generatedTXTs := im.generateTXTRecord(r)
for _, txt := range generatedTXTs {
// If the TXT record is already managed by this instance, skip it
if im.existingTXTs.isManaged(txt) {
continue
}
filteredChanges.Create = append(filteredChanges.Create, txt)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand this logic behind the solution.
Why we could not have something simple like?

for _, r := range filteredChanges.Create {
		if im.existingTXTs.is(not)Managed(r) {
			continue
		}
		if r.Labels == nil {
			r.Labels = make(map[string]string)
		}
		r.Labels[endpoint.OwnerLabelKey] = im.ownerID

		if im.cacheInterval > 0 {
			im.addToCache(r)
		}
	}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion!

The reason the logic is structured this way is because filteredChanges.Create does not initially contain TXT records.
The TXT records are generated on the fly using im.generateTXTRecord(r), and those are what I intend to filter using existingTXTs.

Checking r directly (as in your example) wouldn’t work in this case, since it’s typically an A or CNAME record, not the TXT itself.
Let me know if I misunderstood your suggestion!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current implementation seems quite expensive in terms for computation.

It should be possible to pass any record to im.existingTXTs.is(not)Managed and validate it somehow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed a refactor — the generateTXTRecordWithFilter part is a bit more involved, but the main loop is now cleaner and should be more efficient overall.
Open to feedback, let me know what you think!

@mloiseleur
Copy link
Collaborator

@u-kai Would you please rebase this PR on top of master, where the flaky test has been fixed ?

@szuecs
Copy link
Contributor

szuecs commented Jun 13, 2025

Also, when only the TXT record is deleted, it currently won’t be recreated — this is a separate issue that I plan to address in a future PR, also using existingTXTs.

We don't want to recreate TXT records in this case!
It's the mechanism that identifies the ownership and if you mix a zone with multiple external-dns or external-dns and manual records you would create a mess if you would recreate the deleted txt record. A txt record deletion means, that you don't want to let this record being owned by the external-dns instance.

}

// isNotManaged reports whether the given endpoint's TXT record is absent from the existing set.
// Used to determine whether a new TXT record needs to be created.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the doc string is wrong, but we do not want to recreate the TXT record!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant here was that this is used to determine whether a TXT record should be created — I didn’t intend to say that it is about recreating the record.
But if this is confusing, I’m happy to remove it.

func (im *existingTXTs) reset() {
// Reset the existing TXT records for the next reconciliation loop.
// This is necessary because the existing TXT records are only relevant for the current reconciliation cycle.
im.entries = make(map[recordKey]struct{})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a possible data race, I think we should guard all access to im.entries by a Mutex

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need a Mutex here. From what I can see, TXTRegister is only used inside Controller.RunOnce, which doesn't seem to use any goroutines.
Also, other cache fields in TXTRegister are being updated without a Mutex as well, so it seems consistent not to add one here.

registry/txt.go Outdated
@@ -244,7 +301,9 @@ func (im *TXTRegistry) generateTXTRecord(r *endpoint.Endpoint) []*endpoint.Endpo
txt.WithSetIdentifier(r.SetIdentifier)
txt.Labels[endpoint.OwnedRecordLabelKey] = r.DNSName
txt.ProviderSpecific = r.ProviderSpecific
endpoints = append(endpoints, txt)
if filter(txt) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's basically if true as far as I understand https://github.com/kubernetes-sigs/external-dns/pull/5459/files#r2107584832 , right?
If so please drop the added complexity. If we need filtering than have it but other than that I don't think we should have an if branch here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need the if here because we use isNotManaged for filtering.
The purpose of this change is to filter out existing TXT records when an A record was mistakenly deleted and later recreated by External-DNS.
This filtering is necessary because trying to create a TXT record that already exists would result in an error. So if a TXT record already exists, we avoid creating it again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also addresses the comment raised here: #5459 (comment).

@@ -259,14 +318,18 @@ func (im *TXTRegistry) generateTXTRecord(r *endpoint.Endpoint) []*endpoint.Endpo
txtNew.WithSetIdentifier(r.SetIdentifier)
txtNew.Labels[endpoint.OwnedRecordLabelKey] = r.DNSName
txtNew.ProviderSpecific = r.ProviderSpecific
endpoints = append(endpoints, txtNew)
if filter(txtNew) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see hear 🙇 #5459 (comment)

@@ -279,7 +342,7 @@ func (im *TXTRegistry) ApplyChanges(ctx context.Context, changes *plan.Changes)
}
r.Labels[endpoint.OwnerLabelKey] = im.ownerID

filteredChanges.Create = append(filteredChanges.Create, im.generateTXTRecord(r)...)
filteredChanges.Create = append(filteredChanges.Create, im.generateTXTRecordWithFilter(r, im.existingTXTs.isNotManaged)...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if this is correct

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see hear 🙇 #5459 (comment)

Copy link
Contributor

@szuecs szuecs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit of cleanup and missing negative tests

For sure we do not want to recreate ownership records.

newEndpointWithOwner("record-1.test-zone.example.org", "1.1.1.1", endpoint.RecordTypeA, ownerId),
},
},
// TODO: Test TXT record regeneration when only A/CNAME records exist.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No!

expectedCreate: []*endpoint.Endpoint{
newEndpointWithOwner("record-1.test-zone.example.org", "1.1.1.1", endpoint.RecordTypeA, ownerId),
},
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are tests missing for:

  • do not recreate $RR, if another external-dns instance owns the record
  • do not recreate $RR, if there is no owner

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify in what situation you think this case would occur? 🙇
In this PR, the tests are intended to verify that records (such as A or CNAME) are properly created when they are desired (e.g. via Ingress host settings).
At this point, the records do not yet exist, so naturally they will have no owner at that stage.
In that sense, the "no owner" case is simply the expected initial state, not an exceptional condition we should test for.

Regarding ownership conflicts: I believe this would happen if multiple external-dns instances were deployed in the same cluster and watching the same resources (e.g. the same Ingress), without proper isolation (such as domainFilter).
However, in my view, this is a misconfiguration, and not something we should test for here.

@u-kai
Copy link
Contributor Author

u-kai commented Jun 13, 2025

@szuecs
Thank you for review!

We don't want to recreate TXT records in this case!
It's the mechanism that identifies the ownership and if you mix a zone with multiple external-dns or external-dns and manual records you would create a mess if you would recreate the deleted txt record. A txt record deletion means, that you don't want to let this record being owned by the external-dns instance.

You're right — there are cases where deleting the TXT record is intentional to relinquish ownership.
In that case, recreating it would indeed be incorrect.
However, if the goal is to stop managing certain records, wouldn't it be more appropriate to configure domainFilter or similar mechanisms? What do you think about that approach?

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 25, 2025
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 27, 2025
Copy link
Contributor

@ivankatliarchuk ivankatliarchuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest seems legit to me

We just w8 for @szuecs view

}
}

func (im *existingTXTs) add(r *endpoint.Endpoint) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

function not fully covered

Screenshot 2025-07-03 at 08 01 10

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But agree, we need to be super careful, as pretty much every deployment rely on txt registry

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from ivankatliarchuk. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

External DNS controller does not sync records as expected
5 participants