Suggest Edits response can provide an <old_text> value that is not unique resulting wrong replacements

[stippi]

Summary

Suggest Edits applies edits to additional, undesired, locations if the text to replace is not unique

Steps to trigger the problem:

1. Let the Assistant suggest edits.
2. It might create a patch that includes a search string (where to apply the patch) that is not unique.

Actual Behavior:

The Edit is applied at the first match of the search string.

Expected Behavior:

The LLM is given feedback that it needs to adjust the search string in order to make it unique.

See the screen shot below for a demonstration of the problem. The patch was applied to the first match, where it made no sense. I inspected the patch that the LLM provided and found the issue was multiple matches.

Zed Version and System Specs

Zed: v0.172.10 (Zed)
OS: macOS 15.3.0
Memory: 36 GiB
Architecture: aarch64

[probably-neb]

Hmm, I'm not sure outright failure is the right solution here. But we can definitely consider ways to improve how the identification of what to replace happens to fix issues like this one.

For now I think the workaround is just to review the changes before applying them.

Thanks for reporting!

[stippi]

It does not result in duplicate replacements. It simply replaces the first match. Not sure how you could automatically detect which of the replacements the LLM actually "meant". Failure might be the only option. But it doesn't have to be a failure that the user becomes aware of, just respond to the LLM with the error and let it fix itself by retrying the tool invocation.

[notpeter]

While we are working towards improving the Assistant panel and associated prompting and capabilities, we currently do not support Agentic feedback loops where we are able to evaluate the response from Claude, and then automatically trigger follow-up requests/responses for refining the patches via subsequent calls. We are working to re-architect the Assistant panel (Assistant2) to better support some of these use cases, but in the meantime the user will have to manually reprompt re-prompt / follow-up if the model is providing ambiguous edit suggestions.

You can see the prompt we use for getting Anthropic to generate <old_text>, you might try experimenting with expanding / augmenting that to see if you have better results (e.g. "ensure any <old_text> matches one and only one location" or some such).

zed/assets/prompts/suggest_edits.hbs

Lines 15 to 17 in 22e2b8e

	- <old_text> (optional) - An excerpt from the file's current contents that uniquely
	identifies a range within the file where the edit should occur. Required for all operations
	except `create`.

Of note, your custom Prompts can also significantly alter the behavior and performance of the model. For multiple weeks I was seeing much worse results than my coworkers in the Assistant panel, turns out I had accidentally set my default prompt to include an incomplete/ambiguous sentence fragment which diverted model attention and conflicted with the built-in prompts.

See more info response to your other issue here:

• Suggest Edits: Claude keeps forgetting to add <operation> tags #24651 (comment)

The complexity / fickleness / inconsistency of these models is truly staggering.
Thanks for reporting.

[stippi]

Thanks for the feedback. I still think not checking for exactly one match and simply applying to the first occurrence is wrong (a bug). Also, given that the error cannot currently be communicated to the LLM, the UI should at least make it clear that some edits have errors and cannot be applied. I appreciate how hard it is to make this more robust, the failure rate just feels too high at the moment.

I am working on a coding assistant agent myself and meanwhile I am on the third attempt/approach to getting edits work more reliably. My goal is to eventually contribute to Zed's Assistant feature. I'm not completely sure if this is wanted at the moment. At least I tried to get more insights into what the plans are (opening a discussion that was closed). It would be great to know what the midterm goals are and the plans to get there.