Add annotation tool to highlight changed elements in preview#565
Add annotation tool to highlight changed elements in preview#565
Conversation
Adds a new `annotate` tool that the model calls after completing all code edits to highlight what was changed. Each annotation includes a label and a one-sentence description. The system prompt instructs the model to call it exactly once before returning its final response. Backend: tool schema, runtime handler, and input summarizer. Frontend: amber-themed display in AgentActivity with BsTag icon. https://claude.ai/code/session_01S3R2LH589n6nrSAJkfeDg8
Replace the label field with a CSS selector so annotations can target actual DOM elements. The PreviewComponent now injects a pulsing amber outline into the iframe for each matched selector after the iframe loads. PreviewPane extracts annotations from completed annotate tool events and passes them down. System prompt and tool description updated to instruct the model to provide CSS selectors. https://claude.ai/code/session_01S3R2LH589n6nrSAJkfeDg8
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0f6f61705
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| selector = ensure_str(annotation.get("selector")) | ||
| description = ensure_str(annotation.get("description")) |
There was a problem hiding this comment.
The new annotate runtime only validates that annotations is a non-empty list, then immediately calls annotation.get(...) on each item. If the model emits malformed JSON like annotations: ["#hero"] or [null], this throws an AttributeError instead of returning a structured tool error, which can abort the tool-execution turn rather than producing a recoverable failure result.
Useful? React with 👍 / 👎.
| if (!currentCommit) return []; | ||
| const variant = currentCommit.variants[currentCommit.selectedVariantIndex]; | ||
| const events = variant?.agentEvents || []; | ||
| for (const event of events) { |
There was a problem hiding this comment.
This extraction returns as soon as it sees the first completed annotate event, but agent events are appended in chronological order (newest at the end), so multiple annotate calls in one variant (e.g., retries) will display stale highlights from the oldest event. The preview should read the most recent completed annotation payload instead.
Useful? React with 👍 / 👎.
2 participants
Summary
This PR adds a new
annotatetool that allows the agent to highlight and describe changed elements in the preview pane. When the agent completes code edits, it can call the annotate tool to visually mark which elements were modified and provide descriptions of the changes.Key Changes
Backend - Tool Definition & Runtime
annotatetool definition inbackend/agent/tools/definitions.pywith schema for CSS selectors and change descriptions_annotate()method inbackend/agent/tools/runtime.pyto validate and process annotation databackend/agent/tools/summaries.pyto display annotations in activity logsFrontend - Preview Component
Annotationinterface toPreviewComponent.tsxwithselectoranddescriptionfieldsPreviewPane.tsxto extract and pass them from completed agent eventsFrontend - Activity Display
AgentActivity.tsxwith:Implementation Details
annotatetool event in the current varianthttps://claude.ai/code/session_01S3R2LH589n6nrSAJkfeDg8