Skip to content

Conversation

@notlesh
Copy link
Contributor

@notlesh notlesh commented Aug 5, 2025

This PR is meant to extend the prdoc analysis so that an LLM can do deeper analysis on it.

Roadmap:

  • Save prdocs to disk instead of feeding all of it immediately to LLM
  • Add basic stats/indexing of content
  • Make initial pass at all docs to summarize them
  • Create prompt template to help deeper analysis of content
  • Detect when multiple consecutive releases are involved and process them all
  • Improve prompt:
    • Need better context about relay chain vs parachain (and which is relevant)
    • Needs to better understand what things would actually affect the current codebase
    • Should not need to directly invoke the release analysis prompt
    • Give better direction on distinctions of crate functionality (client vs runtime, primitives, storage, events, etc.)
  • Fetch other context (GH labels, diffs) as needed (or just let the LLM do this?)

@notlesh notlesh marked this pull request as ready for review August 11, 2025 21:51
Copy link
Collaborator

@elfedy elfedy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! some overall pending things to discuss:

  1. rpc_endpoints stuff is still there along with some unsolved merge conflicts (seems there was something weird with the history because it is reintroducing validation logic we removed)
  2. Imo prodoc_analysis_prompts should be part of the prompts module.
  3. Added comments on prdoc tool naming and maybe using a templating engine

.unwrap_or_else(|| "default".to_string())
}

pub async fn fetch_release_prdocs(release: &str) -> Result<PrdocsResult> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we should call this differently given it also gets the github lablels as well.

@notlesh notlesh merged commit 3ab12fd into main Aug 14, 2025
3 checks passed
@notlesh notlesh deleted the notlesh/release-analysis-continuation branch August 14, 2025 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants