Ask an LLM to review our codebase for long-term maintenance issues

[elmisback]

This might give us a more objective perspective on what we are missing.

[elmisback]

Step 1 here will be to make a plan!

• Maybe go through file by file? Or show the whole codebase as one large message/set of messages? Then ask particular questions

  • What are good questions to ask? What are our goals? (How fragile is it, What will maintenance burden be)
  • What kind of context is the model missing/do we not document?
  • We also should think about how an agent with limited memory/time (a real human) would approach the codebase (looking at the readme, what will a reader not know/find?) (Can we provide instructions for a human to navigate the codebase using a model to mitigate this risk?)

• We can think about making this a general tool/script for looking at a repo, e.g. a small website that points at a github repo and does some analysis

• Our edge is that we have a project we care about getting this right for!

[benwang33]

No progress, but I'm thinking about this

[benwang33]

Began making prompts to GPT-4o

Feeding in key files (readme and core files) a few at a time.

Asking LLM to answer these guiding questions:

• What does this file/module do?
• Are there any parts that are hard to understand or undocumented?
• Are there clear entry points and logical structure?
• What improvements or refactorings do you suggest?

[elmisback]

Let's use a much better LLM than 4o, preferably o3/o4-mini-high, Claude 3.7 with thinking, Gemini 2.5 Pro, or maybe R1 or Qwen3-235B. Use them through openrouter (https://openrouter.ai/qwen/qwen3-235b-a22b/providers) and use a cheap endpoint that is hosted in the US or DE and doesn't collect submitted data. The total cost should be under about a cent per query, especially for R1/Qwen3-235B. I'll reimburse you as long as you keep query costs under $5 lol, lmk if you need more than that Gemini 2.5 will probably do the best right now according to benchmarks, see eg picture 2 at https://www.reddit.com/r/LocalLLaMA/comments/1ka6mic/qwen_3/

…

On Wed, Apr 30, 2025, 19:42 Ben Wang ***@***.***> wrote: *benwang33* left a comment (herbie-fp/odyssey#246) <#246 (comment)> Began making prompts to GPT-4o Feeding in key files (readme and core files) a few at a time. Asking LLM to answer these guiding questions: - What does this file/module do? - Are there any parts that are hard to understand or undocumented? - Are there clear entry points and logical structure? - What improvements or refactorings do you suggest? — Reply to this email directly, view it on GitHub <#246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB56HM7SMFNZ3A26636QVYD24GC2TAVCNFSM6AAAAABYDISZFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBTHE2TMNJZG4> . You are receiving this because you were assigned.Message ID: ***@***.***>

[elmisback]

Also, report your results! What did it do when you submitted those queries? You should be able to just share the results as a link here.

[elmisback]

Please add "Are there documentation files which cannot be found through the main README? Are there documentation files which cannot be found from the directory where they are most relevant?

[elmisback]

Please provide a status update for this issue!

[benwang33]

Setup openrouter and credits. No meaningful progress yet.

[elmisback]

We're dropping this and focusing on other issues for the release