verdi llm cli, AiiDA users benefit from LLM; a proof of concept.

[khsrali]

For now, it only works with Groq, because it provides free API.
More backends will be added later.

First, configure for once:
verdi llm configure --backend groq --api-key <API-KEY> -- to generate a key see here

Now you can ask whatever you want, the limit is your imagination 🤞
$ verdi llm cli 'list all processes'

It will then, respond with a generated command and asks for a confirmation (e) before executing it.

$ verdi llm cli 'list all processes'
Generated command: `verdi process list`

Execute[e], Modify[m], or Cancel[c]:: e
$ verdi process list
Command output:

  PK  Created    Process label             ♻    Process State    Process status
----  ---------  ------------------------  ---  ---------------  -----------------------------------------------------------------------------------
7064  77D ago    ArithmeticAddCalculation       ⏵ Waiting        Waiting for transport task: stash

Alternatively the user has the option (m) to edit the command before executing, to fix mistakes:

$ verdi llm cli 'list all processes'
Generated command: `verdi process list`

Execute[e], Modify[m], or Cancel[c]:: m
Please modify the command:: `verdi group list` # my edit
$ verdi group list # my edit
Command output:

  PK  Created    Process label             ♻    Process State    Process status
----  ---------  ------------------------  ---  ---------------  -----------------------------------------------------------------------------------
7064  77D ago    ArithmeticAddCalculation       ⏵ Waiting        Waiting for transport task: stash

Groq is not trained on AiiDA so the outcome is not gonna be magnificent. This PR provide an interface, for future implementation of a trained model. See project: PLACEHOLER

An ideal model after training will be able to answer to such commands:

$ verdi llm cli 'delete are groups with label qe*'
Generated command: `verdi group delete --startswith qe`

OR

$ verdi llm cli 'list all processes I've created since 5 min ago'
Generated command: `verdi process list -p 1 -a | grep -E  '1m|2m|3m|4m|5m'`

The latest one is a nice example. Although the only supported flag for timely listing is --past-days; such a smart interface can first, find out there's no official command to support that, and second, would be able to suggest a workaround.

Although the interface is not finalized, early reviews and discussions are welcome.

UPDATE:

RAG added. Now it's able to answer to use more flags to response precisely:

$verdi llm cli 'list my latest process that finished, show only one result'
Generated command: `verdi process list -O id -D desc -S finished -l 1`

[codecov]

Codecov Report

Attention: Patch coverage is 27.10280% with 78 lines in your changes missing coverage. Please review.

Project coverage is 78.17%. Comparing base (48bd2ac) to head (d7cb3df).

Files with missing lines	Patch %	Lines
src/aiida/cmdline/commands/cmd_smart/cmd_smart.py	26.14%	65 Missing ⚠️
...rc/aiida/cmdline/commands/cmd_smart/llm_backend.py	23.53%	13 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6834      +/-   ##
==========================================
- Coverage   78.29%   78.17%   -0.12%     
==========================================
  Files         566      569       +3     
  Lines       42764    42871     +107     
==========================================
+ Hits        33479    33508      +29     
- Misses       9285     9363      +78     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[edan-bainglass]

Proposed names:

• verdi sebastiaan
• verdi seb
• verdi huber

[khsrali]

@giovannipizzi
How about verdi chat?

[giovannipizzi]

verdi chatbot? Makes more the sense it's a bot replying

[khsrali]

Also verdi chatbot is also nice, only the command gets a bit lengthier though.
So one has to type more to get there.. in a sense 🙃

[agoscinski]

I think it is better to create the verdi_cli.json when the command is first time invoked instead so we don't blow up the src code (for review it is great to see it). Then you could cache the embedding (maybe repository)?

---

Why don't you include the description of the options of the commands in the verdi_cli.json? For example, for verdi process kill --help:

-a, --all                       Kill all processes if no specific processes
                                  are specified.

Seems like useful information.

---

Maybe we instead of letting the user to create the embedding, we upload them for each aiida version so the user never needs any llm package, the user just needs to download it from an external resource. That would make the user's life easier. It behaves then more like RTD.

[agoscinski]

should try to detect gpu, if not then go to cpu

[agoscinski]

This needs to be an optional dependencies! Or maybe we can completely remove it (see third point of my comment)