Skip to content

feat: natural language search #955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Animesh404
Copy link
Member

@Animesh404 Animesh404 commented Jun 7, 2025

What kind of change does this PR introduce?

Feature #951

Summary
![Screenshot 2025-06-07 1848
Screenshot 2025-06-07 184848
14](https
Screenshot 2025-06-07 184838
://github.com/user-attachments/assets/99763645-3a59-47b8-9f2b-3b69bde5f19b)

This PR introduces a natural language command interface for OpenAdapt, allowing users to find and start workflows using simple text queries (e.g., "calculate my expenses") instead of manually searching through a list of recordings. The primary motivation is to make the application more intuitive and accessible, especially for users with many recordings.

The implementation includes:

  • UI for Natural Language Input: A new "💬 What do you want help with today?" option in the system tray opens a dialog for users to enter their query.
  • Enhanced Similarity Search: A new get_enhanced_similarity_search function was implemented. It improves search relevance by creating a dynamic context for each recording based on its window titles, action events (like typing), and any transcribed audio, in addition to its title.
  • Database and Embedding Logic: The database schema was updated to store vector embeddings for recordings, and a backfill script (openadapt/scripts/backfill_embeddings.py) was created to process existing recordings.

Checklist

  • My code follows the style guidelines of OpenAdapt
  • I have performed a self-review of my code
  • If applicable, I have added tests to prove my fix is functional/effective
  • I have linted my code locally prior to submission
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • New and existing unit tests pass locally with my changes

How can your code be run and tested?

  1. First, ensure you have embeddings for your existing recordings by running the backfill script:
    python -m openadapt.scripts.backfill_embeddings
  2. Start the OpenAdapt application:
    python -m openadapt.app.tray
  3. Click the system tray icon and select "💬 What do you want help with today?"
  4. In the input dialog, type a query that relates to one of your recorded workflows. For example, if you have a recording of a calculation, you could type calculate 2+9+11.
  5. A "Choose Your Workflow" dialog will appear, displaying a list of relevant recordings ranked by similarity.
  6. Select a workflow and click "OK" to proceed to the replay configuration screen, where your original search query will be pre-filled.

Other information
This implementation uses the sentence-transformers/all-MiniLM-L6-v2 model for generating embeddings, which is now a dependency. The search logic is designed to be self-contained within Python, avoiding the need for external vector database services.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant