Full-text search in all sources or all diffs

[LinqLover]

To support scenarios like "I think I had used this selector once in the past but don't know when and where". Could be limited to the currently selected branch, a range of commits, or more advanced filters and predicates. Sorting by ancestry should be fine enough. For large repositories, streaming of results could be nice. Optionally, support wildcards (match:) and regexes as well. Maybe draw some inspiration from the Squeak Inbox Talk UI. :-)

[MariusDoe]

For an initial prototype, see 22287b9 (and 4ca936b), which implement streamed searching, with multiple matching strategies and sources, and predicates via arbitrary code execution. Still to do:

• search parsed sources / diffs
• better status updates
  • I think the loading animation looks nice, but it might get annoying after a while and might even be problematic for people who prefer reduced motion
  • currently the status is binary (searching or not), my idea would be to show something like searched X of Y commits where both X and Y increase over time, probably at the top of the list
  • is the number of results at the end interesting? if so, the status at the top of the list could simply stay, otherwise it would disappear once the search is finished
• maybe some shortcuts - filtering by time stamp currently requires you to type commit committer timeStamp and then a condition based on that
• some help text to explain at least the code mode, my idea would be to integrate that into the Settings menu (a help menu item at the top)
• add option to include reflogs to search all reachable commits in the repository (by git's own definition)

[LinqLover]

Wow, that's a neat feature and a cool loading animation! 🤩

What I'd love to see on top most would be:

1. Reveal search hits in text pane: Especially when I search for text in files/diffs or in long commit messages, knowing that this commit contains the searched selector or whatever anywhere is only half the way to finding it somewhere in the code. If you want to implement this you could draw inspiration from the help browser in Squeak or from Squeak Inbox Talk (see readme screenshot, I personally think that I also found a nice architecture for this with class TalkTextSearcher[Match] - not yet having looked at your design :-)). For predicate search, this would be harder or impossible of course, maybe let's only support search results for text search right now.

2. Easy way to search diffs: E.g., for "when was this method added/removed/touched the last time?". As far as I understand, I can currently only search entire files or write my own predicate (for which I yet have to figure out how I would read out all the diffs)?

maybe some shortcuts - filtering by time stamp currently requires you to type commit committer timeStamp and then a condition based on that

+1

searched X of Y commits

You could also try to display the fraction into the loading animation (like a simple progress bar), maybe that'd be more subtle than a new line of text anywhere

is the number of results at the end interesting? if so, the status at the top of the list could simply stay, otherwise it would disappear once the search is finished

Perhaps, but where do I find the status at the top of the list? While search is in progress the list simply starts with the first found commit for me. (In theory you could also display the progress in the window label)

Long list of usability ideas with mostly minor priority:

• The search settings menu is a real expert interface :D Maybe you/we could identify a relevant subset of mechanisms that serve 90% of the use cases and provide it in a simpler interface with <= 3 options? (Technically, the clear button would be optional because you could just delete the filter again when you are done)
  • nit: Cmd + L does not what I would expect it to do (should probably do the same as "clear"?)
  • Also some of the options are mutually exclusive to each other, aren't they? E.g. all the string/regex options don't apply when one selects code? I wonder how this could be made more obvious ...
• Arbitrary predicate search is so cool but also has a learning curve for the interface of the domain objects. Autocompletion support is great for that! Maybe provide some examples for search queries anywhere (a help button/item in the menu, maybe insert a default query when switching to code search [for the first time], maybe sth else)?
  • Did I mention I love the Autocompletion support? If you wanted to make it even better, you could try to restrict the proposed variables and selectors to those relevant for search (I don't know wheter commitList or compiledPatterns are relevant (if yes, for what?), and you most likely shouldn't use setters such as commitList: or commit initialize and plenty more during search). Theoretically you could narrow down the completions like we did in AssemblerECModel
• I was a bit surprised at the beginning that Enter won't start the search, but for longer predicates this option makes sense of course. In web interfaces, the seemingly single-line interface would make me assume that Shift + Enter adds a new line. Then again, the current one might be more idiomatic/expected in Squeak. Maybe ignore this one. :-)
  • But another idea: Have you tried out search-as-you-type, without ever needing to press Enter? Especially for string-based search, this would be much faster as I don't need to type more than necessary until I see the result. Also for predicate search, this would give you the earliest possible feedback.

    • Just prototyped it and liked it. For predicates or regexes, you would need to catch syntax errors and display them non-modally. In SIT, I did something similar by displaying the error messages in the search results pane (could be the commit preview box here) and highlighting the erroneous position in the query:

      

Looking forward to seeing this evolve further! 🚀

[MariusDoe]

Thanks for the extensive feedback! I'm glad you like where this is going :D

1. Good idea. I think I can implement something similar to TalkTextSearcher[Match].
2. Yes. In addition to the parsed sources / parsed diffs, it's probably a good idea to also support searching in raw diffs. I think the easiest solution would be to do something similar to what git log -S does - look for a difference in (number of) occurrences between the two versions. I can probably also extend that to actually not only look at the difference in number of occurrences, but actually a difference in occurences themselves (e. g. moving a string within the same file would not be found by git (I believe), but I think I can also catch that).

You could also try to display the fraction into the loading animation

Yes, I thought about that, but couldn't come up with a good location - the background of the search bar would be nice (filling it from left to right), but I'll have to see whether that looks good. Also, what might be confusing for users, is that the fraction might actually sometimes go down, I'll explain why: As it is currently, I cannot actually show any progress, because I simultaneously look for new commits and immediately filter them, so I don't know how many commits are left. But how I imagined this would be to launch a second process that just looks for commits and let the original process only look for matches in those. The problem with that is that the numbers of commits processed by those two processes can grow independently, which could mean that the quotient of the numbers (your fraction) does not actually grow monotonously. I'm not sure how likely this scenario is, but it is the reason why I originally only wanted to show searched X of Y commits. I could probably extend that string to something like searched X of Y commits (P%), found Z matches (Q%) - that way, the user sees how Y grows, too, and thus the potentially weird changes in P aren't as confusing.

Perhaps, but where do I find the status at the top of the list?

Nowhere yet :D I meant the hypothetical status item described at the end of the previous paragraph :)

In theory you could also display the progress in the window label

Interesting idea, I can experiment with what looks better.

The search settings menu is a real expert interface :D Maybe you/we could identify a relevant subset of mechanisms that serve 90% of the use cases and provide it in a simpler interface with <= 3 options?

Hm, I agree that the menu is quite large at the moment. Maybe we can also shrink it by moving the checkbox groups into submenus? There is already a notion of sections in the menu, as shown by the dividers.

Technically, the clear button would be optional because you could just delete the filter again when you are done

Yes, but don't you think it's more convenient to press a single button than to select the whole text, delete it and (at least currently) press Ctrl+s?

nit: Cmd + L does not what I would expect it to do (should probably do the same as "clear"?)

Interesting observation. I think if you stick to the behavior in other text editors, it should probably not always clear, but revert to the last executed search expression, right? That could be the empty string, but doesn't have to be.

I think this currently doesn't work because I also save the expression in editText:, not only in setText:. I do that because I want the user to switch between modes and keep the current expression (if they start typing code, then notice that the mode is still set to pattern). I can probably do that without breaking Ctrl+l though (introduce a new instVar that saves the editText and use that when switching modes).

Also some of the options are mutually exclusive to each other, aren't they? E.g. all the string/regex options don't apply when one selects code? I wonder how this could be made more obvious ...

Yesn't. They were, but I added self pattern: '...', which you might not have seen yet. That runs the pattern mode on the provided string and returns whether there are matches. Crucially, the same settings from the menu apply to that method as well and that is intentional: My idea was that you can start searching in the pattern mode, then want to add a predicate, e. g. a timeStamp constraint, and you can easily do that with the pattern: method. Again, I should document that :D. Writing this out actually gave me an idea: when switching from pattern to code mode, I could convert the pattern into a self pattern: '...' expression, that would probably save a lot of time and make the method even more discoverable. Although I should probably add some checks so see whether the user actually intended to write code in the first place, as in the 'start typing code, then notice pattern mode is still turned on' scenario.

Maybe provide some examples for search queries anywhere

Yes, this what I meant by some help text to explain at least the code mode, my idea would be to integrate that into the Settings menu (a help menu item at the top) :P

If you wanted to make it even better, you could try to restrict the proposed variables and selectors to those relevant for search (I don't know wheter commitList or compiledPatterns are relevant (if yes, for what?)

Great idea! And thanks for the implementation hint. The two instVars you mentioned should indeed not be accessed by users.

Have you tried out search-as-you-type, without ever needing to press Enter?

I have not, but I thought about it. I'll have to check the performance, but I would guess that it's alright.

For predicates or regexes, you would need to catch syntax errors and display them non-modally.

This is the reason I have not yet implemented it. The non-modal option is a good idea, I'll have to see where the best location for the error display is. Although I think I might keep displaying the errors modally when the user actually presses Ctrl+s. As for highlighting the problematic part: This is probably only useful/necessary for regexes, right? Because code is already highlighted as code and should give you enough visual information to find the problem.

[LinqLover]

Hey, sorry for the late reply, I was on a short vacation and did not read my emails 🙈

2. Yes. In addition to the parsed sources / parsed diffs, it's probably a good idea to also support searching in raw diffs. I think the easiest solution would be to do something similar to what git log -S does - look for a difference in (number of) occurrences between the two versions. I can probably also extend that to actually not only look at the difference in number of occurrences, but actually a difference in occurences themselves (e. g. moving a string within the same file would not be found by git (I believe), but I think I can also catch that).

With diffs, do you mean hunks (like git diff -U0)? ChatGPT tells me that git log -G"regex" will scan patch contents (other than -S) so this should detect moves as well. Maybe support both modes until we find out which of them are more relevant in practice?

You could also try to display the fraction into the loading animation

Yes, I thought about that, but couldn't come up with a good location - the background of the search bar would be nice (filling it from left to right), but I'll have to see whether that looks good. Also, what might be confusing for users, is that the fraction might actually sometimes go down, I'll explain why: As it is currently, I cannot actually show any progress, because I simultaneously look for new commits and immediately filter them, so I don't know how many commits are left. But how I imagined this would be to launch a second process that just looks for commits and let the original process only look for matches in those. The problem with that is that the numbers of commits processed by those two processes can grow independently, which could mean that the quotient of the numbers (your fraction) does not actually grow monotonously. I'm not sure how likely this scenario is, but it is the reason why I originally only wanted to show searched X of Y commits. I could probably extend that string to something like searched X of Y commits (P%), found Z matches (Q%) - that way, the user sees how Y grows, too, and thus the potentially weird changes in P aren't as confusing.

Thanks for the context. Is the initial commit enumeration process actually observably slow (> 0.5 seconds) in any realistic scenarios? Would it be fair enough to display an indeterminate progress until all commits have been enumerated and then switch to a fractional progress bar? One possible design would be to show a progress bar on top (or at the bottom) of the commit list during search that switches from indeterminate to determinate once all commits have been found (like here). Hypothetically, you could also use a heuristic for the number of total commits (like everything found in 1 second), and then switch to determinate, and if the total number of commits unexpectedly increases after that "snapshot", "fake" the progress fraction to avoid the percentage going back (so it grows slower than before), if that makes sense (probably not :D). From my naive perspective, Q% would most likely be much slower than P%, so I would not be too interested in P% and the second number would confuse me. Maybe you could display all the details in a tooltip?

In theory you could also display the progress in the window label

Interesting idea, I can experiment with what looks better.

Simulation Method Finder also has this (enable recursive mode to actually see it), it is probably not perfect either, a stylish progress bar would likely be nicer :)

The search settings menu is a real expert interface :D Maybe you/we could identify a relevant subset of mechanisms that serve 90% of the use cases and provide it in a simpler interface with <= 3 options?

Hm, I agree that the menu is quite large at the moment. Maybe we can also shrink it by moving the checkbox groups into submenus? There is already a notion of sections in the menu, as shown by the dividers.

I'm never sure about this. Submenus could reduce mental load but also hinder effective exploration/serendipity, and mouse interactions will be more tedious (Steering Law).

I just used the pin to keep the menu open which already makes it signicantly more pleasant to experiment with the different options. :-)

Technically, the clear button would be optional because you could just delete the filter again when you are done

Yes, but don't you think it's more convenient to press a single button than to select the whole text, delete it and (at least currently) press Ctrl+s?

Cmd + L is more convenient for me than pressing a button. But it might be all about familiarity. Is there any similar clear button in the built-in Squeak UI? :-)

nit: Cmd + L does not what I would expect it to do (should probably do the same as "clear"?)

Interesting observation. I think if you stick to the behavior in other text editors, it should probably not always clear, but revert to the last executed search expression, right? That could be the empty string, but doesn't have to be.

It depends. Cmd + L stands for canceL, so my intuition would be that it cancels the search rather than the latest edit to the search query. Then again, my intution would also be that you would not need to accept the search expression as discussed below. :-) Anyway, Cmd + L has been undoable since Squeak 6.0, I think, so deleting the entire query here would not be risky.

I think this currently doesn't work because I also save the expression in editText:, not only in setText:. I do that because I want the user to switch between modes and keep the current expression (if they start typing code, then notice that the mode is still set to pattern). I can probably do that without breaking Ctrl+l though (introduce a new instVar that saves the editText and use that when switching modes).

(Note there is also okToRevertChanges: if you need it, it seems to be often paired with editText:...)

Also some of the options are mutually exclusive to each other, aren't they? E.g. all the string/regex options don't apply when one selects code? I wonder how this could be made more obvious ...

Yesn't. They were, but I added self pattern: '...', which you might not have seen yet.

I assumed that would be one of the irrelevant setters :D

That runs the pattern mode on the provided string and returns whether there are matches. Crucially, the same settings from the menu apply to that method as well and that is intentional: My idea was that you can start searching in the pattern mode, then want to add a predicate, e. g. a timeStamp constraint, and you can easily do that with the pattern: method. Again, I should document that :D. Writing this out actually gave me an idea: when switching from pattern to code mode, I could convert the pattern into a self pattern: '...' expression, that would probably save a lot of time and make the method even more discoverable.

That would definitely help! Although...

Although I should probably add some checks so see whether the user actually intended to write code in the first place, as in the 'start typing code, then notice pattern mode is still turned on' scenario.

... this sounds like a possibly brittle heuristic. What if I want to search for diffs in the git/s repo where you changed examples of the search syntax? :-) Maybre relevant analogy: The same conversion could be done when switching between regex/pattern options (through escaping). However, for instance, VS Code does not (it only does that when pressing Ctrl + (Shift +) F while some text is selected). I also tried that in Squeak Inbox Talk but then found it confusing so I disabled it again (https://github.com/hpi-swa-lab/squeak-inbox-talk/blob/d311b295a8521e8f385ff5b57f946103e566f161/packages/SqueakInboxTalk.package/TalkTextSearchModel.class/instance/matchesRegex..st#L8). Most likely because filling out the search form from-left-to-right is a more common tactique than rewriting a text query to a code query, and because a bidirectional translation cannot work always.

Btw, maybe rename "code" in the menu to something else? It sounds very much like "search for code" to me. Maybe "predicate"?

Maybe provide some examples for search queries anywhere

Yes, this what I meant by some help text to explain at least the code mode, my idea would be to integrate that into the Settings menu (a help menu item at the top) :P

👍

Have you tried out search-as-you-type, without ever needing to press Enter?

I have not, but I thought about it. I'll have to check the performance, but I would guess that it's alright.

For predicates or regexes, you would need to catch syntax errors and display them non-modally.

This is the reason I have not yet implemented it. The non-modal option is a good idea, I'll have to see where the best location for the error display is. Although I think I might keep displaying the errors modally when the user actually presses Ctrl+s. As for highlighting the problematic part: This is probably only useful/necessary for regexes, right? Because code is already highlighted as code and should give you enough visual information to find the problem.

As long as the compiler and the syntax highlighter use equivalent parsers. E.g., 1 foo; bar baz is styled as valid, but cannot be compiled. But we should rather fix that. :-)