8.0.0 Release - Valentine's Day Edition 💘

8.0.0 -- 2024-02-14

It's Valentine's Day 💘, so it seems appropriate to release the next major version of Quepid, the tool to give your queries some ❤️.

Exciting News

Judge Judy is now appearing on the Quepid channel. It is now possible to create one or more LLM-based AI judges in Quepid! Set your OpenAI key, add your examples or customizations to the judgement prompt, push the big button, and let the magic box do the rest.

Continuous Experimentation is now a reality in Quepid. You can now schedule automatic evaluation runs on a case on a daily basis. It is now possible to see the most up to date performance results, and track changes over time without loading up Quepid! You can even configure alerts to be emailed to you if your Case score drops!

Scaling Up. Quepid now supports many thousands of Queries in a single Case! Our target is 5000 queries in a single Case, and by adopting background processing and async feedback in the user interface, we can now scale up to handle this. Oh, and query performance has been drastically improved. One long time user with 1000 queries reports that loading the case has dropped from 20 minutes to three minutes ;-). Cases with a large number of queries can have problems completing, due to timeouts or 429 (too busy, try again) errors. Thanks to @mkr the concurrency control for handling query execution has been updated to prevent Quepid hanging up forever waiting for a query to complete.

Expected Reciprocal Rank (ERR) is now available as a community scorer. This scorer is appropriate for multi-valued judgment ranges when evaluating search journeys like known item search.

New User Manual. We've embraced the Diátaxis framework for writing documentation, and consolidated into a single platform the mix of documentation that has been published in various locations.

Bugs Addressed

If you don't have P@10 defined, the migrations blow up. Thanks @frutik for finding the bug. #1093

Basic auth that includes '%' in username or password breaks reloading from http to https. Thanks @brucks24 for #1139

Emoji filled queries like kfc 🍟➕🍔➕🍗 now can be stored in Quepid! Thanks @shuttie for reporting this bug. #1046

Other Changes

• Simpler Deployment. We've updated to Rails 8, and no longer need Redis. We're back to just a server and a MySQL Database ;-).
• Nicer import book page. Gave it some UX love. Kudos to @wrigleyDan
• We use the built into Rails health check end point at /healthcheck. Please check your setup.
• Change tracking of scores to record which scorer produced the score. This is a breaking change, as previous versions don't know what scorer generated a score. For historical scores, the current case scorer is assumed to be the only scorer used on that case. Soon you will be able to run multiple scores and pick one as your "North Star" metric.
• We thought having really fine grained permissions would be important to Quepid, but what we've seen is that it doesn't matter. Indeed, if anything, the permissions structure is confusing to users. Additionally, we've mostly cared "are you an Admin?" to decide if certain things can be done or not. So it's gone.
• We care about privacy. We've dropped Google Analytics in favour of our own first party analytics. We've also dropped the Intercom integration.

What's Changed

• undo the "fit" that made us only run a single test by @epugh in #1076
• Updates oct 2024 by @epugh in #1071
• Add check for the existence of scorer before updating. by @epugh in #1093
• Migration to Rails 8 by @epugh in #1084
• Rails 8 is out, and we drag along a swap from annotate to annotaterb by @epugh in #1098
• Strip out SASS by @epugh in #1113
• swap to released version of Bullet by @epugh in #1102
• Tweaks to text. by @epugh in #1119
• Hard-limit search concurrency to 10 by @mkr in #1103
• Restore proper support for notifying users about Cookies being required policy by @epugh in #1127
• Update jshint structure by @epugh in #1135
• Be smart about if we have no score yet, and share that info. by @epugh in #1132
• ship a first cut of being smarter by @epugh in #1133
• Frogs forever by @epugh in #1136
• Change the close icon to something that renders larger and more visible by @epugh in #1140
• Warn on incompatible characters in basicauth password by @epugh in #1143
• Wire in deleting the case when you abandon it. by @epugh in #1144
• First party analytics by @epugh in #1100
• Fix up the routing logic to keep you in the rating experience. by @epugh in #1145
• Perf optimizations. (again!) by @epugh in #1148
• On to 3.3.6 of Ruby by @epugh in #1157
• Add err scorer by @david-fisher in #1162
• Backout counter_cache by @epugh in #1165
• Review database structures by @epugh in #1167
• Improve links/buttons when creating a Book by @wrigleyDan in #1175
• Meet Judge Judy, she is your AI powered SME by @epugh in #985
• Missing indexes on Search Endpoints by @epugh in #1173
• Spike 5000 queries round deux by @epugh in #1039
• Remove permissions architecture in favour of roles by @epugh in #1185
• Restore the proper encoding, lost somewhere by @epugh in #1188
• Look at database consistency issues by @epugh in #1189
• Testing the flow of everything.... by @epugh in #1192
• fix RRE export format by @epugh in #929
• Properly nest returning judgements for a book under the book route by @epugh in #1187
• OpenAPI query limits being hit by @epugh in #1194
• Can we use Blazer to monitor users? by @epugh in #1205
• FInally remove intercom by @epugh in #1206
• Validate search endpoints by @epugh in #1209
• Simplify book ux by @epugh in #1200
• Need to track which scorer provided the score! by @epugh in #1210
• Manage all Case scores, and let you delete the werid scores. by @epugh in #1213
• Fancier handling of scores and ratings and judgements and fix search endpionts and all the other things that make it hard for @david-fisher to write changelog notes. by @epugh in #1215
• nicer filtering makes loading books faster... by @epugh in #1216
• Document why updating a rating doesn't update the specifics in a query_doc_pair by @epugh in #1219
• Remove owner concept by @wrigleyDan in #1181
• Strip out remaining references to team ownership. by @epugh in #1221
• Testing of Quepid8 Functions by @epugh in #1222
• Revamp Query.query_text to handle emoji better by @epugh in #1223

Full Changelog: v7.18.0...v8.0.0

v7.18.1

7.18.1 - 2024-12-06

We've been hard at work on Quepid8, however we have two items that demand a point release:

• Bug fix when looking up potential Querqy rule use and we hadn't finished running the query. cb9e3f4 by @epugh.

• Prevent run away querying! A user of Quepid has a proxy between Quepid and the search engine, and we were overloading it. 7cb1f55 by @mkr!

Full Changelog: v7.18.0...v7.18.1

7.18.0

7.18.0 - 2024-10-04

Wow! It's been three months since the last release of Quepid, so it's about time.
This release brings Quepid up to Rails 7.2 standard, which gets us ready for some of the interesting things in Rails 8, specifically simplifying what you need to run Quepid.
There is a good path to removing the dependency on Redis, and maybe even MySQL for small setups!

Improvements

• Shrink the Quepid Docker image by 1 GB. This was accomplished by the migration to Rails 7.2 for Quepid in #1058, fixing #1004 by @epugh.

• Expanded Quepid's API for working with Search Endpoints. Thanks @clintbxtreme for opening #1056, and this was fixed in #1057 by @epugh.

• Using SMUI and want to see if your rule was triggered? Now you can, we have a Querqy icon! Previously you would have to dig into the query parsing explanation to know. #1050 by @clamar-VM was fixed by #1053 by @epugh.

Bugs

• When rating images that have a prefix defined, the prefix was being lost. Thanks @clamar-VM for finding this, fixed in @#1054 by @epugh.

• We got too strict with our cookies, and Quepid quit working with Keycloak. Thanks @drieschel for #1049.

What's Changed

• So long docker-compose. Welcome docker compose. by @epugh in #1052
• Fix login and redirect problem with Keycloak by @drieschel in #1049
• introduce a icon letting you know querqy rule matched a query by @epugh in #1053
• adding the prefix when setting up links by @epugh in #1054
• Update nginx Docker tag to v1.27.1 by @renovate in #1045
• Update dependency terser to v1.2.3 by @renovate in #1025
• Extend search endpoints controller to allow delete, complete update, … by @epugh in #1057
• Big combo! Rail 7.2 Dockerfile used for Dockerfile.prod, plus bump R… by @epugh in #1058

New Contributors

• @drieschel made their first contribution in #1049

Full Changelog: v7.17.1...v7.18.0

7.17.1

7.17.1 - 2024-06-26

Lots of small things ;-).

Improvements

• Proxied connections can now handle non ASCII characters like café. #1043 by @epugh. Thanks @miguelmesas for opening up the ticket that tipped me off to this.

• p@10 scorer now is smarter to handle non binary judgements. #991 by @david-fisher. Sorry this took so long to get merged!

• RAG style queries are MUCH longer. @miguelmesas opened up which is fixed by #1042 and #1037 by @epugh.

• Improve the bulk query API to support 1000's of queries. #1037 by @epugh.

• Small tweaks to the layout of the Search Endpoints to move the Destroy action away from the edit action. Thanks @OkkeKlein for reporting #900.

Bugs

• Resetting skipped and can't judge query/doc pairs is fixed. Thanks @pfries for opening #1040. Quite a few other Turbo Drive related navigation issues fixed.

7.17.0 - Breaking the tyranny of the web request/response cycle

7.17. - 2024-06-14

This is a big one! We have broken the tyranny of the web request/response lifecycle by embracing some powerful Rails technologies:

• ActionCable: We now support two way communication from the backend server to your front end browser via Web Sockets. This lets us notify not just the person who starts a long running process of updates, but also any OTHER user who is interested in a specific topic. For example, if someone uploads a Book with many thousands of judgements, then everyone who is part of that team can see the progress for them selves.

• ActiveJob: This let's us run background jobs. Historically we only used it for tracking of events (like you visited a Case), however now we are using it for processing large data sets in the background.

• ActiveStorage: We need to better handle LARGE datasets in Quepid. ActiveStorage lets us build massive export files and store them in the database as a binary until you decide to download the data. Likewise, importing large data files that take time to process is now supported by being able to store them in Quepid.

• Hotwire: HTML over the Wire lets us render HTML on the backend using our familiar Rails MVC, but magically stream the updates to the front end. This lets us add interactivity without requiring us to build out a Single Page Javascript application.

These changes are going to enable a future for Quepid that let's it scale up signficantly and be a richer environment for measuring relevance and working with search!

Okay, now on to our more detailed list of changes:

Features

• You can now run your Query Evaluations OUTSIDE of Quepid, and store the scores INSIDE Quepid. This gives you a place to share this information with folks in your team. See the example Python scripting here: https://github.com/o19s/quepid/tree/main/docs/examples/external_eval. We also enhanced the API documentation. #1034 by @epugh.

• HTML Over the Wire! Now we only load HTML that has changed, which improves page percieved performance. Added new export and import of Books as background jobs with notifications to users over Web Sockets. You then can download the resulting file without waiting for the data processing. Loading aspects of the home page asyncrhonsly so the page loads super fast, and then content renders in as available. #992 by @epugh.

Improvements

• Update to Rails 7.1 standards. #1033 by @epugh.

• Nicer Admin Homepage visual layout. #1031 by @epugh.

• Revamped performance of the Home page to not recalculate the case summaries if you've already cached them in the browser. Much faster page! #1022 and #1029 by @epugh.

• New Case Wizard is smarter about existing Search Endpoints pane. #1012 by @epugh.

• Lots of updates to all the dependencies!

Bugs

• Swapping between HTTP and HTTPS end points would cause issues with your Search Endpoint configuration. #1035 by @epugh fixes #1005.

• We state that judging a query doc pair only requires three jdugements, but didn't actually make that work. #1032 by @epugh and @wrigleyDan fixes #1019 by @wrigleyDan.

• Fixed utf encoding in mysql for query_doc_pairs. #1017 by @epugh fixes #1013 by @wrigleyDan.

• Pass basic auth credentials through when you view a specific document or use Quepid as a proxy to your search engine. #1015 by @epugh fixes #1014 by @david-fisher.

v7.16.1

7.16.1 - 2024-04-09

Bugs

• You can delete a user who has existing judgements, and then that messes up the main Books page because their judgements hang around. Now you are warned about this, and you can choose to make their judgements anonymous. (or just Lock their user account). #999 by @epugh.

Full Changelog: v7.16.0...v7.16.1

7.16.0

7.16.0 - 2024-03-28

Features

• Now have a dropdown for Books similar to Cases that leverages our existing tracking of what books you have viewed. #986 by @epugh.

• We now run scorers over queries that have no ratings and queries that are ZSR. This lets us have smarter logic about how you want to score those situations. #993 by @epugh.

Improvements

• One less query to determine what a user has access to via being an owner or a member of a team while ensuring no duplicate cases/books show up. #982 by @epugh with input from @reid-rigo.

• Remove extranous faraday logs when running tests. #983 by @epugh with input form @reid-rigo fixes #964.

• Nicer onboarding using TMDB dataset for Algolia users. #987 by @chuckmeyer.

Bugs

• Book Importing was broken. Plus now we have nicer formatted error message when you validate a book to import. #989 by @epugh.

Full Changelog: v7.15.1...v7.16.0

v7.15.1

We found a very annoying bug in 7.15.0, so a quick 7.15.1 release was needed!

7.15.1 - 2024-03-13

Bugs

• Looking up Cases for a user is returning duplicates due to missing "distinct" keyword on SQL Query. #980 by @epugh.

• The "bundle exec thor case:create" is NOW updated to handle SearchEndpoints. #981 by @epugh.

7.15.0 - 2024-03-11

Features

• Now track an Explanation for a judgement. If a human clicks "I can't rate this" then they are prompted for an optional explanation why. In the near future, LLM powered judgements will communicate a explanation as well! #978 by @epugh.

• Algolia integration! We have a wonderful contribution from @sumitsarkar that adds Algolia as one of the supported Search engines. #933 by @sumitsarker.

Improvements

• Move to using Thor (an existing CLI tool) for loading all our sample data files, and less custom Rake code. #966 by @epugh.

• Better UX when working with custom search API's and the mapper code. #965 and #963 by @epugh, fixing #959, #960, #961 by @wrigleyDan and #950 by @erikhatcher.

• Proxied Search API requests should follow Redirects. #962 by @epugh fixes #958 by @wrigleyDan.

• Improved performance of checking case access permissions. #955 by @reid-rigo. Thanks Reid!

• Updated the Jaccard, RBO, and Judge comparison notebooks that ship with Quepid. Introduced more robust testing for them. #956 by @epugh.

• Improve the Books API. #951 by @epugh.

• Snapshots used to have "Best Snapshot" option, but it hasn't worked in years, so stripped it out. #948 by @epugh.

Bugs

• Scoring logic was all somewhat broken. Add a query, it doesn't rescore. Delete a query, it doesn't rescore. Have a ZSR query, it impacts scoring in a "wonky" way. Then, when you introduce snapshot comparisons, it all gets weirder. We've gone through at a pretty line by line debugger level and fixed the various interaction bugs. #975 by @epugh.

• Only ship Web Open Font fonts, which simplifies our CSS and set up and eliminates a warning in Firefox console. #techdebt. #974 by @epugh.

• Fixed some bit rot in handling of Tries in the web UI, and do some polish of UX. #967 by @epugh.

• Check length of query_text. #952 by @epugh and @mkr.

• No email provider? No problem! Fixed issue with inviting folks and no email provider setup. #949 by @epugh.

## 7.14.0 - 2024-02-06

7.14.0 - 2024-02-06

Features

• SCALING SCALING SCALING. Now possible, though painful still, to run 5000 queries by moving lots of activities to background processing. Thanks to my colleagues on the Cville Slack communities #ruby channel for lots of advice and help. #916 by @epugh.

• Quepid truncates the document fields in the core display, either via using highlighting to pluck out a section, or just snippeting to 200 characters. However some days you just want to see the entire field! Now you can control this in your field specifciation by prepending the field name with unabridged: tag. #946 by @epugh.

• Need to communicate with your users about a new Scorer or upcoming downtime? You can now post Annoucements to your users. #911 by @mkr and #941 by @epugh fixes #848 for @flaxsearch.

Improvements

• Snapshots require you to be able to "lookup" the document by id from the search engine. However some search engines like Vectara and the Search API don't support this. Reworked how snapshots work to require you to save the document fields at snapshot creation time in order to power the snapshot comparison UI. This may end up being spread to other search engines as well. #945 and #931 by @epugh.

• The logs in Quepid are being overwhelmed by the size of the documents. #942 by @reid-rigo cleans this up. Thank you Reid!

• Stripped out Threshold based alerting on queries, the community never used them. #927 by @epugh.

• Now tracking when a Book was viewed by a user, which powers the dashboard. #925 by @epugh.

• Removed the QUEPID_PROPHET_ANALYTICS environment variable, as Prophet was never a performance issue. #924 by @epugh.

• Dropped old database columns in the tries table that had moved to search_endpoints table. #930 by @epugh.

Bugs

• Restore the thor commands for creating an admin user. #922 by @epugh fixes #888 by @vincetrumental.

• Bug fix for static search endpoint when Quepid is running in a nested host context. #917 by @epugh.

• We were not export notes and options for a query. #926 by @epugh fixes #923 by @peterdm.

Plus we updated to the latest of Ruby and Ruby gems, Mysql, various NPM packages.

v7.13.0

7.13.0 - 2024-01-03

Just in time for the New Year!

A number of changes all wrapped up in one large PR #906 by @epugh. This one is for you @david-fisher ;-).

• Now when you judge a query doc pair for a Judgement Book we update in the background any associated Cases that have that query. This eliminates the need to manually refresh the ratings for a Case from the Judgement Book! Thanks @david-fisher for the suggestion in #829.

• Provide an option to allow queries that exist on a Book to be created in the Case if they are missing. The default behavior is to NOT create queries in a Case from the Book. Thanks to @jvia for this suggestion. #871.

• Band-aid the awkwardness that Cases have Queries/Ratings while Books have QueryDocPairs + Judgements, and that there are attributes on a Query like information need, notes, or options that don't exist on a QueryDocPair. So now we preserve them, which facilitates round tripping better. Upload a Book? Upload a Case? It doesn't matter ;-) Long term we need to merge these two data structures...

• When Books were first added to Quepid, they didn't follow the same permissions/sharing structure that Scores and Cases did. Now you can share a Book with multiple Teams! Fixes #907

• And as an aside, anyone can edit a custom scorer, not just the Owner of it. #762 suggested by @david-fisher.