LANraragi v.0.8.8 - I'm Deranged

[Difegue]

Happy Holidays from your favorite degeneracy library!
I am bringing you the long-awaited gift of indexes to cap off the year.

🛑 Important changes

If you don't want to read the rest of the changelog:

• If you're running from source, LRR now requires two supplementary Redis databases, using indexes 2 and 3 by default. You can change those numbers in lrr.conf.
• If you update a large instance, search will not work for a bit while the indexes are built -- You can follow the status of the stat_indexes Minion job to see when you can start searching again.
• The /api/archives/random endpoint has slightly changed, which means third-party clients might have broken random functionality until you update.
• OPDS-PSE support is in, but won't work for archives that don't use JPEG for their images.

🔍 Search has been rewritten (#555, #444, #563, #657 and more!)

Search has always been the biggest performance bottleneck of the entire server due to using a rather dumb algorithm that always did multiple database passes...
Through the incredible invention of search indexes ^{but for redis}, I'm happy to announce that search is now up to 10 times faster for the most used scenarios.

088search_Trim.mp4

Some notes on the search rewrite:

• Spaces are no longer a delimiter for different tags in a search prompt, commas should now be used instead. This matches what the autocompletion box does and how metadata as a whole is entered into the app, so I don't think this will cause too much pain.
• Search now requires the index tables to be built (does so on app start, and then updates them as tags/titles/etc are modified), so large instances might take a few mins to get a usable search again after updating.
• Title searches will work the same as before (quotes or $ for an exact match, ? * _ % as wildcards. )
• For tags, if you don't use exact matches, tag searches will be wildcarded on both ends (like previous behavior), namespace:tag searches will only be wildcarded at the end (namespace stays intact).
  You can use *namespace:tag if you want the old behavior for some weird reason.

This was one of the biggest jobs on the list, and while it took a few days to get it right I hope you'll enjoy it -- I do think it's quite a gamechanger.
This rewrite doesn't optimize everything (large intersections and sorting by anything else than title will still be a bit slow), so there's certainly still room for improvement.

The new system makes heavy use of Redis sets and sorted sets, which are all lodged in a separate, third database.
See the DB Architecture document. (the config keys moving to a fourth database is a pending change that will land in a future release.)

📚 OPDS PSE Compliance

I've dusted off the OPDS API to make it use the new Search, and took the opportunity to implement a few niceties.

LANraragi now supports OPDS PSE 1.1, which means you can use compatible OPDS clients to read archives page-by-page from the server and update your reading progression. (If server-side progress tracking is enabled)
As a known caveat, OPDS-PSE will currently not work if your archive contains non-JPEG image files.

Considering the surprisingly complete state of third-party LRR clients, I don't think many people use OPDS at all, but it's nice to be a good citizen to standards nonetheless.
And if you can't sideload DuReader, that means you can now use Panels on iOS! At least until Apple finally caves in and allows sideloading on their devices. ^{god bless the EU}

Other changes

• (Remember archive sort key and order on library page #646) New EH layout compatibility

  🐼 Both ex and e-h have recently updated their layout and search API - While the old plugin still works fine on the new layouts, it still got a small update to benefit from the changes.
  🔍 You can now search by gID -- this will fall back to title name if no gId found or search failed. (Thanks @polak14 !)
  🍪 The igneous cookie is now supported, which fixes the problem of using ex in non European and American regions. (Thanks @zizzdog !)

• (Please add a grammar to search archives untagged #709) Carousel now auto-reloads with every new search + has a nicer empty view

  🎠 whee
  🆕 I couldn't really afford to make the carousel auto-reload before since it relies on Search, which had...yeah.

• Add basic Redis password support

  💍 You can now set redis_password in lrr.conf in case your Redis server is password-protected.

• Windows fixes

  🪟 The WSL detection should now work regardless of your system language.
  🛠️ The Windows app should now detect "file not found" WSL failures better.
  🔧 The repair button is also now always available from Settings.

• Miscellaneous bugfixes and improvements

  💎 Login now redirects you to the page you tried to access beforehand
  💎 Update FontAwesome to v6
  💎 Update Mojolicious to v9.30
  💎 (EH plugin can not connect to ex gallery #396) Allow most common http methods when CORS is enabled
  💎 Fix not being able to redownload an URL after having deleted it from the server without a restart
  💎 ?key= can be used again to authenticate API requests. (Mostly just for OPDS, please don't actually use this)

Full Changelog: v.0.8.7...v.0.8.8

Inspired by his and Eno’s trip to the Gugging Psychiatric Clinic in 1994, [...] “Deranged” seems mainly Eno’s work, though one ancestor was Bowie and Nile Rodgers’ “Real Cool World” (there’s also an echo of “Billie Jean” in its opening four-note synth hook)

---

This discussion was created from the release LANraragi v.0.8.8 - I'm Deranged.

[polak14]

One of the best updates yet, new search is really fast!
Would be nice if we can also search for f:twintails instead of female:twintails and so on, other than that its flawless so far for me.
It also barely took 40 seconds to build the index for a 30k archive on my server.