Improve Meilisearch Index upgrade workflow for Ulmo (reindex_studio)

[bradenmacdonald]

Currently the reindex_studio command is required before you can use any of Studio's search features. In addition, because the index format has changed between Teak and Ulmo, it will be necessary for operators to rebuild the search index during the upgrade to Ulmo.

1. The reindex_studio command currently requires --experimental. Now that it's been used in two full releases (Sumac & Teak), the experimental flag should be removed.

2. The reindex_studio command can take a long time to run (hours on large instances), so Tutor should not necessarily run a full synchronous index rebuild during its upgrade script. See this bug report for example.

   • the simplest approach is just to run reindex_studio --initial during the upgrade, (i.e. create a blank index with the new format - search results will be empty but the search code will work without error), and tell the user to run reindex_studio manually to fill in the content (which will be fast if they have little content, or could take ~ a day on very large instances).
   • a more sophisticated approach could be to check if the instance has < 50 courses + libraries and in that case run the full reindex_studio to completion, but if it has more, then just run reindex_studio --initial and tell the user to complete it on their own time.
   • an even better approach would be to just create the blank index (reindex_studio --initial) during init/upgrade and then kick off an async celery task to populate it incrementally in the background.

3. The reindex_studio command has two modes: "normal" (create a new index, populate it, then cut over to it once it's fully populated. If interrupted, must start over again.) or "incremental" (start or continue populating the current index. If interrupted, can be resumed anytime). The "normal" mode should be removed and the "incremental" mode should be the only way of populating the index. This does mean that there could be a period of time during upgrades when Studio search results are incomplete, but it's worth it for simplicity. Also the incremental mode indexes newest content first, so should quickly fill in the content that users most care about.

4. In coordination with Tutor, we should upgrade to the latest version of Meilisearch - especially if features like "Reduce indexing memory usage", "update documents with function", and contains filters are stable because those will be valuable features for us. (Not to mention High Availability, if/when it's implemented.)

   • Note that upgrades to Meilisearch 1.13+ are much easier than prior versions (prior versions required backup/restore, new versions just require an upgrade command like Mongo).

---

In the future we should also think about how to consolidate our search index management so that one command manages all the search indexes, rather than studio and LMS being separate. (Not to mention teams, forum, etc.). But this is blocked until we know if we're going to be supporting TypeSense or any other alternative to Meilisearch or not.

[bradenmacdonald]

@ormsbee @kdmccormick @jmakowski1123 This is a (hopefully small and easy) backend story related to Studio search that I think is important to get done before Ulmo as well.

[regisb]

Have you considered moving the reindex_studio to a database migration script? I mean: we should keep the manage.py management command for manually running that step when necessary, but if the reindex command should be run at precise moments in time, then we would be better off making that command part of the ./manage.py cms migrate workflow.

My reasoning is as follows: as a Tutor maintainer, if I need to run reindex_studio --initial at some moments, but not other, then I need to keep track of whether that command was ever run before. Something like the following:

1. Was the index ever created? If not, then run reindex_studio --initial. If yes...
2. ... does the index have the latest structure? If not, then run reindex_studio --initial.

As a Tutor maintainer I would be much better off if the meilisearch migrations were integrated with other regular migrations. What do you think?

[bradenmacdonald]

I definitely considered that and in many ways I would prefer that as well, but the problem is that (depending on how we implement it), it creates the potential for "migrations" that either (1) take a very long time to run, or (2) erase your entire search index and require you to manually "re-index" stuff later. And neither of these are expected outcomes of running the migrate command.

If there is some way that our migration could kick off an async celery task to re-populate the index after it is modified to the new format, then that could work. But again I don't think people expect migrate to launch celery tasks, so that also seems unexpected.

Open to ideas or things that I may have missed.

[bradenmacdonald]

Oh right, there is also the challenge that changing the Meilisearch index configuration depends on Meilisearch being enabled or not. And in general I think migrations that depend on the state of the instance's config at the time the migration was run can be problematic. (e.g. imagine if someone has the Meilisearch setting off, runs migrations, then turns the setting on - now their index doesn't exist but all migrations are "already applied")

So while I like the simplicity of setting the index config during a migration, I would personally be more comfortable doing that only if we made Meilisearch a dependency of the overall platform and knew that everyone would be using it; if we're going to be supporting "Meilisearch or Elasticsearch" or "Meilisearch or TypeSense" or "no search features" as options, then I think a migration is less appropriate.

[regisb]

I partly agree with your reasons not to include meilisearch migrations in the migrate command. I suggest the following:

1. Perform index structure migrations in model migrations: these migrations should only concern the index structure, and not the index content, such that migrations don't take a very long time to run. (is this possible at all?)
2. Move these migrations to a "meilisearch" Django application, which is added to INSTALLED_APPS only if meilisearch is enabled. Thus, migrations are run only if the feature is enabled.
3. Allow the reindex_studio command to index course content asynchronously (by triggering celery tasks), and only for courses that really need it. (i.e: for courses that don't have any content, if that makes sense?)

I'm not sure if my suggestions make sense, as I didn't look at the detailed implementation of reindex_studio.

[bradenmacdonald]

@regisb Good ideas. I think we can find something like that that will work.

Perform index structure migrations in model migrations: these migrations should only concern the index structure, and not the index content, such that migrations don't take a very long time to run. (is this possible at all?)

It's sort of possible. We can definitely create and update the (empty) index structure with migrations, but modifying the configuration of an index that's already full of data can be very slow (according to the docs). So in the case where the index already exists and is populated with data and we need to change its configuration, it's best to wipe it out and create a new empty index with the right config. (Otherwise the migration could take a long time, causing the same problem we're trying to solve here.) That said, I have never actually tested how long this takes, especially compared to re-indexing all the content.

Move these migrations to a "meilisearch" Django application, which is added to INSTALLED_APPS only if meilisearch is enabled. Thus, migrations are run only if the feature is enabled.

Oh, that should have been obvious to me. That makes a lot of sense.

Allow the reindex_studio command to index course content asynchronously (by triggering celery tasks), and only for courses that really need it. (i.e: for courses that don't have any content, if that makes sense?)

I think we still have to do all courses, in principle. But we do start with the most recent courses first, to reduce the downtime. People are much more likely to care if a newly created course is temporarily missing from the search index compared to one that's ten years old.