Add a Term Cleanup Feature

[dkotter]

Description of the Change

This PR adds in a new Feature, Term Cleanup, that uses the power of embeddings, to compare all terms against each other and then recommending similar terms that could be merged together if desired. This comparison process can be run entirely in WordPress but once you reach around 500 terms, performance really starts to slow down. Similar to the Smart 404 Feature, support for Elasticsearch (using ElasticPress) has been added to significantly speed things up.

This Feature requires two services in order to function:

1. A Provider that can generate embeddings (right now we support OpenAI and Azure OpenAI)
2. A data storage engine that can also run vector queries (currently supports doing this in WordPress only or using Elasticsearch powered by ElasticPress)

Once the Feature is setup, you can start the comparison process which will send all terms off to the embeddings Provider to generate vector data. If configured, this data is then sent to Elasticsearch and vector queries are run there. If not configured, this data remains in WordPress only and comparisons are done there using that data.

Once the process is complete, you'll be given the opportunity to see all the results and decide which terms you want to merge together (or which terms you want to skip). This process can be re-run whenever needed (i.e. as new terms are added).

Closes #795

How to test the Change

1. Enable the Feature with at least one taxonomy turned on
2. Go to the Term Cleanup page and start the process for whatever taxonomy you want
3. Ensure the process completes and results show
4. Ensure you can merge and skip terms
5. If desired, set up Elasticsearch/ElasticPress and test using that

Changelog Entry

Added - New Feature, Term Cleanup, which provides the ability to compare all terms from specific taxonomies and showing recommendations for similar terms that can be merged together

Credits

Props @dkotter, @iamdharmesh, @berkod

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added Critical Flows, Test Cases, and/or End-to-End Tests to cover my change.
• All new and existing tests pass.

[dkotter]

@iamdharmesh I believe I have this feature fully moved over to ClassifAI but would appreciate if you could take a look since you're most familiar with how this functions. I moved the background processing to use Action Scheduler since we are already using that in ClassifAI but I believe everything else has been left the same.

Note @jeffpaul mentioned there are some UI changes we're hoping to make so we can either leave this PR until those are ready or can merge this in (once it looks good) and open a new PR for those changes, but leaving this in draft for now

[iamdharmesh]

@dkotter Thanks for working on this. The PR looks great. I’ve fixed a few issues related to the in-progress job status, canceling the running process, and similarity score for database-based comparison. Please take a quick look at those additional changes when you have time. Otherwise, everything looks good.

[jeffpaul]

I would prefer to get a quick UX review to see if there are some quick iterations to improve here, but if we're otherwise near ready to ship a major release then I'm fine merging this in to ride along there

[iamdharmesh]

@jeffpaul @dkotter I worked on UX items from https://github.com/10up/classifai-cleanup/issues/7, https://github.com/10up/classifai-cleanup/issues/20, https://github.com/10up/classifai-cleanup/issues/19 and https://github.com/10up/classifai-cleanup/issues/18 and listed status on each below. Some of these need discussion.

• Right now we show a Count field, which represents how many times this term has been used. I know this is common terminology in WordPress so maybe is fine but I think there could easily be some confusion here thinking this is the count of how many matching terms were found. Could maybe update this to Used: 621 times to be more clear?
• Often terms don't have a description associated with them. I think we should remove this field if it's empty, rather than showing the Description: label and never having a description shown.
• I'm assuming when we display results we have whatever score was calculated to determine the match. Could we show that score in these results? That way you know how high of a match it actually was? Something like Score: 79% or Match: 80%
• Since we only have a single action (Skip) we could probably change the name of that column to Action or just remove that label all together
• I have some confusion in regards to the merge buttons. At a glance, hard to know what each button does; which term is getting merged? What happens to the other? I think we can make this more clear by either updating the button text or adding some helper text (Need help with finalizing a better wording copy here, if we want to update this.)
• More personal opinion but I think we could benefit from a slight bump across the board with font sizes. Things seem fairly small to me at the moment
• In addition to the above, could also maybe use a slight bump in padding between things
• UX - Adding a link to the posts-by-tag
• UX - An option to sort the terms alphabetically (This is a quick enhancement but as we are showing all the information together in a column, we need to figure out how we can switch between sort by alphabetically and sort by count.)
• Add ability to search for specific term results

Thanks

[dkotter]

I would prefer to get a quick UX review to see if there are some quick iterations to improve here, but if we're otherwise near ready to ship a major release then I'm fine merging this in to ride along there

Things look cleaner to me from the last updates made by @iamdharmesh but I agree could use a more formal review, so I'm fine holding this until then

[github-actions]

@dkotter thanks for the PR! Could you please rebase your PR on top of the latest changes in the base branch?