Add a Term Cleanup Feature

[dkotter]

Description of the Change

This PR adds in a new Feature, Term Cleanup, that uses the power of embeddings, to compare all terms against each other and then recommending similar terms that could be merged together if desired. This comparison process can be run entirely in WordPress but once you reach around 500 terms, performance really starts to slow down. Similar to the Smart 404 Feature, support for Elasticsearch (using ElasticPress) has been added to significantly speed things up.

This Feature requires two services in order to function:

1. A Provider that can generate embeddings (right now we support OpenAI and Azure OpenAI)
2. A data storage engine that can also run vector queries (currently supports doing this in WordPress only or using Elasticsearch powered by ElasticPress)

Once the Feature is setup, you can start the comparison process which will send all terms off to the embeddings Provider to generate vector data. If configured, this data is then sent to Elasticsearch and vector queries are run there. If not configured, this data remains in WordPress only and comparisons are done there using that data.

Once the process is complete, you'll be given the opportunity to see all the results and decide which terms you want to merge together (or which terms you want to skip). This process can be re-run whenever needed (i.e. as new terms are added).

Closes #795

How to test the Change

1. Enable the Feature with at least one taxonomy turned on
2. Go to the Term Cleanup page and start the process for whatever taxonomy you want
3. Ensure the process completes and results show
4. Ensure you can merge and skip terms
5. If desired, set up Elasticsearch/ElasticPress and test using that

Changelog Entry

Added - New Feature, Term Cleanup, which provides the ability to compare all terms from specific taxonomies and showing recommendations for similar terms that can be merged together

Credits

Props @dkotter, @iamdharmesh, @berkod

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added Critical Flows, Test Cases, and/or End-to-End Tests to cover my change.
• All new and existing tests pass.

[dkotter]

@iamdharmesh I believe I have this feature fully moved over to ClassifAI but would appreciate if you could take a look since you're most familiar with how this functions. I moved the background processing to use Action Scheduler since we are already using that in ClassifAI but I believe everything else has been left the same.

Note @jeffpaul mentioned there are some UI changes we're hoping to make so we can either leave this PR until those are ready or can merge this in (once it looks good) and open a new PR for those changes, but leaving this in draft for now

[iamdharmesh]

@dkotter Thanks for working on this. The PR looks great. I’ve fixed a few issues related to the in-progress job status, canceling the running process, and similarity score for database-based comparison. Please take a quick look at those additional changes when you have time. Otherwise, everything looks good.

[jeffpaul]

I would prefer to get a quick UX review to see if there are some quick iterations to improve here, but if we're otherwise near ready to ship a major release then I'm fine merging this in to ride along there

[iamdharmesh]

@jeffpaul @dkotter I worked on UX items from https://github.com/10up/classifai-cleanup/issues/7, https://github.com/10up/classifai-cleanup/issues/20, https://github.com/10up/classifai-cleanup/issues/19 and https://github.com/10up/classifai-cleanup/issues/18 and listed status on each below. Some of these need discussion.

• Right now we show a Count field, which represents how many times this term has been used. I know this is common terminology in WordPress so maybe is fine but I think there could easily be some confusion here thinking this is the count of how many matching terms were found. Could maybe update this to Used: 621 times to be more clear?
• Often terms don't have a description associated with them. I think we should remove this field if it's empty, rather than showing the Description: label and never having a description shown.
• I'm assuming when we display results we have whatever score was calculated to determine the match. Could we show that score in these results? That way you know how high of a match it actually was? Something like Score: 79% or Match: 80%
• Since we only have a single action (Skip) we could probably change the name of that column to Action or just remove that label all together
• I have some confusion in regards to the merge buttons. At a glance, hard to know what each button does; which term is getting merged? What happens to the other? I think we can make this more clear by either updating the button text or adding some helper text (Need help with finalizing a better wording copy here, if we want to update this.)
• More personal opinion but I think we could benefit from a slight bump across the board with font sizes. Things seem fairly small to me at the moment
• In addition to the above, could also maybe use a slight bump in padding between things
• UX - Adding a link to the posts-by-tag
• UX - An option to sort the terms alphabetically (This is a quick enhancement but as we are showing all the information together in a column, we need to figure out how we can switch between sort by alphabetically and sort by count.)
• Add ability to search for specific term results

Thanks

[dkotter]

I would prefer to get a quick UX review to see if there are some quick iterations to improve here, but if we're otherwise near ready to ship a major release then I'm fine merging this in to ride along there

Things look cleaner to me from the last updates made by @iamdharmesh but I agree could use a more formal review, so I'm fine holding this until then

[dkotter]

@Sidsector9 @iamdharmesh I've updated this PR to add support for the new React settings page for this new Feature. Would appreciate a review to ensure I did this properly. I also noticed some flaky tests around the Azure Image Processing that I think I've gotten to be more reliable here.

One question I did have when working on this is do we have a way to support filters in this new React based approach? As an example, this Feature (and other Features have similar things) has a method to get the allowed taxonomies and it passes that through a PHP filter, allowing others to modify the list of taxonomies we output on the settings page. That no longer works since we're outputting this via JS so wondering if there's something I missed to still support that?

[dkotter]

One question I did have when working on this is do we have a way to support filters in this new React based approach? As an example, this Feature (and other Features have similar things) has a method to get the allowed taxonomies and it passes that through a PHP filter, allowing others to modify the list of taxonomies we output on the settings page. That no longer works since we're outputting this via JS so wondering if there's something I missed to still support that?

Bumping this question up again @Sidsector9 @iamdharmesh. Similar question for the Classification Feature. Previously we were using the get_supported_taxonomies method which would filter all taxonomies through is_taxonomy_viewable and then through our custom filter. Seems in the new approach, we not only lose the custom filter but we lose the is_taxonomy_viewable check as well.

[Sidsector9]

I'll pick this up for review and respond to the conversation first thing tomorrow.

[Sidsector9]

@dkotter I've discussed this with @iamdharmesh today, and for the first iteration, we don't have a way to filter taxonomies.

We can implement this in a JS way:

// This will allow devs to filter taxonomies.
const options = wp.hooks.applyFilters( 'classifai-filter-taxonomies', options );

Filtering values:

wp.hooks.addFilter( 'classifai-filter-taxonomies', 'Classifai', function( options ) {
    // remove taxonomies here.
    return options;
} );

@iamdharmesh what do you think about using this approach?

[Sidsector9]

@iamdharmesh there is a way to still use PHP filters, but this will require some changes to the current implementation.

• Fetch array of post types (Make this filterable via PHP).
• Each post type fetched should contain all taxonomies associated to it (Make this filterable via PHP).
• On the JS side, using the taxonomy data inside the post types, render the fields. (Note: A taxonomy can be registered for multiple post types, so while rendering make sure only unique taxonomy items are rendered.)

[iamdharmesh]

LGTM. Thanks @dkotter.

I have updated the PR to retrieve taxonomies from window.classifAISettings and made some minor updates.

[jeffpaul]

@iamdharmesh I note that there are still a couple checkboxes open in your comment farther above here... if those are still todo would you please get those into a new issue so we can track those separately?

[iamdharmesh]

Sure thing @jeffpaul. Opened #833