Strategy for updating fields that change with time

[matteocontrini]

Hello everyone,

In my use case I need to index a collection of posts of an online forum (it's about 600k posts). One of the fields I'd like to add and use for ranking is whether the author of the post is an "expert" user with good reputation, something I already have in my database.

However this reputation score (which could really be just a boolean in its simplest form) obviously changes with time. While experimenting with Meilisearch I've added this reputation score as a field of each post document, however it is my understanding that it's not possible to then update this field value in batch, so updating it would mean re-creating the whole index or updating all the posts of a user every time its "reputation" changes, which is probably quite expensive.

Is there something I'm missing? Will the future support for multi-index searches help with this?

Thank you!

P.S. I hope this is the right place to ask, I don't see a "support" category

P.P.S. the page where I'm writing this thing has a Support link in the sidebar which brings to a 404

[gmourier]

Hello @matteocontrini! 👋

Yes, this is the right place!

You are not missing something, today It may be costly if only this field is updated very often. Do you only have to index this field afterward? How often? Did you tested it to measure costs?

At some point, we thought of having fields as part of meta-data that could be quickly indexed (bypassing most of the structures used in the search). What is the purpose of this field in your use case?

Thanks!

PS: Thanks for the 404. It's now fixed 🙏

[matteocontrini]

Hi @gmourier, thanks for the quick reply.

What is the purpose of this field in your use case?

In this specific case it's used as a ranking rule, to give more priority to posts from users with a higher reputation.

Do you only have to index this field afterward? How often? Did you tested it to measure costs?

I still have to test document updates to be honest, but I assumed they are expensive. This specific field wouldn't need to be updated very often so it would probably be feasible, but it could still mean updating tens of thousands of documents, one by one I guess?

A slightly different use case would be to have shared metadata attached to a document. For example, posts are organized in threads and each post could have the thread's title stored as an attribute. If the title of the thread changes, all the posts would need to be updated and they could be thousands. If the field is not used for search/ranking I could just extract it from the database when needed but this would mean that I can't rely solely on the Meilisearch query response. 🤔

I'm thinking aloud at this point 😄

Thanks!

[gmourier]

Hay @matteocontrini 👋

Sorry for the late reply.

We could potentially reduce the cost on some data that change often like a price, a reputation, a counter, etc if they don't need to be fully indexed.

Meilisearch computes a set of data structures to make the data searchable, filterable and sortable quickly on search.

Maybe a dedicated structure for these fields would be interesting, something like a _meta object in a document. They would certainly have limitations (being used only for sorting or filtering maybe)

but it could still mean updating tens of thousands of documents

The bigger the batch the better it is. Some indexing costs will be shared among all the documents in the batch for certain data structures instead of being re-computed per indexed document.

[matteocontrini]

Thanks for the reply, no worries for the delay.

but it could still mean updating tens of thousands of documents

The bigger the batch the better it is. Some indexing costs will be shared among all the documents in the batch for certain data structures instead of being re-computed per indexed document.

What I mean is that the Meilisearch API requires to provide the full list of documents to be updated, and doesn't have an "update by query" API. This is something that Elasticsearch has for example. This would change things quite a bit, if available!

[mvaisberg]

@matteocontrini any solution for your problem?

i have a similar issue. I have 200k documents that have a "order" field that is updated in every one document each one hour, to shuffle my content. Are there a solution for that without reindex the entere collection of documents every hour?

[matteocontrini]

I don't use Meilisearch anymore so I'm not sure how much I can help. I personally switched to Elasticsearch which provides an "update by query" operation that helps with this kind of scenarios. At a first glance, it seems that Meilisearch still doesn't have such operation, so you're probably left with with reindexing (happy to be contradicted).

[gmourier]

Hey @mvaisberg and @matteocontrini 👋

We've implemented a differential indexing mechanism, and based on a test case around the use case mentioned here, we've had some significant improvements in the indexing speed.

It should be coming in v1.6!

We'll also be working on an API for updating a document-matched set from a query in the future.

[matteocontrini]

Thanks for the update!

[zgjimgjonbalaj]

Just to add another use case here.

We've also got an index with just under 100k documents (and growing rapidly). Each document contains tags, authors and category objects that can each be individually updated as well. The individual attributes themselves also contain a url and name that can change from time to time.

For now we are happy to re-index all documents and this doesn't seem to take more than 30 minutes but am wondering if there is a better way to accomplish this and minimize CPU/RAM impact?

Thanks!

[gmourier]

Hey @zgjimgjonbalaj, also pinging @matteocontrini and @mvaisberg because it may interest you!

I can see 2 solutions so far

Tip

As a general rule of thumb, don't delete/clear index documents before performing massive updates if not everything needs to be updated inside the documents set. Meilisearch leverages mechanisms to not re-index what doesn't need to be.

Solution 1: Get documents by filter and update the retrieved subset.

• Reduce the number of documents to be updated by fetching them with a filter — API docs
• Update that document subset with either AddOrUpdateDocuments or AddOrReplaceDocuments

Solution 2: Update documents by function

@Kerollmops has worked on an available prototype that allows documents to be updated by function (it supports filtering to a subset).

You can get more info about it from that public document, and we would be very happy to get your feedback and if it could help! 🙏

[zgjimgjonbalaj]

Hey @gmourier, very much appreciate you taking a look at this and for the added tips!

We are currently working on something similar to #1 but instead of querying Meilisearch we are determining what documents need to be updated on our end and performing a mass re-index on those documents. We were fortunate to have some methods in place that let us know affected/associated documents each time the targeted attributes are update. Our hope is to use the updateDocuments() method on the PHP wrapper. I believe under the hood this is using the mentioned AddOrUpdateDocuments.

I took a brief look at the Rhai function from @Kerollmops and this seems like a great idea for certain type of document updates. Looking forward to testing this locally at some point to see results. One immediate feedback would be the ease of adoption and learning curve associated with it and wonder if you have plans to include a more standard API at some point.

Thanks!

[mvaisberg]

Hey @gmourier really thanks for your support.

My case have not sense update documents because i change all of them. I have a "sort_order" number that change every hour for all documents so i have to reindex all every hour.

I will try to update the meilisearch version to check if the reindex time improve.