Improve default separators / Being able to modify them

[aidanhs]

Description (@gmourier)

• Adding user most desired separators as default to the tokenizer. The & character for example.
• Being able to specify / modify the separators according to a use-case. Some users might consider any separator as part of a specific business nomenclature thus searchable as part of a word. e.g. node.js <- the . is part of a word.

---

Initial comment (@aidanhs)

Describe the bug
Searching for 'hi' in a document set with 'hi' and '&hi' only returns the one result.

To Reproduce

$ curl 'http://127.0.0.1:7700/indexes/myindex/documents' --data '[{"id":0,"text":"hi"},{"id":1,"text":"&hi"}]'
{"updateId":0}
$ curl 'http://127.0.0.1:7700/indexes/myindex/search' --data '{ "q": "hi" }'
{"hits":[{"id":0,"text":"hi"}],"offset":0,"limit":20,"nbHits":1,"exhaustiveNbHits":false,"processingTimeMs":1,"query":"hi"}

Expected behavior
I expect both results to be returned

MeiliSearch version: v0.20.0

[aidanhs]

The same applies for []<>.

I guess this relates to https://docs.rs/unicode-segmentation/1.8.0/unicode_segmentation/trait.UnicodeSegmentation.html#tymethod.split_word_bounds which the tokeniser uses.

I'm do understand <>, but & seems like an odd choice to not have as a word boundary in the unicode annex http://www.unicode.org/reports/tr29/#Word_Boundaries

[aidanhs]

(I'm happy for this to be closed and re-raised as a feature suggestion rather than a bug if you prefer)

[gmourier]

@aidanhs I've made this issue a product discussion. I added a Description part to the first message of this discussion but I've kept your initial comment.

I'm testing this format, it may change in the future 🤓

[aidanhs]

Thanks!

[curquiza]

Hello @aidanhs!

This is indeed an expected behavior with the current search engine. & is not a separator (like . or !) but a character (like b or c).
So, when you type &hi, the search engine will only retrieve documents containing words starting with &hi.

This is not a bug, but indeed a feature request 🙂
@gmourier do you want to move it on your side? Not only the & is concerned but also <, > and any other characters we could consider as a separator!

Thanks a lot for your feedback @aidanhs!!

[gmourier]

Thanks, @aidanhs and @curqui!

There are two things that seem important here.

• Adding user most desired separators as default to the tokenizer. The & character for example.
• Being able to specify / modify the separators according to a use-case. Some users might consider any separator as part of a specific business nomenclature thus searchable as part of a word. e.g. node.js <- the . is part of a word.

[warmwhisky]

Does anyone know how I can customize the separators? I have meilisearch 0.27.1 installed on Centos from source.
I am using meilisearch-php 0.23

Most of my pages are based on guitar tunings e.g. EADGBE, but a lot of tunings contain "sharps" in them like this D#D#C#C#G#G# so when they search with sharps the actual search is D D C C G G this causes large response times.

Another problem I have is that I have millions of pages and bots like Google and Bing are on my site constantly. Many of the pages trigger Meili just by visiting a page where videos relevant to guitar tunings are showed. So when I get 60000 hits from google in one day my site grinds to a halt until the bot realises they cannot get a decent response and then slow down their indexing. See my current Google crawl stats for the last

Some pages have a tunings like this G#D#G#D#G#C#D#G#C# and they also have G#D#G#D#G#C#D#G#C# in the name. Related videos are displayed using the tunings and the name so sometime the search term will be
G#D#G#D#G#C#D#G#C# G#D#G#D#G#C#D#G#C# which I believe turns into G D G D G C D G C G D G D G C D G C This is where my site really cannot deliver even a page load without idle timeouts from meilisearch

Another problem I have is that I have millions of pages and bots like Google and Bing are on my site constantly. Many of the pages trigger Meili just by visiting a page where videos relevant to guitar tunings are showed. So when I get 60000 hits from google in one day my site grinds to a halt until the bot realises they cannot get a decent response and then slow down their indexing. See my current Google crawl stats for the last

The orange line is response time from my server to Google. Where you see the response jump, around 25th September, is when I went live with Meilisearch.

[gmourier]

Hi @warmwhisky, sorry to be late I've noted to answer you here

We had a quick discussion with @Kerollmops and @ManyTheFish to evaluate your issue and see if it could help to customize the separators.

Changing the separators will unfortunately not help you to improve the performance; I was wrong.

We know, however, that being able to customize the separators will essentially help improve relevancy for any language's "business" in your example, music notation!

What I would advise you to do is to reduce the searchableAttributes as much as possible to the ones that are the most critical for your use case; this should help to improve the search speed. Finally, I know it's not ideal, but I think caching the results could greatly help you if you don't need fresh data.

I am sorry for the back and forth.

@ManyTheFish, @Kerollmops, anyone, If you have a work-around that could help, it would be awesome.

Thanks!

[gmourier]

I may have an idea, but I'm unsure if it will help.

Did you tried to replace the # with a non-separator char in your dataset and then transform a search query containing every # to the corresponding char present in the dataset before hitting Meilisearch? It sounds super hacky, but you would completely erase the separators from the equation IMO.

Normalization may be a blocker here 💭 é would become e I think; so it will be misinterpreted with a E chord. Choosing a char that doesn't collide with a chord could works. e.g, Z/X etc.

[ManyTheFish]

Hello @gmourier @warmwhisky,

1. The first thing to try in order to reduce the query time is to remove the ranking rule Attributes from your settings, it's probably the one that takes time in your search queries.

2. If it's not sufficient, the best workaround I can suggest is to remove # from your documents before inserting them into Meilisearch, for instance D#D#C#C#G#G# should be replaced by DDCCGG, the drawback of this workaround is forcing you to search DDCCGG instead of D#D#C#C#G#G# or D D C C G G to find your document.

3. Replacing the character is another solution, but I didn't find any spoofing character that is not considered a separator by Meilisearch, Moreover, it would force you to replace for every search query all the # with the chosen spoofing character to make it work.

On my side, I will think of replacing the Latin segmenter with something else, unfortunately, I don't know the drawbacks of removing the # from the separators and it won't be a quickfix. 🤔

[gmourier]

Thanks, @ManyTheFish! Unfortunately, I think that 1. doesn't help because by doing this, @warmwhisky will lose the notion of note alterations like flats and sharps.

[warmwhisky]

Hello @gmourier @ManyTheFish

Thanks so much for the suggestions and your time on this issue I have.

I will try No.1 first. Though I do not understand the implications mentioned about it by @gmourier. I will have to test it out and see the difference in results and performance

1. The first thing to try in order to reduce the query time is to remove the ranking rule Attributes from your settings, it's probably the one that takes time in your search queries.

No.2 would also match tunings where there is no # e.g. D#G#C#F#A#D# and DGCFAD are completely different tunings with hundreds of different pages related to each.

2. If it's not sufficient, the best workaround I can suggest is to remove # from your documents before inserting them into Meilisearch, for instance D#D#C#C#G#G# should be replaced by DDCCGG, the drawback of this workaround is forcing you to search DDCCGG instead of D#D#C#C#G#G# or D D C C G G to find your document.

No.3 sounds more doable from my end. Though I am unsure how to edit the results as they get indexed to Meilisearch. Perhaps this can be done with meiliesearch-php?

3. Replacing the character is another solution, but I didn't find any spoofing character that is not considered a separator by Meilisearch, Moreover, it would force you to replace for every search query all the # with the chosen spoofing character to make it work.

I use tunings in URLs but I replace the # with an 's' as in sharp. so D#D#C#C#G#G# becomes `dsdscscsgsgs'. I can imagine how I could use a special character like '¬'.
I did see a list of separators in your docs, there was not too many chars. maybe 15 or 20.
Do you have an exhaustive list of separators?

[ManyTheFish]

Hello @warmwhisky, the segmentation is handled by unicode-segmentation. This library is the one that choose how do we split the words

[warmwhisky]

Hello @gmourier @ManyTheFish

I have managed to reduce my load time from Load: 22.21 s to Load: 905 ms which is a fantastic improvement.

I managed to use the search and replace suggestion with Laravel Scout. You can see my solution on this comment meilisearch/meilisearch-php#411 (comment)

Thanks for all of your suggestions!

[gmourier]

Hi @warmwhisky 👋

Thanks for your feedback, I'm glad to see that the workaround is helping you.

We'll work on natively improving it all in the following months.

[macraig]

Hello everyone 👋

We just released a 🧪 prototype that allows customizing tokenization and we'd love your feedback.

How to get the prototype?

Using docker, use the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-tokenizer-customization-2

From source, compile Meilisearch on the prototype-tokenizer-customization-2 tag

How to use the prototype?

You can find all the details in the PR.

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing this new feature!

You can test it by using:

• The release assets
• The Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.4.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on September 25th, 2023

⚠️ RC (release candidates) are not recommended for production

[macraig]

Hey folks 👋

v1.4.0 has been released! 🦓 You can now customize tokenization by adding or removing tokens from the separator tokens and non-separator tokens lists. ✨

Note:

📚 Separator tokens
📚 Non-separator tokens

[AnirudhaGohokar]

great work! its too difficult to configure these things in elastic search.

[warmwhisky]

Hello, I have finally migrated my server so I can move away from 0.27.2.

Upon removing my custom search queries which stopped the search using # for musical searches I am finding that search is not slow atall. I am working in dev so maybe that is why?

My question is should I be using a custom Separator Token instead of the default #? and of so is there any direction or docs I can follow to do this?

Edit:
I think I have this sorted

 // set sharp/#/flat/♭ as non separator token
$client->index('guitar_tunings')->updateNonSeparatorTokens(['#','♭']);

There is an improvement in performance. Instead of "D#D#C#C#G#G#" returning 1000 results it returns only the exact matches. It is so perfect... Thank you!