Experimental feature:CONTAINS and Prefix / Suffix filter operators

[gmourier]

Starting a fresh discussion for a CONTAINSoperator.

Initially expressed by

#539 (comment)
#407 (comment)

---

Comment made by @eelco-weblog.

For example, an e-commerce site wants to display Apple Airpods, but not Apple Airpods Max. Now, we can create an array of all the words in the title, save it as another property such as title_words and filter something like this:

(title_words = airpod and title_words != max)

When we can use CONTAINS / NOT CONTAINS, we do not have to create another property and filter directly on the title:

(title CONTAINS airpod and title NOT CONTAINS max)

[gmourier]

Wouldn't supporting regex expressions permits to do everything?

CONTAINS START_WITH END_WITH could be helpers creating regular expressions.

But it would also be possible to use regex syntax if needed.

In what cases could you use a regex? Is it something that seems critical?

[irevoire]

On the technical side;
Using a regex would permit you to do everything in roughly 10s per request. (obviously, it depends on the size of the dataset, but you can totally forget about the search-as-you-type experience we want to propose with a regex).
starts_with could already be implemented without any new DBs or anything added.
ends_with needs a new DB, but it's doable. It'll increase the write amplification and the indexing time, but it's nothing terrible, I think 🤔
Or if we want something closer to a regex, I think we could basically handle anything that contains at most ONE star in it;

• *apple
• apple*
• apple*max

But not;

• apple*macbook*max*
• *apple*

[Nakusu]

Any updates about that ?
We need to create a filter that use contain string operator or somthing like regex.

My use case is the following : I have a database of users with their past and current professional experiences, and I need to do a filter with it. Like someone woking on a company called %companyName%.

[nand2]

Hi @qdequele !

Thanks for your answer.
Following up on @Nakusu : what you propose will work indeed for a single word search, but won't for a multi words search; let's take the example of searching inside a job description field, let's take this example:

I worked at the truck conception service and previously before at the car safety service.

I would like to search for "truck safety" ; if we split the field by words as you suggest, in this example it will match even though it shouldn't.
What we are looking is indeed the equivalent of SQL LIKE "%truck safety%".

Thanks!

[qdequele]

Hello @nand2, I feel like your example is not really a filter. Perhaps I'm wrong. But the @Nakusu issue was about how to filter with regex when I thought your example was about searching in text. If you have to search in a text, I would just use the basic search feature of Meilisearch. If it's not enough for your use case today, could I know why?

[nand2]

@qdequele Thanks; sorry I should have said, I work on the same project as @Nakusu :)
If I expand the example: let's take this :

[
  {
    name: 'John',
    surname: 'Doe',
    general_description: 'I know stuff about truck safety etc',
    experiences: [
      {
        company: 'Volkswagen',
        description: 'I worked at the truck conception service and previously before at the car safety service.',
      },
      {
        company: 'xxx',
        description: 'xxx',
      },
    ]
  },
  {
    name: 'John2',
    surname: 'Doe2',
    general_description: 'I know stuff about truck safety etc',
    experiences: [
      {
        company: 'Volkswagen',
        description: 'I worked at the truck safety service.',
      },
      {
        company: 'xxx',
        description: 'xxx',
      },
    ]
  },
  {
    name: 'John3',
    surname: 'Doe3',
    general_description: 'I know stuff about truck safety etc',
    experiences: [
      {
        company: 'Tesla',
        description: 'I worked at the truck safety service.',
      },
      {
        company: 'xxx',
        description: 'xxx',
      },
    ]
  }
]

I would like to search for persons who worked on 'truck safety' at Volkswagen only. It should match John2 Doe2 only.
Here I cannot use the general basic search feature, because it would match all 3 because of the mention of "truck safety" on "general_description".
I would like something like (experiences.company = "Volkswagen" and experiences.description LIKE "%truck safety%")
Thanks!

[vahidalvandi]

i have same isuue please show me how i can find any company tesla ??

[Nakusu]

We had an call with their devlopper team and the contains operator will be implete in the future version, it's not available at this time

[Kerollmops]

Hello everyone 👋,

Here is the current prototype you can use to test the new CONTAINS, STARTS WITH and ENDS WITH filters to build an advanced search filter interface.

💬 Current state of that feature

The aspect we are most concerned about with this feature is search performance.

Indeed, it is possible that CONTAINS and ENDS WITH perform poorly, given your dataset. The time to find the documents matching a query with these operators increases with the total number of bytes of all the facet values of all the documents in an index.

We would like to hear your feedback on using this prototype, and we would also like your input on the following questions to help us to decideing on a go-no-go decision for a generally available status of that feature in the future:

• Have you considered using the q parameter to solve your need for a CONTAINS operator and if not why?
• Have you considered using the search for facet values feature to solve your need for CONTAINS operator and if not why?

---

How to run the prototype?

You need to start from a fresh new database (remove the previously used data.ms) and use the following Docker image:

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-contains-starts-ends-with-filters-0

How to use it?

You can use the feature this way or look at the original PR for more info.

Send some documents to Meilisearch

cat movies.json | xh 'localhost:7700/indexes/movies/documents' 'content-type:application/json'

Define the attributes you want to filter on

echo '{ "filterableAttributes": ["genres"] }' | xh PATCH 'localhost:7700/indexes/movies/settings'

Use CONTAINS

CONTAINS allows you to filter the documents for which the filtered attribute contains the given string. It's similar to the SQL LIKE operator used in that way %venture%

echo '{"q": "spiderman", "filter": "genres CONTAINS venture" }' | xh 'localhost:7700/indexes/movies/search'

Use STARTS WITH

STARTS WITH allows you to filter the documents for which the filtered attribute starts with the given string. It's similar to the SQL LIKE operator used in that way venture%

echo '{"q": "spiderman", "filter": "genres STARTS WITH venture" }' | xh 'localhost:7700/indexes/movies/search'

Use ENDS WITH

ENDS WITH allows you to filter the documents for which the filtered attribute ends with the given string. It's similar to the SQL LIKE operator used in that way %venture

echo '{"q": "spiderman", "filter": "genres ENDS WITH venture" }' | xh 'localhost:7700/indexes/movies/search'

---

⚠️ We do NOT recommend using this prototype in production. This is only for test purposes.

Everyone is more than welcome to give feedback and to report any issue or bug you might encounter when using this prototype. Thanks in advance for your involvement. It means a lot to us ❤️

[rtjampens]

Hi Kerollmops,

A quick feedback on your amazing work.

Everything looks fine on our side. I mainly tested the CONTAINS operator which is the one we are interested in. I loaded a 1Gb JSON file into meilisearch (containing 1,000,001 documents), indexed it and configured one field of the JSON as a filterable attribute. I then proceeded to request the index, and so far the performance is still great : between 10ms and 30ms consistently (if I refer to the processingTimeMs in the response).

Some info you may find interesting (feel free to ask for more if that can help you):

http://localhost:7777/indexes/participants_lab2/stats

{
  "numberOfDocuments": 1000001,
  "isIndexing": false,
  "fieldDistribution": {
    "client": 1000001,
    "id": 1000001,
    "partner": 1000001,
    "phoneNumber": 1000001,
    "pro": 1000001,
    "type": 1000001
  }
}

The index contains 1,000,001 documents following the same structure (but with different values of course), e.g.

{
    "type": "pro",
    "id": "aa991c8c-64b0-363f-8039-71ed6c5ba509",
    "phoneNumber": "0533236444",
    "pro": {
        "id": "8f393a97-3d80-34d1-b629-6ed206e8ce99",
        "phoneNumber": "0278002362",
        "companyName": "Lecomte SA",
        "email": "[email protected]",
        "subjectToVat": true,
        "status": "not_candidate",
        "personnalIdentity": {
            "civility": "m",
            "firstName": "C\u00e9cile",
            "lastName": "Delmas"
        },
        "affectedRegion": {
            "code": "SO",
            "label": "Sud Ouest"
        },
        "address": {
            "address1": "44, impasse de Pierre\n19147 Dijoux-sur-Vasseur",
            "address2": "Chambre 056",
            "postalCode": "93262",
            "city": "PerrotVille",
            "countryName": "France, Fr"
        }
    },
    "client": null,
    "partner": null
},

We tried to filter on the pro.companyName with the following curl command:

curl -X POST 'http://localhost:7777/indexes/participants_lab2/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "filter": [
      "pro.companyName CONTAINS \"Lecomte\""
    ]
  }'

{
  "hits": [ ... ],
  "query": "",
  "processingTimeMs": 18,
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 1000
}

So for this looks very good :)

[Kerollmops]

Hey @rtjampens,

Thank you very much for your time and the tests you made. It is very interesting. Fortunately, your dataset combined with your filter is an excellent example of good use of CONTAINS. I fear people using this filter with much longer values than a short company name.

• Is your dataset auto-generated or a real organic one?
• How long does it take when you try to search for SA, for example?
• How long does it take when you try to search for an impossible-to-match character, i.e. 狗?
• What if you do all your tests and the one above with another, longer field i.e. pro.address.address1?
• What if you combine much more of those new filters together?

      pro.companyName CONTAINS 'Lecomte'
  AND pro.companyName ENDS WITH 'SA'
  AND pro.companyName CONTAINS 'comte'
  AND pro.companyName CONTAINS 'le'

Take the time you need to answer,
Have a nice day 🍥

[rtjampens]

Is your dataset auto-generated or a real organic one?

It is an auto-generated one unfortunately, so distribution of words does not totally represent a real-world scenario. I can provide the raw JSON files if you want.

How long does it take when you try to search for SA, for example?

On this very same dataset, it takes around 55ms (between 50ms and 63ms in my tests)

How long does it take when you try to search for an impossible-to-match character, i.e. 狗?

Still on this dataset, it takes around 20ms (between 15ms and 30ms, but clearly more biased towards 15ms)

What if you do all your tests and the one above with another, longer field i.e. pro.address.address1?

Here are some tests on the different scenarios, response times keep getting consistent :

• "Normal/common" word

"filter": [ "pro.address.address1 CONTAINS \"impasse\"" ]

Response times are ~80ms

• Very short word

"filter": [ "pro.address.address1 CONTAINS \"de\"" ]

Response times are ~110ms

• Impossible-to-match character

"filter": [ "pro.address.address1 CONTAINS \"狗\"" ]

Response times are ~65ms

What if you combine much more of those new filters together?

"filter": [ 
  "pro.companyName CONTAINS 'Lecomte' AND pro.companyName ENDS WITH 'SA' AND pro.companyName CONTAINS 'comte' AND pro.companyName CONTAINS 'le'"
]

This query returns less time-consistent results, varying from 70ms to 93ms (biased towards 70ms though)

Feel free to ask for moar tests :D

Just as a side-note : while I understand the concerns on performance, I think developpers should be aware of the performance impact for using these operators. If it is clearly stated in the documentation, it's more than fine IMO

[Kerollmops]

Thank you very much!

It is an auto-generated one unfortunately, so distribution of words does not totally represent a real-world scenario. I can provide the raw JSON files if you want.

I am very interested, could you send me an email or link it here, please?

This query returns less time-consistent results, varying from 70ms to 93ms (biased towards 70ms though)

That is astonishing! I haven't done any optimization on the matter, and you have quite good processing times.

Feel free to ask for moar tests :D

Then I'll ask you more questions 😄

• What is the number of documents returned from each filter above?
• What's the number of different facet values you have in pro.companyName and pro.address.address1?
• What are your machine specs, i.e. RAM, CPUs, Disk type?
• What if you try this filter query?

      pro.companyName CONTAINS 'Lecomte'
  AND pro.companyName ENDS WITH 'SA'
  AND pro.companyName CONTAINS 'comte'
  AND pro.address.address1 CONTAINS 'de'
  AND pro.address.address1 CONTAINS 'impasse'

Just as a side-note : while I understand the concerns on performance, I think developpers should be aware of the performance impact for using these operators. If it is clearly stated in the documentation, it's more than fine IMO

Unfortunately, the performance impact can be used as a way of DDoSing servers. If we expose those operators, they can be used by anyone to DDoS a Meilisearch server, and we don't want that to be possible. We recently sped up the search algorithm to avoid that.

[Kerollmops]

Hey @rtjampens,

Have you got time to read my message and measure the time Meilisearch takes for those queries?

Have a nice day 🎨

[Nakusu]

Hi,

The implementation of CONTAINS and STARTS WITH operators is perfect but we have another question.
I will explain it below.

Index :

[
  {
    id: 1082,
    name: "xxxx",
    firstname: "xxxx",
    email: "xxxx",
    experiences: [
      {
        name: "Airbus & Inc",
        ongoing: 0,
      },
      {
       name: "Apple",
       ongoing: 1,
      },
    ]
  },
  {
    id: 1083,
    name: "xxxx",
    firstname: "xxxx",
    email: "xxxx",
    experiences: [
      {
        name: "Airbus & Inc",
        ongoing: 1,
      },
      {
       name: "Apple",
       ongoing: 1,
      },
    ]
  },
]

When we make this research with filters about experiences, we have a weird comportement.
Research filter: (experiences.name CONTAINS 'Airbus' AND experiences.ongoing = 1).
The research about nested doesn't add conditions on others nested research, in this case the research filter i use above will return element with id 1082 and 1083.
We expected a return of only the element with id 1083 which respect our filter research with an experience which contains a name with "Airbus" and which had the field ongoing at 1.

What is the research correct filter to use in this case ?

[gmourier]

Hey @Nakusu 👋

Thanks for your feedback about the CONTAINS and STARTS WITH operators.

Indeed, it's impossible to have that behavior currently because nested objects are matched as a whole. There is no sense of sub-documents. Elasticsearch solves this with a particular "nested" object type which we don't have as Algolia and Typesense.

A workaround for that may exist; Reformating the data around the experience could work 🤔

e.g., Every document is an experience, and you duplicate the contact information in every document. In that case, it will only return records (experiences) that match the filters. WDYT?

[GelistirmeKF]

Hi,

We have a category field that has flexible and hierarcial structure. If we want to list a parent category's items, we think that CONTAINS keyword would help us.

We suppose that every item will have a field like category tree for example: categoryTree: "-> Main Root Category -> Parent1 Category -> Parent2 Category -> ... -> ParentN Category -> Sub Category". If we want to list Parent2 category's items, we will do this by filtering "categoryTree CONTAINS '-> Parent2 Category ->'" command.

Do you have any suggestions for solving this requirement without CONTAINS keyword?

Kindly regards,
KF

[gmourier]

Hey @GelistirmeKF 👋

What do you mean by list Parent2 category's items? Is it for all the docs or just for one? Why do you want to do that?

If you can add some frontend screens to see what you want to achieve, it could help a lot!

Thanks!

[GelistirmeKF]

Thank you @gmourier,

We have a category field that has hierarcial structure. For example, a harddisk product is in a 'Storage' category; also in 'Computer Parts' , 'Computer and Tablets' and 'Electronics' categories which are parent category of 'Storage' category. That's why, in Meilisearch for this harddisk product, we kept this info in "categoryTree" field having 'Electronics -> Computer and Tablets -> Computer Parts -> Storage' data. So if we want to search "Computer Parts" products in Meilisearch, we can use "categoryTree CONTAINS '-> Computer Parts ->'" for products index.

The category hierarchy for harddisk product is like this:
Electronics
|-> Computer and Tablets
|-> Computer Parts
|-> Storage

Is there any way to search "Computer Parts" category's products without CONTAINS keyword in Meilisearch now?

Thanks in advance :)

[gmourier]

Hey @GelistirmeKF 👋

Thanks for the details.

If I understood correctly, you don't have to use the CONTAINS keyword for that. You can use a hierarchical menu to let your users navigate and narrow their search per category.

I've edited a code sandbox so you can see it in action (note: I'm using instant-meilisearch, but it's possible to do it in a pure custom way if you want - instant-meilisearch is just a UI search kit made of built-in customizable components to build a search interface quickly)

Code sandbox link: https://j5mc54.csb.app/

Here is how the documents are structured

[
    {
        "id": 1,
        "name": "Seagate BarraCuda 1 To (ST1000DM014)",
        "hierarchicalCategories": {
            "lvl0": "Electronics",
            "lvl1": "Electronics > Computer and Tablets",
            "lvl2": "Electronics > Computer and Tablets > Computer Parts",
            "lvl3": "Electronics > Computer and Tablets > Computer Parts > Storage"
        } 
    },
    {
        "id": 2,
        "name": "Wacom Cintiq Pro 24",
        "hierarchicalCategories": {
            "lvl0": "Electronics",
            "lvl1": "Electronics > Computer and Tablets",
            "lvl2": "Electronics > Computer and Tablets > Tablets"
        } 
    },
    {
        "id": 3,
        "name": "LG 75QNED816RE",
        "hierarchicalCategories": {
            "lvl0": "Image & Sound",
            "lvl1": "Image & Sound > Television",
            "lvl2": "Image & Sound > Television > TV"
        } 
    },
    {
        "id": 4,
        "name": "Canon EOS 2000D + EF-S 18-55 mm IS II",
        "hierarchicalCategories": {
            "lvl0": "Image & Sound",
            "lvl1": "Image & Sound > Camera",
            "lvl2": "Image & Sound > Camera > Reflex"
        } 
    }
]

I defined hierarchicalCategories as a filterableAttributes index setting, and then I used a HierarchicalMenu Instant-meilisearch widget in the front end code.

<HierarchicalMenu
          attributes={[
            "hierarchicalCategories.lvl0",
            "hierarchicalCategories.lvl1",
            "hierarchicalCategories.lvl2",
            "hierarchicalCategories.lvl3"
          ]}
/>

Let me know if I misunderstood! Thanks

[GelistirmeKF]

Thank you so much! @gmourier

This is a clean solution for us, but we found another way similar to your proposal. We assigned these category items to an attribute (categoryTree) in products index as an array by following:

Product Item1:
"categoryTree": ["Electronics", "Computer and Tablets", "Computer Inner Parts", "Processors"]

Product Item2:
"categoryTree": ["Electronics", "Computer and Tablets", "Computer Inner Parts", "Sound Cards"]

(Also added "categoryTree" attribute as an FilterableAttribute settings)

If we list products index items by "filter": "(categoryTree = 'Computer Inner Parts')", then item1 and item2 are listed.

If we list products index items by "filter": "(categoryTree = 'Sound Cards')", then only item2 is listed.

Hope to help others for similar cases

Appreciate for your detail explanation and example 🙏

[cu-cdabo]

Hi, we are also hoping for a progress update as we have need of the "starts with" filter. We have a CMS and have found Meilisearch to be amazing, the part that we've had to build a workaround for is being able to restrict search to a particular folder and subfolders. We allow users to embed their own search field on their pages and choose a base folder, so for the following data for instance, we'd like to be able to say "url starts with /parent-folder/"

[
    {
        "id": "89DC7790-C8D1-46AB-4FB8-08DB7DE8FAA1",
        "name": "Page 1",
        "searchableContent": "Main content Lorem ipsum dolor sit amet...",
        "url": "/parent-folder/child-folder-1/page-1"
    },
    {
        "id": "F7B60467-750A-493E-4FB7-08DB7DE8FAA1",
        "name": "Page 2",
        "searchableContent": "Main content Lorem ipsum dolor sit amet...",
        "url": "/parent-folder/page-2"
    }
]

Thanks for all your hard work, Meilisearch really has been a godsend, and because performance is so great we've been able to workaround this issue by simply searching everything and filtering out unwanted results in our own code, but would be ideal to have a single search request

[2567910]

We are considering moving away from meilisearch because we need to "contain string operator" on the filter.

Our data includes a parameter called crashCase = Some user entered string
We need to filter all crashCases that include the keyword "Tire".

We cannot use the query because crashCase is not searchable. (Its too long)

Is there a workaround for this?

[curquiza]

Hello @2567910 what do you mean with "We cannot use the query because crashCase is not searchable. (Its too long)"?

You mean the indexing is too long if you put crashCase as searchableAttributes?

[honhimW]

Hi, since there is already a prototype version for experience, I would like to know when this feature can be added to Experimental features as an optional feature to preview?😊

[macraig]

Hi @honhimW ! We don't have an ETA for the feature yet as we have some concerns about how it would affect search performance depending on the dataset (see comment above ).

Were you able to test the prototype? Any feedback you have on the prototype would help us assess the potential risks of adding it as an Experimental Feature.

[macraig]

@honhimW FYI we're adding it as Experimental in our next release, please see the message below.

[honhimW]

@macraig Thank you guys. Eventually, I don’t have to build it myself anymore.(❁´◡`❁)

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing an experimental CONTAINS filter operator!
You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.10.0-rc.0

You are welcome to leave your feedback or report bugs in this discussion.

Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on August 26th, 2024

⚠️ RC (release candidates) are not recommended for production