Allow getting many documents by id

[ghost]

After reading around I think it's only possible to get one document by it's unique id. It would be nice if we could give an array of ids and get the result back in a single request. Right now I am making a request for each id.

Thanks.

[paligiannis]

I need to address this issue too. Any ideas of implementing that?

[gmourier]

Hello @san-serif and @paligiannis! Thank you for using MeiliSearch and taking the time to contribute to this project. Currently, we are putting a lot of effort into the release of the new engine but it may be something that we want to address in the future. Can you please tell me more about your needs regarding this feature? Thanks!

[paligiannis]

@gmourier It seems like we need something to fetch specific records from the database. Usually, the use case is something like getting ids from database like products[ids]=100,300,400,500. So it's something like filtering specific results and them use them with facets of any other implementation of Meilisearch.

[ghost]

Hey @gmourier! This is kind of low priority for me as MeiliSearch is already working great for me. I want this feature because I want to get some data without directly querying my main database (mostly for backoffice stuff).

[paligiannis]

@san-serif How have you implemented such a thing?

[ghost]

@paligiannis

I'm using javascript and I essentially did it like this:

(async () => {
  const client = new MeiliSearch({ host: "https://endpoint" });
  const index = client.index("products");

  const ids = [100,300,400,500];

  const promises = ids.map(id => index.getDocument(id));

  const documents = await Promise.all(promises);

})();

[gmourier]

Hi! Thank you for your answers.

I mark the issue as a feature request to track it in our backlog. You can also propose this feature as an idea here. It allows people to vote for it and make it more visible in terms of importance.

[ghost]

@MarinPostma Just to clarify I think this is low priority because @gmourier mentioned you guys are working on a new engine, I did not want to give the impression that I think this is super important.

I don't know how to explain my use case better. I want to query data directly from MeiliSearch because I think it would be faster than my database.

[jiminy-billy-bob]

We solved this by declaring the id field as an attribute for faceting. Then we simply do something like this:

await meilisearch.getIndex('myIndex').search(undefined, {
  facetFilters: [documents.map(d => `id:${d.id}`)],
  limit: 999999
});

This should return documents with id foo and bar.

[tty2]

I see that this discussion is outdated but I'm curious if there is a convenient way to get documents by ids array using current engine?
I'm using golang api and can't find common way to do that.

[mikerogerz]

I was unable to find any optimal way via the API, so I did something like this (in PHP) to pull and update existing documents with new data:

$offset = 0;
$moreDocs = true;
do {
	$docs = collection($index->getDocuments([
		'limit' => 10000,
		'offset' => $offset,
		'attributesToRetrieve' => 'objectID,id,another_attr'
	]));
	if (!$docs->isEmpty()) {
		// match docs to update (based on id, etc), update attribute, and push updates to queue via API updateDocuments()
		
		$offset += 10000;
	} else {
		$moreDocs = false;
	}
} while ($moreDocs);

Not the most elegant solution, but it is/was the only reliable way I was able to match all documents, especially when indexing long-form content (multiple docs sharing the same id, but with different chunks of data to match on), so pulling individual documents wouldn't work either. Being able to pull by array would be extremely useful, in my opinion.. to grab a few hundred/thousand at a time.

[tacman]

Although this is an older post, I'm surprised it hasn't been solved yet. Getting a single document by primary key is simple, it shouldn't be hard to get a set of documents by an array of primary keys. Am I missing something?

[derekperkins]

This has been solved now

• Get Documents by Batch #644 (comment)

[irevoire]

Thanks, do you know what kind of API you want to see?

[tacman]

I think expanding on /documents (GET) and /fetch (POST) would be seamless.

curl -X GET 'http://localhost:7700/indexes/dogs/documents?limit=1&filter=doggo=bernese'

curl -X GET 'http://localhost:7700/indexes/dogs/documents?ids=cody,finn,brandy,gambit'

curl -X POST http://localhost:7700/indexes/dogs/documents/fetch
-H 'Content-Type: application/json'
--data-binary '{ "limit": 1, "ids": ["cody","finn"] }'

That way, it could be combined with filter and search, which would rock.

// psuedo-code

$ids = $this->getStudentIds(['class' => 'civics101']); // some database query
$index->getDocuments(['ids'=>$ids, 'filter'=> ['gender'=>'nonbinary']);

[irevoire]

Oh, interesting I was wondering if the filter and ids should be excluding each other or could be used together.
I’ll send your proposition to the team; thanks for your time 🔥

[tacman]

basically, everywhere you can send in a filter, you should be able to send in a list of ids, since really the primary key is a property but because it's unique, it is handled differently than a filter. But it's fundamentally just a way of accessing the data.

[macraig]

@tacman , in your proposal are the ids and the filter working as an AND or an OR? That is, do you want the intersection of ids AND documents that match the filter, or any document that has one of the requested ids OR matches the filter? We're assuming it's an AND, but wanted to check with you to make sure we don't miss your use case.

[tacman]

Here's an example of why this is important. I have two indexes, movies and actors. Both have a million rows (which meili, because it's awesome, handles without a sweat).

When I look up a movie, I want to get the actors and some data about them, but that data depends on the application. One application may want birthday, another nationality or languages. I don't want to store all the actor data in the movie database, I simply want an array of actors by code or imdb id or whatever.

So I get my movie results back and aggregate actor list so I have a single set of all the actors referenced from the movie result. I now want to fetch the documents in the actor index, by id, get back all the data (or the data my application specifies), merge it with my result set and send it off to the next step. Obviously fetching the actor data by id may need to be done in batches.

I don't want to iterate through the list of hundreds of actors one at a time (loading the document by id), I want to load all the actor data as efficiently as possible.

Facets are a terrible approach. They're not designed for millions of individual ids -- that's the role of the ID!

In a relational database, this would be a simple inner join between the movie and actor tables. In meili, it's a simple 2-index lookup, but is impossible without the ability to lookup a batch of documents by id.

The meili developers have created a blazingly fast lookup for searching, I imagine it would be very simple for them to bypass the "hard" searching and simple return a collection of documents based on the id.

My actual application is more complicated than this, but I hope this example is clear enough to warrant prioritizing batch lookup by id in the next release. Thank you for your consideration.

[tacman]

Here's an example of why this is important. I have two indexes, movies and actors. Both have a million rows (which meili, because it's awesome, handles without a sweat).

When I look up a movie, I want to get the actors and some data about them, but that data depends on the application. One application may want birthday, another nationality or languages. I don't want to store all the actor data in the movie database, I simply want an array of actors by code or imdb id or whatever.

So I get my movie results back and aggregate actor list so I have a single set of all the actors referenced from the movie result. I now want to fetch the documents in the actor index, by id, get back all the data (or the data my application specifies), merge it with my result set and send it off to the next step. Obviously fetching the actor data by id may need to be done in batches.

I don't want to iterate through the list of hundreds of actors one at a time (loading the document by id), I want to load all the actor data as efficiently as possible.

Facets are a terrible approach. They're not designed for millions of individual ids -- that's the role of the ID!

In a relational database, this would be a simple inner join between the movie and actor tables. In meili, it's a simple 2-index lookup, but is impossible without the ability to lookup a batch of documents by id.

The meili developers have created a blazingly fast lookup for searching, I imagine it would be very simple for them to bypass the "hard" searching and simple return a collection of documents based on the id.

My actual application is more complicated than this, but I hope this example is clear enough to warrant prioritizing batch lookup by id in the next release. Thank you for your consideration.

[macraig]

Hi @tacman , apologies for the late reply. We currently rely on facets as our approach, though we recognize it might not be the ideal solution for every use case. Other than the preference to not use facets from a theoretical standpoint, are there any examples where this blocking your use case?

[tacman]

Suppose there were a special exemption for searching by the primary key? That is, if the primary key is imdb_id, and the /fetch endpoint requests imdb_id=1,2,3, instead of throwing an error and saying "hey, you need to set imdb_id as filterable" it say "this is the primary key, I already have the index, this doesn't need to be set as filterable".

That would be natural for developers and wouldn't require any additional tweaks to the endpoint.

And those of us using the "set the id to filterable" technique would simply need to remove that.

There's no value to setting a facet / filterable property to something that has millions of unique ids. But we need a way to get batch documents by id. Again, thanks for your consideration.

[tacman]

It's a terrible solution. Facets don't make sense for unique values. ANYTHING where you need to access the data in meili by id. In my case, I have related data as a list of keys, so I fetch the data first, then I need to fetch all the records with those ids. Meili is a search engine, but also a datastore, and it's shocking that I can't get multiple records by id. Facets aren't designed for millions of unique records, but meili is.

…

On Wed, Jan 24, 2024 at 11:09 AM Maria Craig ***@***.***> wrote: Hi @tacman <https://github.com/tacman> , apologies for the late reply. We currently rely on facets as our approach, though we recognize it might not be the ideal solution for every use case. Other than the preference to not use facets from a theoretical standpoint, are there any examples where this blocking your use case? — Reply to this email directly, view it on GitHub <#99 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXIQIHRDW5QDMMDVFJIUDYQFMANAVCNFSM5E734W52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQMRTGY3TIMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dnsang1611]

Did we solve this problem yet? I currently need this feature for my application.

[tacman]

No solution as of yet. I'm not sure I've done a good enough job of explaining how important it is. And easy it is, relative to everything else meilisearch already does. Unfortunately, facets are fine for small datasets, so the need isn't obvious in many cases.

I keep a parallel database in sqlite that is used for the lookup by id. Not ideal.

[macraig]

Hi @dnsang1611 and @tacman, the team’s aware of how valuable this feature would be, but we’ve had a lot of other priorities to juggle. That said, we’re currently discussing the details and trying to prioritize it in one of the upcoming releases this year. We really appreciate your patience and all the feedback you’ve been sharing with us.

[ed-sparkes]

i solved by adding id as filterableAttribute and then just using a filter, any reason why this is a bad approach ?

[tacman]

yes, it won't scale. Filters (aka facets) aren't designed for unique values, but a limited set of repeated values. But if you're dataset is small, that solution will work fine. If you've got a dataset of a million people and have a set of 100,000 social security numbers (the key) you have to look them up one at a time.

[tacman]

In my particular use case, I just need to lookup the records by a set of ids, which are coming from another meilisearch. (I have an idea about relations, but way more complicated than this, and admittedly harder to justify). But largely this request is to allow me to populate related data. So if every user has a city code, I fetch the users I want, and now want to display the related city data, which I don't want to repeat in the user table. Nor do I want to look up each city one at a time.

Being able to search within a set of ids is interesting and likely useful, but that would complicate the task considerably. So I'd be fine with just looking up by document ids. I realize my example above showed both -- fine for brainstorming, but also fine for a later revision.

Thanks for your consideration on this.