Does langchain Documents work in BM25S?

[pradhandebasish2046]

from langchain.docstore.document import Document

doc = Document(page_content="text", metadata={"source": "local"})

Instead of a list of string, I want to give a list of Langchain documents but it's not working

[xhluca]

bm25s does not provide support for langchain objects, as it is not a necessary dependency.

I believe it is common to see langchain add internal support to a specific library (e.g. openai, rank-bm25, huggingface, see this for example). Perhaps it would make sense to open a PR on the langchain repo: https://github.com/langchain-ai/langchain

Otherwise, it'd make sense to convert the langchain document object to text, I think it should be documented in the langchain docs.

[pradhandebasish2046]

As I am Working on RAG and I want some metadata of that corresponding retrieved text, is there any way I can access it using bm25s retriever?

[xhluca]

if you don't pass a corpus to bm25s during init in retrieve(), it will return the index and scores. you can use the index to find the relevant document from your langchain Document list objects.

[mspronesti]

Opened this PR to integrate bm25s into langchain, as I needed some features of this library. Feel free to have a look if still interested.

[xhluca]

Nice!