v2.0.0 - 🔍 Embeddings, Tokens and Prefixes Oh My!

🎉 What's New

Added

• Added returnEmbedding option to chunkit and cramit functions to include embeddings in the output.
• Added returnTokenLength option to chunkit and cramit functions to include token length in the output.
• Added chunkPrefix option to prefix each chunk with a task instruction (e.g., "search_document: ", "search_query: ").
• Updated README to document new options and add RAG tips for using chunkPrefix with embedding models that support task prefixes.

⚠️ Breaking Change

• Returned array chunks is now an array of objects with text, embedding, and tokenLength properties. Previous versions returned an array of strings.

---

If you find this library useful please consider sending me a tip to support my work 😀

🍵 tip me here