Merge pull request #115 from SylphAI-Inc/li

[Extra pacakges + colab quick start]

.gitignore

@@ -42,3 +42,4 @@ storage/
 /*.dot
 /*.svg
 /*.csv
+index.faiss

README.md

@@ -1,18 +1,42 @@
 ![LightRAG Logo](https://raw.githubusercontent.com/SylphAI-Inc/LightRAG/main/docs/source/_static/images/LightRAG-logo-doc.jpeg)
-<!-- [![release](https://img.shields.io/github/v/release/SylphAI-Inc/LightRAG?sort=semver)](https://github.com/SylphAI-Inc/LightRAG/releases) -->
-<!-- [![Dependency Status](https://img.shields.io/librariesio/github/SylphAI-Inc/LightRAG?style=flat-square)](https://libraries.io/github/SylphAI-Inc/LightRAG) -->
-[![PyPI](https://img.shields.io/pypi/v/lightRAG?style=flat-square)](https://pypi.org/project/lightRAG/)
-[![GitHub star chart](https://img.shields.io/github/stars/SylphAI-Inc/LightRAG?style=flat-square)](https://star-history.com/#SylphAI-Inc/LightRAG)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/lightRAG?style=flat-square)](https://pypistats.org/packages/lightRAG)
-[![Open Issues](https://img.shields.io/github/issues-raw/SylphAI-Inc/LightRAG?style=flat-square)](https://github.com/SylphAI-Inc/LightRAG/issues)
-[![License](https://img.shields.io/github/license/SylphAI-Inc/LightRAG)](https://opensource.org/license/MIT)
-[![](https://dcbadge.vercel.app/api/server/zt2mTPcu?compact=true&style=flat)](https://discord.gg/zt2mTPcu)
+
+<p align="center">
+    <a href="https://discord.gg/ezzszrRZvT">
+        <img alt="discord-invite" src="https://dcbadge.vercel.app/api/server/ezzszrRZvT?style=flat">
+    </a>
+</p>
+
+<p align="center">
+    <a href="https://pypi.org/project/lightRAG/">
+        <img alt="PyPI Version" src="https://img.shields.io/pypi/v/lightRAG?style=flat-square">
+    </a>
+    <a href="https://colab.research.google.com/drive/1TKw_JHE42Z_AWo8UuRYZCO2iuMgyslTZ?usp=sharing">
+        <img alt="Try Quickstart in Colab" src="https://colab.research.google.com/assets/colab-badge.svg">
+    </a>
+    <!-- <a href="https://pypistats.org/packages/lightRAG">
+        <img alt="PyPI Downloads" src="https://img.shields.io/pypi/dm/lightRAG?style=flat-square">
+    </a> -->
+    <!-- <a href="https://github.com/SylphAI-Inc/LightRAG/issues">
+        <img alt="Open Issues" src="https://img.shields.io/github/issues-raw/SylphAI-Inc/LightRAG?style=flat-square">
+    </a> -->
+    <a href="https://opensource.org/license/MIT">
+        <img alt="License" src="https://img.shields.io/github/license/SylphAI-Inc/LightRAG">
+    </a>
+</p>
+
+
+
+<!-- <a href="https://colab.research.google.com/drive/1PPxYEBa6eu__LquGoFFJZkhYgWVYE6kh?usp=sharing">
+        <img alt="Try Quickstart in Colab" src="https://colab.research.google.com/assets/colab-badge.svg">
+    </a> -->
+
+
 
 
 ### ⚡ The Lightning Library for Large Language Model Applications ⚡
 
 *LightRAG* helps developers build and optimize *Retriever-Agent-Generator* pipelines.
-Embracing a design philosophy similar to *PyTorch*, LightRAG is *light*, *modular*, and *robust*, with a 100% readable codebase.
+Embracing similar design pattern to *PyTorch*, LightRAG is *light*, *modular*, and *robust*, with a 100% readable codebase.
 
 
 # Why LightRAG?
@@ -209,10 +233,6 @@ You:
 ## Model-agnostic
 
 
-You can switch to any model simply by using a different model_client (provider) and model_kwargs.
-Let's use OpenAI's gpt-3.5-turbo model on the same pipeline.
-
-
 You can switch to any model simply by using a different `model_client` (provider) and `model_kwargs`.
 Let's use OpenAI's `gpt-3.5-turbo` model.
 

docs/source/get_started/installation.rst

@@ -12,7 +12,14 @@ To install the package, run:
 
    pip install lightrag
 
+If you know you will need `openai` and `faiss-cpu`, you can do so with:
 
+.. code-block:: bash
+
+   pip install lightrag[openai, faiss]
+
+.. note::
+   Check the `Optional Packages` section for more information on the available packages.
 
 2. Set up API keys
 ~~~~~~~~~~~~~~~~~~~
@@ -58,21 +65,24 @@ This setup ensures that LightRAG can access all necessary configurations during
 
 
 LightRAG currently has built-in support for (1) OpenAI, Groq, Anthropic, Google, and Cohere, and (2) FAISS and Transformers.
-You can find all optional packages at :class:`utils.lazy_import.OptionalPackages`.
+You can find all optional packages at :class:`OptionalPackages<utils.lazy_import.OptionalPackages>`.
 Make sure to install the necessary SDKs for the components you plan to use.
 Here is the list of our tested versions:
 
 
 .. code-block::
 
-    openai = "^1.12.0"
-    groq = "^0.5.0"
-    faiss-cpu = "^1.8.0"
-    sqlalchemy = "^2.0.30"
-    cohere = "^5.5.8"
-    pgvector = "^0.2.5"
-    anthropic = "^0.26.0"
-    google-generativeai = "^0.5.4"
+   openai = "^1.12.0"
+   groq = "^0.5.0"
+   faiss-cpu = "^1.8.0"
+   sqlalchemy = "^2.0.30"
+   pgvector = "^0.3.1"
+   torch = "^2.3.1"
+   anthropic = "^0.31.1"
+   google-generativeai = "^0.7.2"
+   cohere = "^5.5.8"
+
+You can install the optional packages with either `pip install package_name` or `pip install lightrag[package_name]`.
 
 
 

docs/source/index.rst

@@ -8,10 +8,23 @@
 
 .. raw:: html
 
+   <p align="center">
+
+
+    <a href="https://colab.research.google.com/drive/1TKw_JHE42Z_AWo8UuRYZCO2iuMgyslTZ?usp=sharing">
+        <img alt="Try Quickstart in Colab" src="https://colab.research.google.com/assets/colab-badge.svg">
+    </a>
+   </p>
+
    <div style="text-align: center; margin-bottom: 20px;">
       <a href="https://pypi.org/project/lightRAG/"><img src="https://img.shields.io/pypi/v/lightRAG?style=flat-square" alt="PyPI Version"></a>
+      <a href="https://github.com/SylphAI-Inc/LightRAG" style="display: inline-block; margin-left: 10px;">
+         <img src="https://img.shields.io/badge/GitHub-LightRAG-blue?logo=github&style=flat-square" alt="GitHub Repo">
+      </a>
       <a href="https://star-history.com/#SylphAI-Inc/LightRAG"><img src="https://img.shields.io/github/stars/SylphAI-Inc/LightRAG?style=flat-square" alt="GitHub Stars"></a>
-      <a href="https://discord.gg/ezzszrRZvT">  <img src="https://img.shields.io/discord/1065084981904429126?style=flat-square" alt="Discord"></a>
+      <a href="https://discord.gg/ezzszrRZvT">
+        <img alt="discord-invite" src="https://dcbadge.vercel.app/api/server/ezzszrRZvT?style=flat">
+      </a>
       <a href="https://opensource.org/license/MIT"><img src="https://img.shields.io/github/license/SylphAI-Inc/LightRAG" alt="License"></a>
    </div>
 

lightrag/CHANGELOG.md

@@ -1,5 +1,14 @@
+## [0.1.0-beta.2] - 2024-07-15
+
+### Modified
+- Make `LocalDB` a component for better visualization.
+- Add extra packages in dependencides.
+
+## [0.1.0-beta.1] - 2024-07-15
+
 ### Added
 - `Sequential` adds `acall` method.
+- Add extra packages so that users can install them with `pip install lightrag[extra]`.
 
 ## [0.0.0-beta.1] - 2024-07-10
 

lightrag/PACKAGING.md

@@ -0,0 +1,29 @@
+#Poetry Packaging Guide
+## Development
+
+To install optional dependencies, use the following command:
+
+```bash
+poetry install --extras "openai groq faiss"
+```
+Install more extra after the first installation, you will use the same command:
+
+```bash
+poetry install --extras "anthropic cohere google-generativeai pgvector"
+```
+
+## Extra Dependencies
+Add the optional package in dependencides.
+
+Build it locally:
+```bash
+poetry build
+```
+
+Test the package locally:
+
+Better to use a colab to update the whl file and test the installation.
+
+```bash
+pip install "dist/lightrag-0.1.0b1-py3-none-any.whl[openai,groq,faiss]"
+```

lightrag/README.md

@@ -179,8 +179,8 @@ Use the following code will let us see the prompt after it is formatted:
 
 ```python
 
-qa2.generator.print_prompt(
-        output_format_str=qa2.generator.output_processors.format_instructions(),
+qa.generator.print_prompt(
+        output_format_str=qa.generator.output_processors.format_instructions(),
         input_str="What is LLM?",
 )
 ```

lightrag/lightrag/components/retriever/faiss_retriever.py

@@ -80,6 +80,15 @@ class FAISSRetriever(
 
     We choose cosine similarity and convert it to range [0, 1] by adding 1 and dividing by 2 to simulate probability in [0, 1]
 
+    Install FAISS:
+
+    As FAISS is optional package, you can install it with pip for cpu version:
+    ```bash
+    pip install faiss-cpu
+    ```
+    For GPU version:
+    You might have to use conda to install faiss-gpu:https://github.com/facebookresearch/faiss/wiki/Installing-Faiss
+
     References:
     - FAISS: https://github.com/facebookresearch/faiss
     """

lightrag/lightrag/core/db.py

@@ -19,7 +19,7 @@
 
 
 @dataclass
-class LocalDB(Generic[T]):
+class LocalDB(Generic[T], Component):
     __doc__ = r"""LocalDB with in-memory CRUD operations, data transformation/processing pipelines, and persistence.
 
     LocalDB is highly flexible.
@@ -110,6 +110,9 @@ class LocalDB(Generic[T]):
         default_factory=dict, metadata={"description": "Map function setup by key"}
     )
 
+    def __post_init__(self):
+        super().__init__()
+
     @property
     def length(self):
         return len(self.items)
@@ -272,6 +275,7 @@ def add(
             for key, transformed_docs in transformed_items.items():
                 self.transformed_items[key].extend(transformed_docs)
 
+    # TODO: rename it better to add the condition filter
     def fetch_items(self, condition: Callable[[T], bool]) -> List[T]:
         """Fetch items with a condition."""
         return [item for item in self.items if condition(item)]

lightrag/lightrag/utils/lazy_import.py

@@ -21,7 +21,7 @@ class OptionalPackages(Enum):
     ANTHROPIC = ("anthropic", "Please install anthropic with: pip install anthropic")
     GOOGLE_GENERATIVEAI = (
         "google.generativeai",
-        "Please install google-generativeai to use GoogleGenAIClient",
+        "Please install google-generativeai with: pip install google-generativeai",
     )
     TRANSFORMERS = (
         "transformers",