-
Notifications
You must be signed in to change notification settings - Fork 69
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
adding compose readme and adjusted default readme (edited gitignore f…
…or the config)
- Loading branch information
Matt Feinberg
committed
Nov 15, 2024
1 parent
300d074
commit a0c8f11
Showing
3 changed files
with
57 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
llm_model: | ||
name: "meta/llama3-8b-instruct" | ||
|
||
# AI Catalog | ||
url: "https://integrate.api.nvidia.com/v1" | ||
|
||
# Locally running NIM | ||
# url: "http://llm-nim:8000/v1" | ||
|
||
embedding_model: | ||
name: "nvidia/nv-embedqa-e5-v5" | ||
|
||
# AI Catalog | ||
url: "https://integrate.api.nvidia.com/v1" | ||
|
||
# Locally running NIM | ||
#url: "http://nv-embedqa-e5-v5:8000/v1" | ||
|
||
reranking_model: | ||
# AI Catalog | ||
name: "nv-rerank-qa-mistral-4b:1" | ||
url: "https://integrate.api.nvidia.com/v1" | ||
|
||
# Locally running NIM | ||
#name: "nvidia/nv-rerankqa-mistral-4b-v3" | ||
#url: "http://nv-rerankqa-mistral-4b-v3:8000/v1/" | ||
|
||
# Milvus and Redis have been configured with env variables. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# How to Run NIM Anywhere with Compose | ||
|
||
## Steps to Get Started | ||
|
||
1. **Clone and Configure** | ||
- Follow the existing clone and configuration instructions. | ||
- You can **skip the mount setup**. | ||
- Make sure to set your **NGC API KEY secret**. | ||
|
||
2. **Start the Application** | ||
- Scroll down to the **Apps** section in your environment. | ||
- Look for the **Compose** section under Applications. | ||
- From the dropdown menu: | ||
- Select the number of GPUs you have. | ||
- Click **Start**. | ||
- This will start up the following containers based on your profile: | ||
- **0 GPUs**: `milvus` and `redis` | ||
- **1 GPU**: `milvus`, `redis`, and the LLM `NIM` | ||
- **2 GPUs**: `milvus`, `redis`, LLM `NIM`, and Embedding `NIM` | ||
- **3 GPUs**: `milvus`, `redis`, LLM `NIM`, Embedding `NIM`, and Reranking `NIM` | ||
|
||
3. **Launch Additional Services** | ||
- After Compose starts up the containers, you can also start: | ||
- **Chain Server** | ||
- **Chat Frontend** | ||
- **Jupyter Lab** | ||
|