Skip to content

Commit

Permalink
adding compose readme and adjusted default readme (edited gitignore f…
Browse files Browse the repository at this point in the history
…or the config)
  • Loading branch information
Matt Feinberg committed Nov 15, 2024
1 parent 300d074 commit a0c8f11
Show file tree
Hide file tree
Showing 3 changed files with 57 additions and 2 deletions.
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# General ignores
.DS_Store
**/tmp
code/config.yaml
# code/config.yaml

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down Expand Up @@ -58,4 +58,4 @@ cover/
# Workbench Project Layout
data/scratch/*
data/dataset
!data/scratch/.gitkeep
!data/scratch/.gitkeep
28 changes: 28 additions & 0 deletions code/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
llm_model:
name: "meta/llama3-8b-instruct"

# AI Catalog
url: "https://integrate.api.nvidia.com/v1"

# Locally running NIM
# url: "http://llm-nim:8000/v1"

embedding_model:
name: "nvidia/nv-embedqa-e5-v5"

# AI Catalog
url: "https://integrate.api.nvidia.com/v1"

# Locally running NIM
#url: "http://nv-embedqa-e5-v5:8000/v1"

reranking_model:
# AI Catalog
name: "nv-rerank-qa-mistral-4b:1"
url: "https://integrate.api.nvidia.com/v1"

# Locally running NIM
#name: "nvidia/nv-rerankqa-mistral-4b-v3"
#url: "http://nv-rerankqa-mistral-4b-v3:8000/v1/"

# Milvus and Redis have been configured with env variables.
27 changes: 27 additions & 0 deletions compose-README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# How to Run NIM Anywhere with Compose

## Steps to Get Started

1. **Clone and Configure**
- Follow the existing clone and configuration instructions.
- You can **skip the mount setup**.
- Make sure to set your **NGC API KEY secret**.

2. **Start the Application**
- Scroll down to the **Apps** section in your environment.
- Look for the **Compose** section under Applications.
- From the dropdown menu:
- Select the number of GPUs you have.
- Click **Start**.
- This will start up the following containers based on your profile:
- **0 GPUs**: `milvus` and `redis`
- **1 GPU**: `milvus`, `redis`, and the LLM `NIM`
- **2 GPUs**: `milvus`, `redis`, LLM `NIM`, and Embedding `NIM`
- **3 GPUs**: `milvus`, `redis`, LLM `NIM`, Embedding `NIM`, and Reranking `NIM`

3. **Launch Additional Services**
- After Compose starts up the containers, you can also start:
- **Chain Server**
- **Chat Frontend**
- **Jupyter Lab**

0 comments on commit a0c8f11

Please sign in to comment.