Skip to content

Commit

Permalink
[Doc] Update GPU package installation (#22204)
Browse files Browse the repository at this point in the history 
### Description
* Add CUDA/CuDNN requirements for 1.19.x
* Update installation guide for cuda 11 package and nightly.
* Remove a sentence for ROCm: `Alternatively, each major ORT release has
a corresponding C/C++ ROCm package, found
[here](https://github.com/microsoft/onnxruntime/releases/).` since
onnxruntime_rocm package is not available in recent releases.

### Motivation and Context
Doc is not up to date.
  • Loading branch information
@tianleiwu
tianleiwu authored Sep 24, 2024
1 parent 85c3f0f commit bc1bce9
Show file tree
Hide file tree
Showing 4 changed files with 27 additions and 23 deletions.
8 changes: 4 additions & 4 deletions docs/execution-providers/CUDA-ExecutionProvider.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,13 @@ Because of [Nvidia CUDA Minor Version Compatibility](https://docs.nvidia.com/dep

ONNX Runtime built with cuDNN 8.x is not compatible with cuDNN 9.x, and vice versa. You can choose the package based on CUDA and cuDNN major versions that match your runtime environment (For example, PyTorch 2.3 uses cuDNN 8.x, while PyTorch 2.4 or later used cuDNN 9.x).

Note: starting ORT 1.19, **CUDA 12.x** becomes default version when distributing ONNX Runtime GPU packages.
Note: starting ORT 1.19, **CUDA 12.x** becomes default version when distributing ONNX Runtime GPU packages in pypi.

### CUDA 12.x

To install CUDA 12 package, please look at [Install ORT](../install).

| ONNX Runtime | CUDA | cuDNN | Notes |
|---------------|--------|-------|----------------------------------------------------------------------|
| 1.19.x | 12.x | 9.x | Avaiable in pypi. Compatible with PyTorch >= 2.4.0 for cuda 12.x. |
| 1.18.1 | 12.x | 9.x | cuDNN 9 is required. No Java package. |
| 1.18.0 | 12.x | 8.x | Java package is added. |
| 1.17.x | 12.x | 8.x | Only C++/C# Nuget and Python packages are released. No Java package. |
Expand All @@ -51,7 +50,8 @@ To install CUDA 12 package, please look at [Install ORT](../install).

| ONNX Runtime | CUDA | cuDNN | Notes |
|----------------------|--------|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| 1.18.x | 11.8 | 8.x | |
| 1.19.x | 11.8 | 8.x | Not available in pypi. See [Install ORT](../install) for detail. Compatible with PyTorch <= 2.3.1 for CUDA 11.8. |
| 1.18.x | 11.8 | 8.x | Available in pypi |
| 1.17<br>1.16<br>1.15 | 11.8 | 8.2.4 (Linux)<br/>8.5.0.96 (Windows) | Tested with CUDA versions from 11.6 up to 11.8, and cuDNN from 8.2 up to 8.9 |
| 1.14<br>1.13 | 11.6 | 8.2.4 (Linux)<br/>8.5.0.96 (Windows) | libcudart 11.4.43<br/>libcufft 10.5.2.100<br/>libcurand 10.2.5.120<br/>libcublasLt 11.6.5.2<br/>libcublas 11.6.5.2<br/>libcudnn 8.2.4 |
| 1.12<br>1.11 | 11.4 | 8.2.4 (Linux)<br/>8.2.2.26 (Windows) | libcudart 11.4.43<br/>libcufft 10.5.2.100<br/>libcurand 10.2.5.120<br/>libcublasLt 11.6.5.2<br/>libcublas 11.6.5.2<br/>libcudnn 8.2.4 |
Expand Down
6 changes: 3 additions & 3 deletions docs/execution-providers/TensorRT-ExecutionProvider.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ Note: starting ORT 1.19, **CUDA 12** becomes default version when distributing O

| ONNX Runtime | TensorRT | CUDA |
| :----------- | :------- | :------------- |
| 1.19-main | 10.2 | **12.2**, 11.8 |
| 1.18 | 10.0 | 11.8, 12.2 |
| 1.17 | 8.6 | 11.8, 12.2 |
| 1.19-main | 10.2 | **12.x**, 11.8 |
| 1.18 | 10.0 | 11.8, 12.x |
| 1.17 | 8.6 | 11.8, 12.x |
| 1.16 | 8.6 | 11.8 |
| 1.15 | 8.6 | 11.8 |
| 1.14 | 8.5 | 11.6 |
Expand Down
10 changes: 5 additions & 5 deletions docs/get-started/with-python.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ pip install onnxruntime

### Install ONNX Runtime GPU (CUDA 12.x)

The default CUDA version for ORT is 12.2.
The default CUDA version for ORT is 12.x.

```bash
pip install onnxruntime-gpu
Expand Down Expand Up @@ -260,8 +260,8 @@ If using pip, run `pip install --upgrade pip` prior to downloading.
|[onnxruntime](https://pypi.org/project/onnxruntime)|CPU (Release)| Windows (x64), Linux (x64, ARM64), Mac (X64), |
|[ort-nightly](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly)|CPU (Dev) | Same as above |
|[onnxruntime-gpu](https://pypi.org/project/onnxruntime-gpu)|GPU (Release)| Windows (x64), Linux (x64, ARM64) |
|[ort-nightly-gpu for CUDA 11.*](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-gpu) |GPU (Dev) | Windows (x64), Linux (x64, ARM64) |
|[ort-nightly-gpu for CUDA 12.*](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ort-cuda-12-nightly/PyPI/ort-nightly-gpu) |GPU (Dev) | Windows (x64), Linux (x64, ARM64) |
|[ort-nightly-gpu for CUDA 11.*](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ort-cuda-11-nightly/PyPI/ort-nightly-gpu) |GPU (Dev) | Windows (x64), Linux (x64, ARM64) |
|[ort-nightly-gpu for CUDA 12.*](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-gpu) |GPU (Dev) | Windows (x64), Linux (x64, ARM64) |

Before installing nightly package, you will need install dependencies first.
```
Expand All @@ -270,12 +270,12 @@ python -m pip install coloredlogs flatbuffers numpy packaging protobuf sympy

Example to install ort-nightly-gpu for CUDA 11.*:
```
python -m pip install ort-nightly-gpu --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
python -m pip install ort-nightly-gpu --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-11-nightly/pypi/simple/
```

Example to install ort-nightly-gpu for CUDA 12.*:
```
python -m pip install ort-nightly-gpu --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ort-cuda-12-nightly/pypi/simple/
python -m pip install ort-nightly-gpu --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/
```

For Python compiler version notes, see [this page](https://github.com/microsoft/onnxruntime/tree/main/docs/Python_Dev_Notes.md)
Expand Down
26 changes: 15 additions & 11 deletions docs/install/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,25 +46,29 @@ For ONNX Runtime GPU package, it is required to install [CUDA](https://developer
pip install onnxruntime
```

#### Install ONNX Runtime GPU (CUDA 11.x)
The default CUDA version for ORT is 11.8.
#### Install ONNX Runtime GPU (CUDA 12.x)
The default CUDA version for ORT in pypi is 12.x since 1.19.0.

```bash
pip install onnxruntime-gpu
```

#### Install ONNX Runtime GPU (CUDA 12.x)
For Cuda 12.x, please use the following instructions to install from [ORT Azure Devops Feed](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview)
#### Install ONNX Runtime GPU (CUDA 11.x)
For Cuda 11.x, please use the following instructions to install from [ORT Azure Devops Feed](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/onnxruntime-cuda-11/PyPI/onnxruntime-gpu/overview) for 1.19.2 or later.

```bash
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/
```

#### Install ONNX Runtime GPU (ROCm)
For ROCm, please follow instructions to install it at the [AMD ROCm install docs](https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.0.0/). The ROCm execution provider for ONNX Runtime is built and tested with ROCm 6.0.0
For older version <= 1.18.1, you can download from pypi with explicit version like the following:
```bash
pip install onnxruntime-gpu==1.18.1
```

To build from source on Linux, follow the instructions [here](https://onnxruntime.ai/docs/build/eps.html#amd-rocm). Alternatively, each major ORT release has a corresponding C/C++ ROCm package, found [here](https://github.com/microsoft/onnxruntime/releases/).
#### Install ONNX Runtime GPU (ROCm)
For ROCm, please follow instructions to install it at the [AMD ROCm install docs](https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.0.0/). The ROCm execution provider for ONNX Runtime is built and tested with ROCm 6.0.0.

To build from source on Linux, follow the instructions [here](https://onnxruntime.ai/docs/build/eps.html#amd-rocm).

### Install ONNX to export the model

Expand Down Expand Up @@ -96,7 +100,7 @@ dotnet add package Microsoft.ML.OnnxRuntime

#### Install ONNX Runtime GPU (CUDA 12.x)

The default CUDA version for ORT is 12.2
The default CUDA version for ORT is 12.x

```bash
# GPU
Expand Down Expand Up @@ -405,8 +409,8 @@ below:
|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|
| Python | If using pip, run `pip install --upgrade pip` prior to downloading. | | |
| | CPU: [**onnxruntime**](https://pypi.org/project/onnxruntime) | [ort-nightly (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly/overview) | |
| | GPU (CUDA/TensorRT) for CUDA 11.x: [**onnxruntime-gpu**](https://pypi.org/project/onnxruntime-gpu) | [ort-nightly-gpu (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-gpu/overview/) | [View](../execution-providers/CUDA-ExecutionProvider.md#requirements) |
| | GPU (CUDA/TensorRT) for CUDA 12.x: [**onnxruntime-gpu**](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/onnxruntime-cuda-12/PyPI/onnxruntime-gpu/overview/) | [ort-nightly-gpu (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ort-cuda-12-nightly/PyPI/ort-nightly-gpu/overview/) | [View](../execution-providers/CUDA-ExecutionProvider.md#requirements) |
| | GPU (CUDA/TensorRT) for CUDA 12.x: [**onnxruntime-gpu**](https://pypi.org/project/onnxruntime-gpu) | [ort-nightly-gpu (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-gpu/overview/) | [View](../execution-providers/CUDA-ExecutionProvider.md#requirements) |
| | GPU (CUDA/TensorRT) for CUDA 11.x: [**onnxruntime-gpu**](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/onnxruntime-cuda-11/PyPI/onnxruntime-gpu/overview/) | [ort-nightly-gpu (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ort-cuda-11-nightly/PyPI/ort-nightly-gpu/overview/) | [View](../execution-providers/CUDA-ExecutionProvider.md#requirements) |
| | GPU (DirectML): [**onnxruntime-directml**](https://pypi.org/project/onnxruntime-directml/) | [ort-nightly-directml (dev)](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/ort-nightly-directml/overview/) | [View](../execution-providers/DirectML-ExecutionProvider.md#requirements) |
| | OpenVINO: [**intel/onnxruntime**](https://github.com/intel/onnxruntime/releases/latest) - *Intel managed* | | [View](../build/eps.md#openvino) |
| | TensorRT (Jetson): [**Jetson Zoo**](https://elinux.org/Jetson_Zoo#ONNX_Runtime) - *NVIDIA managed* | | |
Expand Down

0 comments on commit bc1bce9

Please sign in to comment.