Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NCSA - new conda installation instructions #80

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4b98d74
adding yml file for conda environment
tcnichol Oct 31, 2024
026afed
changes and adding new config for planet data
tcnichol Nov 14, 2024
34b3694
start of docs for conda install
tcnichol Nov 25, 2024
1a28d32
more instructions for conda installation
tcnichol Nov 25, 2024
adefaff
new better instructios for installing dependencies using conda env
tcnichol Nov 25, 2024
c70985d
set cyclopts version to 2.9.9
tcnichol Nov 25, 2024
77dc1ca
mps for mac
tcnichol Nov 25, 2024
3818501
adding a python class for running the pipeline for easier debugging
tcnichol Nov 26, 2024
bf56b17
Merge branch 'main' into ncsa
tcnichol Nov 27, 2024
f940bd5
trying to use metal for mac os
tcnichol Nov 29, 2024
b973c6f
error now due to nvml library need to fix to work with mac
tcnichol Nov 29, 2024
6c3d1f8
Merge branch 'main' into ncsa
tcnichol Dec 2, 2024
7cd4bf0
on sd-gpu this version is needed for cyclopts
tcnichol Dec 2, 2024
fd5f647
add project name
tcnichol Dec 2, 2024
719bda3
Merge remote-tracking branch 'origin/ncsa' into ncsa
tcnichol Dec 2, 2024
76ddf1f
full paths for directories in run_native
tcnichol Dec 2, 2024
f998520
full paths for directories in run_native
tcnichol Dec 2, 2024
2ce6527
script creates and re-installs the conda environment if it already ex…
tcnichol Dec 3, 2024
b736712
better instructions
tcnichol Dec 3, 2024
d5bc317
fixes error, not sure if this is a good policy, but works enough
tcnichol Dec 3, 2024
15aac9c
reverting change
tcnichol Dec 4, 2024
fe104ab
Merge branch 'main' into ncsa
tcnichol Jan 1, 2025
d45a02b
adding command to readme
tcnichol Jan 1, 2025
e8d63bb
setup for running fast pipeline
tcnichol Jan 1, 2025
923117b
fix environment creation
tcnichol Jan 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,26 @@ At the first start, VSCode should ask you if you want to install the recommended
The settings should be automaticly used by VSCode.
Both should provide the developers with a better experience and enforce code-style.

## Create Conda Environment from Script

This script will create (or recreate) the conda environment from a bash script.

Prereq
- [Conda] (https://docs.anaconda.com/miniconda/): Link to miniconda. Can use any other conda
- Install by running this command `. create_env.sh`
- The conda environment is installed, activate it and type `darts --help`

## Manual Conda Environment setup

If you prefer to install conda manually, here are steps

Prereq:
- [Conda] (https://docs.anaconda.com/miniconda/): Link to miniconda. Can use any other conda
- Begin by creating a conda environment `conda create -n darts-nextgen python=3.11`
- Install dependencies from project.toml `pip install '.[dev]'`
- Do the same in other packages (darts-acquisition to darts-superresolution)


## Environment setup

Prereq:
Expand Down Expand Up @@ -128,3 +148,7 @@ logging.basicConfig(
install(show_locals=True) # Change to False if you encounter too large tracebacks
init_ee("ee-project") # Replace with your project
```

currently running planet pipeline using this command

`darts run-native-planet-pipeline-fast --config-file=planet_config.toml`
50 changes: 50 additions & 0 deletions create_env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/bin/bash

CONDA_ENVS="$(conda env list)"
current_directory=$(pwd)
echo "${current_directory}"

echo "${CONDA_ENVS}"

conda remove -n darts-nextgen -all -y

conda create -n darts-nextgen python=3.11 -y

conda activate darts-nextgen

CUR_PYTHON="$(which python)"

pip install '.[dev]'


cd ${current_directory}/darts-acquisition
new_dir=$(pwd)
echo "${new_dir}"
pip install '.[dev]'

cd ${current_directory}/darts-ensemble
pip install '.[dev]'

cd ${current_directory}/darts-export
pip install '.[dev]'

cd ${current_directory}/darts-postprocessing
pip install '.[dev]'

cd ${current_directory}/darts-preprocessing
pip install '.[dev]'

cd ${current_directory}/darts-segmentation
pip install '.[dev]'


cd ${current_directory}/darts-superresolution
pip install '.[dev]'


cd ${current_directory}/darts-utils
pip install '.[dev]'

cd ${current_directory}

echo Created Environment
30 changes: 30 additions & 0 deletions darts/src/darts/run_native.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
from darts.native import run_native_planet_pipeline, run_native_planet_pipeline_fast

Check failure on line 1 in darts/src/darts/run_native.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (D100)

darts/src/darts/run_native.py:1:1: D100 Missing docstring in public module
from pathlib import Path
orthotiles_dir = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/input/planet/PSOrthoTile"

Check failure on line 3 in darts/src/darts/run_native.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (I001)

darts/src/darts/run_native.py:1:1: I001 Import block is un-sorted or un-formatted
scenes_dir = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/input/planet/PSScene"
arcticdem_slope_vrt = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/input/ArcticDEM/slope.vrt"
arcticdem_elevation_vrt = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/input/ArcticDEM/elevation.vrt"
output_data_dir = "/home/toddn/pdg/darts-nextgen/data/output"
model_dir = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/models"
arctic_dem_dir = "/home/toddn/pdg/darts-nextgen/rts_dtaset01/input/ArcticDEM"

tcvis_dir = "/home/toddn/pdg/darts-nextgen/tcvis_dir"

if __name__ == "__main__":
print("Running the pipeline")
run_native_planet_pipeline_fast(orthotiles_dir=Path(orthotiles_dir),
scenes_dir=Path(scenes_dir),
output_data_dir=Path(output_data_dir),
arcticdem_dir=Path(arctic_dem_dir),
model_dir=Path(model_dir),
tcvis_dir=Path(tcvis_dir),
ee_project='pdg-project-406720')

run_native_planet_pipeline(orthotiles_dir=Path(orthotiles_dir),
scenes_dir=Path(scenes_dir),
output_data_dir=Path(output_data_dir),
arcticdem_slope_vrt=Path(arcticdem_slope_vrt),
arcticdem_elevation_vrt=Path(arcticdem_elevation_vrt),
model_dir=Path(model_dir),
tcvis_dir=Path(tcvis_dir),
ee_project='pdg-project-406720')

Check failure on line 30 in darts/src/darts/run_native.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (E111)

darts/src/darts/run_native.py:30:1: E111 Indentation is not a multiple of 4

Check failure on line 30 in darts/src/darts/run_native.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (E113)

darts/src/darts/run_native.py:30:1: E113 Unexpected indentation

Check failure on line 30 in darts/src/darts/run_native.py

View workflow job for this annotation

GitHub Actions / lint

Ruff (W292)

darts/src/darts/run_native.py:30:64: W292 No newline at end of file
7 changes: 6 additions & 1 deletion darts/src/darts/utils/cuda.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,12 @@ def decide_device(device: Literal["cuda", "cpu", "auto"] | int | None) -> Litera
# We can't provide a default value for device in the parameter list because then we would need to import torch at
# top-level, which would make the CLI slow.
if device is None:
device = "cuda" if torch.cuda.is_available() and has_cuda_and_cupy() else "cpu"
if torch.cuda.is_available() and has_cuda_and_cupy():
device = "cuda"
elif torch.backends.mps.is_available():
device = "mps"
else:
device = "cpu"
logger.info(f"Device not provided. Using {device}.")
return device

Expand Down
Empty file added environment.yml
Empty file.
10 changes: 10 additions & 0 deletions planet_config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[darts]
orthotiles-dir = "/home/toddn/pdg/darts-nextgen/data/input/planet/PSOrthoTile"
scenes-dir = "/home/toddn/pdg/darts-nextgen/data/input/planet/PSScene"
arcticdem-dir = "/home/toddn/pdg/darts-nextgen/rts_dataset01/input/ArcticDEM"
arcticdem-slope-vrt = "/home/toddn/pdg/darts-nextgen/rts_dataset01/input/ArcticDEM/slope.vrt"
arcticdem-elevation-vrt = "/home/toddn/pdg/darts-nextgen/rts_dataset01/input/ArcticDEM/elevation.vrt"
output-data-dir = "/home/toddn/pdg/darts-nextgen/rts_dataset01/output"
model-dir = "/home/toddn/darts-nextgen/rts_dataset01/models"
tcvis-dir= "/home/toddn/pdg/darts-nextgen/tcvis_dir"
ee-project = "pdg-project-406720"
Loading