Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CT Open MPI #200

Merged
merged 2 commits into from
Oct 29, 2024
Merged

CT Open MPI #200

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f374fe7
test without Open MPI
mgates3 Oct 28, 2024
071f0dd
more robust test for idle GPUs
mgates3 Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/setup_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,17 @@ ${CC} --version
${FC} --version

#----------------------------------------------------------------- MPI
# Test Open MPI with CPU and CUDA.
# Test Open MPI with CPU and CUDA. --> temporarily use Intel MPI.
# Test Intel MPI with ROCm and SYCL.
# Note: Open MPI hides SYCL devices, at least in our current CI.
if [ "${device}" = "cpu" -o "${device}" = "gpu_nvidia" ]; then
if false; then
#if [ "${device}" = "cpu" -o "${device}" = "gpu_nvidia" ]; then
print "======================================== Load Open MPI"
quiet module load openmpi
export OMPI_CXX=${CXX}
export OMPI_CC=${CC}
export OMPI_FC=${FC}
unset UCX_NET_DEVICES # complains about unavailable interface mlx5_0:1

echo "mkl_blacs = openmpi" >> make.inc
else
Expand Down
8 changes: 5 additions & 3 deletions test/idle_gpus.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,15 @@
if (s):
gpu = s.group( 1 )

# If using > half the memory, assume it is not idle.
# If using >= 10 MiB or 5% utilization, assume it is not idle.
# Typically idle is 1 MiB and 0% utilization.
# Docker can't see processes in section 2.
s = re.search( '^\| +N/A +\d+C +\w+ +\d+W +/ +\d+W *\| +(\d+)MiB +/ +(\d+)MiB', line )
s = re.search( '^\| +N/A +\d+C +\w+ +\d+W +/ +\d+W *\| +(\d+)MiB +/ +(\d+)MiB *\| +(\d+)%', line )
if (s):
used_mem = int( s.group( 1 ) )
total_mem = int( s.group( 2 ) )
if (used_mem > 0.5*total_mem):
percent = int( s.group( 3 ) )
if (used_mem >= 10 or percent >= 5):
gpus[ gpu ] = 0
else:
# Match process lines:
Expand Down
Loading