Skip to content

Commit

Permalink
ci: Fix broken CI for ASWF 2021 and 2022 containers (#1905)
Browse files Browse the repository at this point in the history
Identical in spirit to the coresponding PR for OpenImageIO.

The background here takes some explaining...

The ASWF's prepared Docker containers that reproduce a CentOS7-based
environment corresponding to VFX Platform 2021 and 2022 years contain
a glibc that is too old to run a newer version of "node" -- which
GitHub Actions themselves use. The old one is no longer receiving
support, and GitHub finally changed the policy that was allowing its
GHA runners to still run the old version that worked with those ASWF
containers.  The new node version that is happy on the GHA runners
requires a glibc newer than the one in the containers. So that breaks
all our CI runs in 2021 and 2022 VFX Platform configurations.

See this thread, if you dare:
https://academysoftwarefdn.slack.com/archives/C0169RX7MMK/p1732574400981949

Anyway, Jean-Francois Panisset proposed and tested a fix, which lets
us continue working (for now) by downloading an old node implemenation
(which is happy with the old glibc) and installing it atop the
conainers, so it's this old one that the GH actions will use. It feels
like it's all held together by chewing gum and duct tape, but it
unbreaks our CI and hopefully it will keep working until all the
studios have switched fully to RHEL/Alma/Rocky 9.x and we no longer
care about testing on CentOS-7 based containers or supporting the VFX
Platform years that are too old to be relevant to the new OS that the
studios are using.

Signed-off-by: Larry Gritz <[email protected]>
  • Loading branch information
lgritz authored Nov 29, 2024
1 parent bcb5adb commit 541e870
Showing 1 changed file with 177 additions and 59 deletions.
236 changes: 177 additions & 59 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ permissions: read-all

jobs:

aswf:
aswf-old:
name: "VFX${{matrix.vfxyear}} ${{matrix.desc}}"
strategy:
fail-fast: false
Expand All @@ -43,6 +43,7 @@ jobs:
runner: ubuntu-latest
container: aswftesting/ci-osl:2021-clang11
vfxyear: 2021
old_node: 1
cxx_std: 17
openexr_ver: v3.1.3
openimageio_ver: v2.4.13.0
Expand All @@ -56,6 +57,7 @@ jobs:
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang13
vfxyear: 2022
old_node: 1
cxx_std: 17
openimageio_ver: v2.5.17.0
python_ver: 3.9
Expand All @@ -67,6 +69,7 @@ jobs:
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang12
vfxyear: 2022
old_node: 1
cxx_std: 17
opencolorio_ver: v2.2.1
openimageio_ver: main
Expand All @@ -80,54 +83,140 @@ jobs:
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang13
vfxyear: 2022
old_node: 1
cxx_std: 17
openimageio_ver: release
python_ver: 3.9
pybind11_ver: v2.9.0
simd: sse2
batched: b4_SSE2
- desc: icc/C++17 llvm14 py3.9 oiio-main avx2
nametag: linux-icc
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang14
vfxyear: 2022
cc_compiler: icc
cxx_compiler: icpc
cxx_std: 17
fmt_ver: 7.1.3
opencolorio_ver: v2.3.2
openimageio_ver: v2.5.17.0
# Changes to OIIO's simd.h starting in commit 68666db9 (from PR
# #4187) seem to trigger compiler bugs in icc and generate wrong
# SIMD code. It's probably not worth tracking down for just this
# obsolete compiler. Just lock down to OIIO 2.5 for icc builds to
# avoid the problem.
# openimageio_ver: e41ac03c0b21 # works
# openimageio_ver: 68666db994d5 # broken
python_ver: 3.9
pybind11_ver: v2.9.0
# simd: avx2,f16c
batched: b8_AVX2_noFMA
setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF -DEXTRA_CPP_ARGS=-fp-model=consistent"
OPENIMAGEIO_CMAKE_FLAGS=-DBUILD_FMT_VERSION=7.1.3
USE_OPENVDB=0 OPENCOLORIO_CXX=g++
- desc: icx/C++17 llvm14 py3.9 oiio2.3 avx2
nametag: linux-icx
- desc: oldest everything gcc9/C++17 llvm9 py3.7 oiio2.3 no-simd
nametag: linux-oldest
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang14
vfxyear: 2022
cc_compiler: icx
cxx_compiler: icpx
container: aswftesting/ci-osl:2021-clang11
vfxyear: 2021
old_node: 1
cxx_std: 17
fmt_ver: 7.1.3
opencolorio_ver: v2.3.2
openimageio_ver: main
python_ver: 3.9
pybind11_ver: v2.9.0
simd: avx2,f16c
batched: b8_AVX2_noFMA
setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF" USE_OPENVDB=0
OPENCOLORIO_CXX=g++
openexr_ver: v3.1.0
openimageio_ver: v2.4.13.0
python_ver: 3.7
pybind11_ver: v2.7.0
simd: 0
setenvs: export PUGIXML_VERSION=v1.8
CMAKE_VERSION=3.15.5
ENABLE_OPENVDB=0


runs-on: ${{matrix.runner}}
container:
image: ${{ matrix.container }}
volumes:
- /node20217:/node20217:rw,rshared
- /node20217:/__e/node20:ro,rshared
env:
CXX: ${{matrix.cxx_compiler}}
CC: ${{matrix.cc_compiler}}
CMAKE_CXX_STANDARD: ${{matrix.cxx_std}}
FMT_VERSION: ${{matrix.fmt_ver}}
OPENCOLORIO_VERSION: ${{matrix.opencolorio_ver}}
OPENEXR_VERSION: ${{matrix.openexr_ver}}
OPENIMAGEIO_VERSION: ${{matrix.openimageio_ver}}
PYBIND11_VERSION: ${{matrix.pybind11_ver}}
PYTHON_VERSION: ${{matrix.python_ver}}
USE_BATCHED: ${{matrix.batched}}
USE_SIMD: ${{matrix.simd}}
ABI_CHECK: ${{matrix.abi_check}}
steps:
# Install nodejs 20 with glibc 2.17, to work around the face that the
# GHA runners are insisting on a node version that is too new for the
# glibc in the ASWF containers prior to 2023.
- name: install nodejs20glibc2.17
if: matrix.old_node == '1'
run: |
curl --silent https://unofficial-builds.nodejs.org/download/release/v20.18.1/node-v20.18.1-linux-x64-glibc-217.tar.xz | tar -xJ --strip-components 1 -C /node20217 -f -
# We would like to use harden-runner, but it flags too many false
# positives, every time we download a dependency. We should use it only
# on CI runs where we are producing artifacts that users might rely on.
# - name: Harden Runner
# uses: step-security/harden-runner@248ae51c2e8cc9622ecf50685c8bf7150c6e8813 # v1.4.3
# with:
# egress-policy: audit # TODO: change to 'egress-policy: block' after couple of runs
- uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b # v3.0.2
- name: Prepare ccache timestamp
id: ccache_cache_keys
run: echo "::set-output name=date::`date -u +'%Y-%m-%dT%H:%M:%SZ'`"
- name: ccache
id: ccache
uses: actions/cache@c3f1317a9e7b1ef106c153ac8c0f00fed3ddbc0d # v3.0.4
with:
path: /tmp/ccache
key: ${{github.job}}-${{matrix.nametag}}-${{steps.ccache_cache_keys.outputs.date}}
restore-keys: ${{github.job}}-${{matrix.nametag}}-
- name: Build setup
run: |
${{matrix.setenvs}}
src/build-scripts/ci-startup.bash
- name: Remove existing OpenEXR
if: matrix.openexr_ver != ''
run: |
sudo rm -rf /usr/local/include/OpenEXR
sudo rm -rf /usr/local/lib*/cmake/OpenEXR
sudo rm -rf /usr/local/lib*/libOpenOpenEXR*
sudo rm -rf /usr/local/lib*/python3.9/site-packages/OpenOpenEXR*
- name: Remove existing OIIO
if: matrix.openimageio_ver != ''
run: |
sudo rm -rf /usr/local/include/OpenImageIO
sudo rm -rf /usr/local/lib*/cmake/OpenImageIO
sudo rm -rf /usr/local/lib*/libOpenImageIO*
sudo rm -rf /usr/local/lib*/python3.9/site-packages/OpenImageIO*
- name: Dependencies
run: |
${{matrix.depcmds}}
src/build-scripts/gh-installdeps.bash
- name: Build
run: |
src/build-scripts/ci-build.bash
- name: Testsuite
if: matrix.skip_tests != '1'
run: src/build-scripts/ci-test.bash
- name: Check out ABI standard
if: matrix.abi_check != ''
uses: actions/checkout@3df4ab11eba7bda6032a0b82a6bb43b11571feac # v4.0.0
with:
ref: ${{matrix.abi_check}}
path: abi_standard
- name: Build ABI standard
if: matrix.abi_check != ''
run: |
mkdir -p abi_standard/build
pushd abi_standard
export LD_LIBRARY_PATH="${PWD}/build/lib:${LD_LIBRARY_PATH}"
src/build-scripts/ci-build.bash
popd
- name: Check ABI
if: matrix.abi_check != ''
run: |
src/build-scripts/ci-abicheck.bash ./build abi_standard/build \
liboslexec liboslcomp liboslquery liboslnoise
- uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4.4.3
if: failure()
with:
name: osl-${{github.job}}-${{matrix.nametag}}
path: |
build/cmake-save
build/compat_reports
build/*.cmake
build/CMake*
build/testsuite/*/*.*
aswf:
name: "VFX${{matrix.vfxyear}} ${{matrix.desc}}"
strategy:
fail-fast: false
matrix:
include:
- desc: gcc11/C++17 llvm15 py3.10 oiio-rel avx2
nametag: linux-vfx2023
runner: ubuntu-latest
Expand Down Expand Up @@ -164,33 +253,19 @@ jobs:
simd: avx2,f16c
batched: b8_AVX2
# setenvs: export CONAN_PACKAGES="ptex/2.4.2@aswf/vfx2024"
- desc: oldest everything gcc9/C++17 llvm9 py3.7 oiio2.3 no-simd
nametag: linux-oldest
runner: ubuntu-latest
container: aswftesting/ci-osl:2021-clang11
vfxyear: 2021
cxx_std: 17
openexr_ver: v3.1.0
openimageio_ver: v2.4.13.0
python_ver: 3.7
pybind11_ver: v2.7.0
simd: 0
setenvs: export PUGIXML_VERSION=v1.8
CMAKE_VERSION=3.15.5
ENABLE_OPENVDB=0

# Address and leak sanitizers
- desc: sanitizers
nametag: sanitizer
runner: ubuntu-latest
container: aswftesting/ci-osl:2022-clang13
vfxyear: 2022
container: aswftesting/ci-osl:2024-clang17
vfxyear: 2024
cc_compiler: clang
cxx_compiler: clang++
cxx_std: 17
openimageio_ver: main
python_ver: 3.9
pybind11_ver: v2.9.0
python_ver: "3.11"
pybind11_ver: v2.11.1
simd: avx2,f16c
batched: b8_AVX2
setenvs: export OSL_CMAKE_FLAGS="-DSANITIZE=address -DOSL_TEST_BIG_TIMEOUT=1200"
Expand Down Expand Up @@ -244,6 +319,49 @@ jobs:
OPENIMAGEIO_CMAKE_FLAGS="-DUSE_PYTHON=0"
CMAKE_BUILD_TYPE=RelWithDebInfo

- desc: icc/C++17 llvm14 py3.9 oiio-main avx2
nametag: linux-icc
runner: ubuntu-latest
container: aswftesting/ci-osl:2023-clang15
vfxyear: 2023
cc_compiler: icc
cxx_compiler: icpc
cxx_std: 17
fmt_ver: 7.1.3
opencolorio_ver: v2.3.2
openimageio_ver: v2.5.17.0
# Changes to OIIO's simd.h starting in commit 68666db9 (from PR
# #4187) seem to trigger compiler bugs in icc and generate wrong
# SIMD code. It's probably not worth tracking down for just this
# obsolete compiler. Just lock down to OIIO 2.5 for icc builds to
# avoid the problem.
# openimageio_ver: e41ac03c0b21 # works
# openimageio_ver: 68666db994d5 # broken
python_ver: "3.10"
pybind11_ver: v2.10.0
# simd: avx2,f16c
batched: b8_AVX2_noFMA
setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF -DEXTRA_CPP_ARGS=-fp-model=consistent"
OPENIMAGEIO_CMAKE_FLAGS=-DBUILD_FMT_VERSION=7.1.3
USE_OPENVDB=0 OPENCOLORIO_CXX=g++
- desc: icx/C++17 llvm14 py3.9 oiio2.3 avx2
nametag: linux-icx
runner: ubuntu-latest
container: aswftesting/ci-osl:2023-clang15
vfxyear: 2023
cc_compiler: icx
cxx_compiler: icpx
cxx_std: 17
fmt_ver: 7.1.3
opencolorio_ver: v2.3.2
openimageio_ver: main
python_ver: "3.10"
pybind11_ver: v2.10.0
simd: avx2,f16c
batched: b8_AVX2_noFMA
setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF" USE_OPENVDB=0
OPENCOLORIO_CXX=g++

runs-on: ${{matrix.runner}}
container:
image: ${{ matrix.container }}
Expand Down

0 comments on commit 541e870

Please sign in to comment.