Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update build system to support latest upstream spt3g #192

Draft
wants to merge 36 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6120483
Update build system to support latest upstream spt3g
tskisner Nov 21, 2024
cedcaf6
Fix cmake build of compiled extension. Fix use of upstream Quat clas…
tskisner Nov 21, 2024
31077b3
Update wheel build scripts. Add pyproject.toml file.
tskisner Nov 21, 2024
cc18b2f
work in progress
tskisner Nov 27, 2024
bac844d
Work on cleaning up build files
tskisner Jan 6, 2025
ba2e3b4
Fix numpy 2 API problems. Get pyproject.toml + setup.py working. Re…
tskisner Jan 7, 2025
db236c0
Specify python executable in setup.py
tskisner Jan 7, 2025
01bb568
Restore overrides to cmake find python, which apparently is still nee…
tskisner Jan 7, 2025
6f3793f
Manually set the numpy include path. Update repair wheel wrapper to …
tskisner Jan 7, 2025
7823b18
Attempt to fix numpy-2 header include path in docker builds
tskisner Jan 7, 2025
be9212f
Another attempt at cmake builds in docker
tskisner Jan 7, 2025
486cb7d
Add new test workflow that uses conda for dependencies.
tskisner Jan 7, 2025
1b81a92
Fix cmake variables. Add file with conda dev requirements.
tskisner Jan 7, 2025
3b4af63
Add dependabot workflow
tskisner Jan 7, 2025
f11c8b7
Rename github workflows for consistency
tskisner Jan 7, 2025
69c937e
Rename github workflows for consistency between test and deploy actions.
tskisner Jan 7, 2025
af4df42
Fix typo
tskisner Jan 7, 2025
fde597d
In pyproject.toml, depend on the same numpy version used by astropy t…
tskisner Jan 7, 2025
b4ecf01
Remove use of root_dir in glob, for compatibility with python-3.9
tskisner Jan 9, 2025
6664258
Remove distutils from setup.py
tskisner Jan 9, 2025
78eec13
For backwards compatibility, export all symbols from compiled extensi…
tskisner Jan 9, 2025
e9dfc3d
Fix python 3.9 timezone alias
tskisner Jan 9, 2025
060fc77
Expand conda test matrix
tskisner Jan 9, 2025
6201c7c
Attempt to fix clang compiler warning / error
tskisner Jan 9, 2025
3a0b8a3
Remove 3.13 from build matrix until all dependencies support it. Upd…
tskisner Jan 9, 2025
7ea82f5
More fixes for int64->32 casting in healpix code. Temporarily enable…
tskisner Jan 9, 2025
3dbac25
Use main requirements file for cibuildwheel deps.
tskisner Jan 9, 2025
ca9ff49
Fix library location for repairing wheels
tskisner Jan 13, 2025
69ae189
More macos compile fixes
tskisner Jan 14, 2025
0a109c4
More debugging of imports in bundled spt3g
tskisner Jan 15, 2025
2b19888
Remove static linking patch. Attempt to fix wheel repair with explic…
tskisner Jan 15, 2025
437b1e5
Re-enable conda tests for debugging.
tskisner Jan 16, 2025
02f64c2
Avoid linking to boost libraries directly, since these should be brou…
tskisner Jan 22, 2025
846e78b
Add workflow dispatch to conda tests
tskisner Jan 22, 2025
98971ea
Add workflow dispatch to wheel tests
tskisner Jan 22, 2025
a5b0aa7
Make a temporary copy of wheel test workflow to enable running from b…
tskisner Jan 22, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Please see the documentation for all configuration options:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates

version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Build Official Docker Images
name: Deploy Official Docker Images

on:
push:
Expand All @@ -12,14 +12,10 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
# Fetch all history for all tags and branches
with:
fetch-depth: 0
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8

# Build
- name: Build docker images
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Deploy tagged releases.

name: Deploy Release
name: Deploy Release Wheels

on:
push:
Expand All @@ -17,7 +17,7 @@ concurrency:

jobs:
build_wheels:
name: Build wheel for cp${{ matrix.python }}-${{ matrix.builder }}_${{ matrix.arch }}
name: Deploy for cp${{ matrix.python }}-${{ matrix.builder }}_${{ matrix.arch }}
runs-on: ${{ matrix.os }}
strategy:
# Ensure that a wheel builder finishes even if another fails. Useful for
Expand All @@ -26,10 +26,6 @@ jobs:
matrix:
include:
# Linux 64 bit manylinux
- os: ubuntu-latest
arch: x86_64
python: 38
builder: manylinux
- os: ubuntu-latest
arch: x86_64
python: 39
Expand All @@ -42,6 +38,10 @@ jobs:
arch: x86_64
python: 311
builder: manylinux
- os: ubuntu-latest
arch: x86_64
python: 312
builder: manylinux

# MacOS x86_64. The macos-13 runner is the last
# Intel-based runner version. At some point we'll
Expand All @@ -50,28 +50,66 @@ jobs:
arch: x86_64
python: 39
builder: macosx
deploy: 13.0
- os: macos-13
arch: x86_64
python: 310
builder: macosx
deploy: 13.0
- os: macos-13
arch: x86_64
python: 311
builder: macosx
deploy: 13.0
- os: macos-13
arch: x86_64
python: 312
builder: macosx
deploy: 13.0

# MacOS arm64
- os: macos-latest
arch: arm64
python: 310
builder: macosx
deploy: 14.0
- os: macos-latest
arch: arm64
python: 311
builder: macosx
deploy: 14.0
- os: macos-latest
arch: arm64
python: 312
builder: macosx
deploy: 14.0
env:
CIBW_BUILD: cp${{ matrix.python }}-${{ matrix.builder }}_${{ matrix.arch }}
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.builder }}2014
CIBW_MANYLINUX_I686_IMAGE: ${{ matrix.builder }}2014
CIBW_TEST_SKIP: "*-macosx_arm64"
CIBW_BUILD_VERBOSITY: 3
CIBW_ENVIRONMENT_LINUX: "CC=gcc CXX=g++ CFLAGS='-O3 -fPIC' CXXFLAGS='-O3 -fPIC -std=c++14' BOOST_ROOT=/usr/local"
CIBW_ENVIRONMENT_MACOS: "MACOSX_DEPLOYMENT_TARGET=13.0 CC=gcc-14 CXX=g++-14 CFLAGS='-O3 -fPIC' CXXFLAGS='-O3 -fPIC -std=c++14' BOOST_ROOT=/usr/local FLAC_ROOT=/usr/local BLAS_LIBRARIES=/usr/local/lib/libscipy_openblas.dylib"
CIBW_ENVIRONMENT_LINUX: >
CC=gcc
CXX=g++
CFLAGS='-O3 -fPIC'
CXXFLAGS='-O3 -fPIC -std=c++17'
BOOST_ROOT=/usr/local
BLAS_LIBRARIES='-L/usr/local/lib -lopenblas -fopenmp -lm -lgfortran'
CIBW_ENVIRONMENT_MACOS: >
MACOSX_DEPLOYMENT_TARGET=${{ matrix.deploy }}
CC=clang
CXX=clang++
CFLAGS='-O3 -fPIC'
CXXFLAGS='-O3 -fPIC -std=c++17 -stdlib=libc++'
BOOST_ROOT=/usr/local
FLAC_ROOT=/usr/local
BLAS_LIBRARIES='/usr/local/lib/libopenblas.dylib'
CIBW_BEFORE_BUILD_LINUX: ./wheels/install_deps_linux.sh
CIBW_BEFORE_BUILD_MACOS: ./wheels/install_deps_osx.sh ${{ matrix.builder }}_${{ matrix.arch }}
CIBW_BEFORE_BUILD_MACOS: ./wheels/install_deps_osx.sh
CIBW_REPAIR_WHEEL_COMMAND_LINUX: ./wheels/repair_wheel_linux.sh {dest_dir} {wheel}
CIBW_REPAIR_WHEEL_COMMAND_MACOS: ./wheels/repair_wheel_macos.sh {dest_dir} {wheel} {delocate_archs}
CIBW_BEFORE_TEST: export OMP_NUM_THREADS=2
CIBW_TEST_REQUIRES: pytest
CIBW_TEST_REQUIRES: pytest pixell
CIBW_TEST_COMMAND: >
python -c 'import so3g.smurf.reader; from spt3g import core' &&
python -m pytest {package}/test
Expand All @@ -82,16 +120,16 @@ jobs:
- uses: actions/setup-python@v5
name: Install Python
with:
python-version: '3.10'
python-version: '3.12'

- name: Install cibuildwheel
run: |
python -m pip install twine cibuildwheel==2.20.0
python -m pip install twine cibuildwheel==2.22.0

- name: Build wheel
run: |
python -m cibuildwheel --output-dir wheelhouse

- name: Upload to PyPI
run: |
python -m twine upload wheelhouse/so3g*cp${{ matrix.python }}-${{ matrix.builder }}*.whl
python -m twine upload wheelhouse/so3g*cp${{ matrix.python }}-${{ matrix.builder }}*${{ matrix.arch }}*.whl
95 changes: 95 additions & 0 deletions .github/workflows/test_conda.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# In general, we try to run on:
# - The oldest supported python
# - The latest stable python that is the common default on most systems and conda
# - (During transitions) The newly released bleeding edge python

name: Run Tests with Conda

on:
workflow_dispatch:
push:
branches:
- master
pull_request:
branches:
- master

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
name: Python-${{ matrix.python }} on ${{ matrix.arch }}
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python: "3.9"
arch: Linux-x86_64
- os: ubuntu-latest
python: "3.11"
arch: Linux-x86_64
- os: ubuntu-latest
python: "3.12"
arch: Linux-x86_64
- os: macos-latest
python: "3.10"
arch: MacOSX-x86_64
- os: macos-latest
python: "3.12"
arch: MacOSX-x86_64
- os: macos-latest
python: "3.10"
arch: MacOSX-arm64
- os: macos-latest
python: "3.12"
arch: MacOSX-arm64
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Conda Base
run: |
sudo rm -rf /usr/share/miniconda \
&& sudo rm -rf /usr/local/miniconda \
&& curl -SL -o miniforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-${{ matrix.arch }}.sh \
&& bash miniforge.sh -b -f -p ~/conda \
&& source ~/conda/etc/profile.d/conda.sh \
&& conda activate base \
&& conda update -n base --yes conda

- name: Check Conda Config
run: |
source ~/conda/etc/profile.d/conda.sh \
&& conda activate base \
&& conda info \
&& conda list \
&& conda config --show-sources \
&& conda config --show

- name: Install Dependencies
run: |
source ~/conda/etc/profile.d/conda.sh \
&& conda create --yes -n test python=${{ matrix.python }} \
&& conda activate test \
&& conda install --yes --file conda_dev_requirements.txt \
&& pip install -r test-requirements.txt

- name: Install
run: |
source ~/conda/etc/profile.d/conda.sh \
&& conda activate test \
&& python3 -m pip install .

- name: Run Tests
run: |
source ~/conda/etc/profile.d/conda.sh \
&& conda activate test \
&& export OMP_NUM_THREADS=2 \
&& pytest ./test
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
name: Run Tests
name: Run Tests with Docker

# on: workflow_dispatch
on:
push:
branches-ignore: [ master ]
pull_request:
branches:
- master
# pull_request:
# branches:
# - master

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -15,11 +19,7 @@ jobs:

steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8
uses: actions/checkout@v4

- name: Build docker images
run: |
Expand Down
Loading
Loading