Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix docker cache DEMO #1073

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Fix docker cache DEMO #1073

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a9dc3e5
try
andarut Aug 20, 2024
a945855
reload
andarut Aug 20, 2024
aab9690
update cache
andarut Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
219 changes: 110 additions & 109 deletions .github/workflows/Build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: build
on:
pull_request:
branches:
- master
- master

env:
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: true
Expand Down Expand Up @@ -42,82 +42,83 @@ jobs:
name: "${{matrix.os}}/${{matrix.compiler}}/c++${{matrix.cpp}}/asan=${{matrix.asan}}/ubsan=${{matrix.ubsan}}"

steps:
- uses: actions/checkout@v3

- name: Get polyfills repo
uses: actions/checkout@v3
with:
repository: 'VKCOM/kphp-polyfills'
path: 'kphp-polyfills'

- name: Cache docker image
uses: actions/cache@v3
id: docker-image-cache
with:
path: kphp-build-env-${{matrix.os}}.tar
key: docker-image-cache-${{matrix.os}}-${{ hashFiles('.github/workflows/Dockerfile.*', 'tests/python/requirements.txt') }}

- name: Build and save docker image
if: steps.docker-image-cache.outputs.cache-hit != 'true'
run: |
docker build -f $GITHUB_WORKSPACE/.github/workflows/Dockerfile.${{matrix.os}} $GITHUB_WORKSPACE \
-t kphp-build-img-${{matrix.os}} \
--cache-from=type=local,src=kphp-build-img-${{matrix.os}}-cache
docker tag kphp-build-img-${{matrix.os}} kphp-build-img-${{matrix.os}}-cache
docker save kphp-build-img-${{matrix.os}}-cache -o kphp-build-env-${{matrix.os}}.tar

- name: Load docker image from cache
if: steps.docker-image-cache.outputs.cache-hit == 'true'
run: docker load --input kphp-build-env-${{matrix.os}}.tar

- name: Start docker container
run: |
docker run -dt --name kphp-build-container-${{matrix.os}} kphp-build-img-${{matrix.os}}-cache
docker cp $GITHUB_WORKSPACE/. kphp-build-container-${{matrix.os}}:${{env.kphp_root_dir}}

- name: Add git safe directory
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"git config --global --add safe.directory ${{env.kphp_root_dir}}"

- name: Build all
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"cmake -DCMAKE_CXX_COMPILER=${{matrix.compiler}} -DCMAKE_CXX_STANDARD=${{matrix.cpp}} -DADDRESS_SANITIZER=${{matrix.asan}} -DUNDEFINED_SANITIZER=${{matrix.ubsan}} -DPDO_DRIVER_MYSQL=ON -DPDO_DRIVER_PGSQL=ON -DPDO_LIBS_STATIC_LINKING=ON -S ${{env.kphp_root_dir}} -B ${{env.kphp_build_dir}} && make -C ${{env.kphp_build_dir}} -j$(nproc) all"

- name: Run unit tests
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"make -C ${{env.kphp_build_dir}} -j$(nproc) test"

- name: Compile dummy PHP script
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"cd ${{env.kphp_build_dir}} && echo 'hello world' > demo.php && ${{env.kphp_root_dir}}/objs/bin/kphp2cpp --cxx ${{matrix.compiler}} demo.php && kphp_out/server -o --user kitten"

- name: Polyfills composer install
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"composer install -d ${{env.kphp_polyfills_dir}}"

- name: Run python tests
id: python_tests
continue-on-error: true
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"chown -R kitten /home && su kitten -c 'GITHUB_ACTIONS=1 KPHP_TESTS_POLYFILLS_REPO=${{env.kphp_polyfills_dir}} KPHP_CXX=${{matrix.compiler}} python3.7 -m pytest --tb=native -n$(nproc) ${{env.kphp_root_dir}}/tests/python/'"

- name: Prepare python tests artifacts
if: steps.python_tests.outcome == 'failure'
run: docker cp kphp-build-container-${{matrix.os}}:${{env.kphp_root_dir}}/tests/python/_tmp/ ${{runner.temp}} &&
rm -rf ${{runner.temp}}/_tmp/*/working_dir

- name: Upload python tests artifacts
uses: actions/upload-artifact@v3
if: steps.python_tests.outcome == 'failure'
with:
path: ${{runner.temp}}/_tmp/

- name: Fail pipeline if python tests failed
if: steps.python_tests.outcome == 'failure'
run: exit 1

- name: Remove docker container
run: docker rm -f kphp-build-container-${{matrix.os}}
- uses: actions/checkout@v3

- name: Get polyfills repo
uses: actions/checkout@v3
with:
repository: "VKCOM/kphp-polyfills"
path: "kphp-polyfills"

# - name: Cache docker image
# uses: actions/cache@v3
# id: docker-image-cache
# with:
# path: kphp-build-env-${{matrix.os}}.tar
# key: docker-image-cache-${{matrix.os}}-${{ hashFiles('.github/workflows/Dockerfile.*', 'tests/python/requirements.txt') }}

- name: Build and save docker image
if: steps.docker-image-cache.outputs.cache-hit != 'true'
run: |
docker build -f $GITHUB_WORKSPACE/.github/workflows/Dockerfile.${{matrix.os}} $GITHUB_WORKSPACE \
-t kphp-build-img-${{matrix.os}} \
--cache-from=type=local,src=kphp-build-img-${{matrix.os}}-cache
docker tag kphp-build-img-${{matrix.os}} kphp-build-img-${{matrix.os}}-cache
docker save kphp-build-img-${{matrix.os}}-cache -o kphp-build-env-${{matrix.os}}.tar

# - name: Load docker image from cache
# if: steps.docker-image-cache.outputs.cache-hit == 'true'
# run: docker load --input kphp-build-env-${{matrix.os}}.tar

- name: Start docker container
run: |
docker run -dt --name kphp-build-container-${{matrix.os}} kphp-build-img-${{matrix.os}}-cache
docker cp $GITHUB_WORKSPACE/. kphp-build-container-${{matrix.os}}:${{env.kphp_root_dir}}

- name: Add git safe directory
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"git config --global --add safe.directory ${{env.kphp_root_dir}}"

- name: Build all
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"cmake -DCMAKE_CXX_COMPILER=${{matrix.compiler}} -DCMAKE_CXX_STANDARD=${{matrix.cpp}} -DADDRESS_SANITIZER=${{matrix.asan}} -DUNDEFINED_SANITIZER=${{matrix.ubsan}} -DPDO_DRIVER_MYSQL=ON -DPDO_DRIVER_PGSQL=ON -DPDO_LIBS_STATIC_LINKING=ON -S ${{env.kphp_root_dir}} -B ${{env.kphp_build_dir}} && make -C ${{env.kphp_build_dir}} -j$(nproc) all"

- name: Run unit tests
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"make -C ${{env.kphp_build_dir}} -j$(nproc) test"

- name: Compile dummy PHP script
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"cd ${{env.kphp_build_dir}} && echo 'hello world' > demo.php && ${{env.kphp_root_dir}}/objs/bin/kphp2cpp --cxx ${{matrix.compiler}} demo.php && kphp_out/server -o --user kitten"

- name: Polyfills composer install
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"composer install -d ${{env.kphp_polyfills_dir}}"

- name: Run python tests
id: python_tests
continue-on-error: true
run: docker exec kphp-build-container-${{matrix.os}} bash -c
"chown -R kitten /home && su kitten -c 'GITHUB_ACTIONS=1 KPHP_TESTS_POLYFILLS_REPO=${{env.kphp_polyfills_dir}} KPHP_CXX=${{matrix.compiler}} python3.7 -m pytest --tb=native -n$(nproc) ${{env.kphp_root_dir}}/tests/python/'"

- name: Prepare python tests artifacts
if: steps.python_tests.outcome == 'failure'
run:
docker cp kphp-build-container-${{matrix.os}}:${{env.kphp_root_dir}}/tests/python/_tmp/ ${{runner.temp}} &&
rm -rf ${{runner.temp}}/_tmp/*/working_dir

- name: Upload python tests artifacts
uses: actions/upload-artifact@v3
if: steps.python_tests.outcome == 'failure'
with:
path: ${{runner.temp}}/_tmp/

- name: Fail pipeline if python tests failed
if: steps.python_tests.outcome == 'failure'
run: exit 1

- name: Remove docker container
run: docker rm -f kphp-build-container-${{matrix.os}}

build-macos:
runs-on: ${{matrix.os}}-12
Expand All @@ -129,36 +130,36 @@ jobs:
cpp: 17

name: "${{matrix.os}}/${{matrix.compiler}}/c++${{matrix.cpp}}"

steps:
- uses: actions/checkout@v3

# because of https://github.com/orgs/Homebrew/discussions/4612
- name: Check Environment
run: |
export HOMEBREW_NO_INSTALL_FROM_API=0
brew untap --force homebrew/cask

- name: Setup Environment
run: |
brew tap shivammathur/php
brew update
brew install re2c cmake coreutils openssl libiconv re2 pcre yaml-cpp zstd googletest shivammathur/php/[email protected]
brew link --overwrite --force shivammathur/php/[email protected]
/usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 -m pip install --upgrade pip --break-system-packages && /usr/local/Frameworks/Python.framework/Versions/3.12/bin/pip3 install --break-system-packages jsonschema

- name: Run cmake
run: cmake -DCMAKE_CXX_COMPILER=${{matrix.compiler}} -DCMAKE_CXX_STANDARD=${{matrix.cpp}} -DDOWNLOAD_MISSING_LIBRARIES=On -S $GITHUB_WORKSPACE -B ${{runner.workspace}}/build

- name: Build all
run: make -C ${{runner.workspace}}/build -j$(nproc) all
- name: Run unit tests
run: make -C ${{runner.workspace}}/build -j$(nproc) test

- name: Compile dummy PHP script
working-directory: ${{runner.workspace}}/build
run: |
echo 'hello world' > demo.php
$GITHUB_WORKSPACE/objs/bin/kphp2cpp --cxx ${{matrix.compiler}} demo.php
kphp_out/server -o
- uses: actions/checkout@v3

# because of https://github.com/orgs/Homebrew/discussions/4612
- name: Check Environment
run: |
export HOMEBREW_NO_INSTALL_FROM_API=0
brew untap --force homebrew/cask

- name: Setup Environment
run: |
brew tap shivammathur/php
brew update
brew install re2c cmake coreutils openssl libiconv re2 pcre yaml-cpp zstd googletest shivammathur/php/[email protected]
brew link --overwrite --force shivammathur/php/[email protected]
/usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 -m pip install --upgrade pip --break-system-packages && /usr/local/Frameworks/Python.framework/Versions/3.12/bin/pip3 install --break-system-packages jsonschema

- name: Run cmake
run: cmake -DCMAKE_CXX_COMPILER=${{matrix.compiler}} -DCMAKE_CXX_STANDARD=${{matrix.cpp}} -DDOWNLOAD_MISSING_LIBRARIES=On -S $GITHUB_WORKSPACE -B ${{runner.workspace}}/build

- name: Build all
run: make -C ${{runner.workspace}}/build -j$(nproc) all

- name: Run unit tests
run: make -C ${{runner.workspace}}/build -j$(nproc) test

- name: Compile dummy PHP script
working-directory: ${{runner.workspace}}/build
run: |
echo 'hello world' > demo.php
$GITHUB_WORKSPACE/objs/bin/kphp2cpp --cxx ${{matrix.compiler}} demo.php
kphp_out/server -o
Loading