debug_bb5.md

Debugging pull requests on BlueBrain5

Build failures

Build failures should be found under the tab "Tests" in the GitLab pipeline interface that is linked from the GitHub pull request.

Future changes to the deployment may list build failures directly and/or include build logs.

Concretization failures

If a pull request against the current state of the deployment fails during concretization, the following steps may help debug issues.

1. Check out the pull request in your local Spack setup, and make sure that you sourced your local Spack clone.

   Set temporary root directory on BB5 if not setup already:

   export TMPDIR=${TMPDIR-/tmp}

   Then unset or reset any variables that may interfere with the PR building:

   unset SPACK_SYSTEM_CONFIG_PATH
   export SPACK_USER_CACHE_PATH=${TMPDIR}/debug_spack_pr_$(whoami)_cache

2. Note the job that is failing, i.e., libraries in the applications stage, with GitLab job id 140355.

   Copy its configuration to different location and use it:

   export SPACK_USER_CONFIG_PATH=${TMPDIR}/debug_spack_pr_$(whoami)_config
   
   export FAILED_JOB=140355
   export STUB=/gpfs/bbp.cscs.ch/ssd/gitlab_map_jobs/bbpcihpcdeploy
   cp -r ${STUB}/J${FAILED_JOB}/hpc/spack/spack_config ${SPACK_USER_CONFIG_PATH}

   Then set the installation directory to a temporary writable one:

   spack config add config:install_tree:root:${TMPDIR}/debug_spack_pr_$(whoami)_software
   spack config add config:module_roots:tcl:${TMPDIR}/debug_spack_pr_$(whoami)_modules

3. Recreate the environment used to build in the CI and activate it. Recall that applications was failing in libraries:

   spack env create pr_debug ${SPACK_ROOT}/bluebrain/deployment/environments/applications_libraries.yaml
   spack env activate pr_debug

4. Trigger the concretization:

   spack concretize -f

5. If debugging a build is desired, install the environment:

   spack install

6. Fix any issue arising, re-iterate starting from 4. The FAQ may help with debugging certain issues.

7. Clean everything up:

   rm -r ${TMPDIR}/debug_spack_pr_$(whoami)_cache
   rm -r ${TMPDIR}/debug_spack_pr_$(whoami)_config
   rm -r ${TMPDIR}/debug_spack_pr_$(whoami)_modules
   rm -r ${TMPDIR}/debug_spack_pr_$(whoami)_software

   Finally, to ensure a clean reset of your shell environment, close your SSH connection to BlueBrain5.