Make PyPi packages compliant with PEP 625

[amaltaro]

Impact of the new feature
WMCore in general

Is your feature request related to a problem? Please describe.
I just received the following email [1] reporting that some of our WMCore package names are not compliant with PEP 625, which will soon start being enforced by PyPi.

Describe the solution you'd like
From what I read, we need to replace - sign by _ in the package names, for all packages uploaded to PyPi (global-workqueue, reqmgr2ms-pileup, etc etc).

Describe alternatives you've considered
No date was provided, but we better do it sooner than later.

Additional context
[1]
"""
This email is notifying you of an upcoming deprecation that we have determined may affect you as a result of your recent upload to 'reqmgr2ms-transferor'.

In the future, PyPI will require all newly uploaded source distribution filenames to comply with PEP 625. Any source distributions already uploaded will remain in place as-is and do not need to be updated.

Specifically, your recent upload of 'reqmgr2ms-transferor-2.3.8rc5.tar.gz' is incompatible with PEP 625 because it does not contain the normalized project name 'reqmgr2ms_transferor'.

In most cases, this can be resolved by upgrading the version of your build tooling to a later version that supports PEP 625 and produces compliant filenames.
"""

[vkuznet]

From my understanding the steps which will be required to fix this issue are the following:

• change requirements.txt and replace - with _ in WM specific package names, e.g. we need to change the following line:

pymongo~=4.2.0                # wmcore,wmagent-devtools,reqmgr2ms-unmerged,reqmgr2ms-output,reqmgr2ms-pileup

to

pymongo~=4.2.0                # wmcore,wmagent_devtools,reqmgr2ms_unmerged,reqmgr2ms_output,reqmgr2ms_pileup

• modify .github/workflows/pypi_build_and_images.yaml and adjust values in target list of strategy matrix, e.g.

        target: [wmagent, wmagent-devtools, wmcore, reqmon, reqmgr2, global-workqueue, acdcserver, reqmgr2ms-unmerged,
                 reqmgr2ms-output, reqmgr2ms-pileup, reqmgr2ms-rulecleaner, reqmgr2ms-transferor, reqmgr2ms-monitor]

to

        target: [wmagent, wmagent_devtools, wmcore, reqmon, reqmgr2, global_workqueue, acdcserver, reqmgr2ms_unmerged,
                 reqmgr2ms_output, reqmgr2ms_pileup, reqmgr2ms_rulecleaner, reqmgr2ms_transferor, reqmgr2ms_monitor]

• ensure that GitHub Action will properly build packages
• verify that new packages will be uploaded to PyPI
• change CMSKubernetes/docker/pypi areas to use underscores
• modify all Dockerfiles in CMSKubernetes/docker/pypi area to use new notations
• build new images with new names

@amaltaro, am I missing something?

[amaltaro]

Yes, I think you covered it well. In general, we have to cover this new convention in:

1. packaging and publishing to pypi
2. deploying it in Dockerfiles
3. deploying it in virtual environments
4. CMSKubernetes can likely remain as is - such that it does not impose any needs on extra work from the HTTP team

This can affect either manual or automatic mechanisms (e.g. GH pipelines).

In addition, I don't know if this file is used: https://github.com/dmwm/WMCore/blob/master/tools/build_pypi_packages.sh#L17, if it is, then we need to update it as well.
And perhaps this one too: https://github.com/dmwm/WMCore/blob/master/setup_dependencies.py

FYI @belforte @LinaresToine this might affect you as well

[belforte]

thanks Alan. I think CRAB is not affected because we do not really use pypi but build our CRABServer on top of your docker image registry.cern.ch/cmsweb/wmagent-base:pypi-20230705 . Maybe we should use a newer one, but that's a different story. For TW and Publisher we simply start from Pytho3.8 image and pull code from GH. "pip install crab_tw" was too much for our skills.

[vkuznet]

@amaltaro I found major issue with constrains on GH action and python package itself which will not allowed this transition. Full details of my investigation can be found in #12233 (comment) and #12233 (comment)

Or, to avoid this mess we should abandon usage of dashes and underscores in python packages, e.g. use reqmgr2mspileup as pypi package and also build corresponding tar-ball, e.g. reqmgr2mspileup-2.3.8rc13.tar.gz. This will work fine with existing PEPs and requirement from PEP-625, but as you can see the names will be collapsed and I don't know if it is desired (as current names are used not only for tar-balls, pypi packages but also for images, service env-points, etc.). I think it is worth to discuss this at our regular meeting to decide which direction (if any) we should go.