regex library depedency is causing installation problems

[ssbarnea]

Context

MacOS arm64 python 3.12 or newer using uv instead of pip to install dependencies.

I raised this bug on mkdocs-material in order to re-evaluate the need for keeping this a direct dependency. If we are not able to find an alternative, we should at least consider making it a soft-dependency as it would be much easier to not use the search plugin that not being able to use mkdocs-material.

Bug description

AFAIK, regex library is only used inside the search plugin but is listed as a permanent dependency. Apparently bugs like mrabarnett/mrab-regex#413 and mrabarnett/mrab-regex#547 show that this binary dependency is problematic as it causes installation failures for a notable number of users.

This dependency is not using PEP-517 packaging.

Related links

• ModuleNotFoundError: No module named 'regex._regex' mrabarnett/mrab-regex#413
• incomplete regex install when using uv (linux and macos) mrabarnett/mrab-regex#547

Reproduction

Steps to reproduce

> uv pip install mkdocs-material
> python3 -m "import regex"
ModuleNotFoundError: No module named 'regex'

Browser

No response

Before submitting

• I have read and followed the bug reporting guidelines.
• I have attached links to the documentation, and possibly related issues and discussions.
• I assure that I have removed all customizations before submitting this bug report.
• I have attached a .zip file with a minimal reproduction using the built-in info plugin.

[squidfunk]

Thanks for reporting. We need regex for Chinese search. I'm not sure how this could be made a "soft dependency" (I think you mean optional?), as search is a central plugin. pip is still the gold standard, so it sounds more like a problem with uv. If we can find another way to check for Chinese characters, we might swap it out. Other than that, we could only move it to an optional dependency in the next major release, since this is a breaking change for all Chinese users.

[facelessuser]

I would agree that if it works with pip, the standard, and not uv that maybe the problem is with uv. Regardless of uv being preferred by some. But maybe I don't have a full understanding of the real problem.

[alexvoss]

should this work?

python3 -m "import regex"

would that not be looking for a module "import regex"?

I installed regex as described in a venv and running "import regex" at the Python prompt works fine. Material also seems to work fine, thought I only tried a toy project. edit: replaced the -m in the command above with -c and it works as well.

[facelessuser]

I use regex all the time. I install it on various machines via pip and I never have had problem with import regex. I also don't heavily use uv, I think I've played with it, but I still rely on pip mainly.

[ssbarnea]

We don't need to make a decision right away, especially as it will take few days to figure out exactly what is causing this issue. Apparently is not uv fault. Now I suspect the way python is compiled/installed or the non standard packaging of regex wheel. -- I find it bit fascinating because it seems to be a recurring issue (first report ~5 years old), even black was affected by it psf/black#1207 but they sorted it by removing it.

I don't publish anything in Chinese and I bet that people that do are less than 10-20% of mkdocs-material users. This would make it a good case for adding this as an extra. Whoever needs that functionality could install mkdocs[chinese] (or whatever name we decide to expose this feature as). And yes, I can understand that this would be breaking change.

[facelessuser]

If the exact problem could be quantified, a PR could be issued over at regex. I don't think anyone knows exactly what the issue is....

[squidfunk]

I don't publish anything in Chinese and I bet that people that do are less than 10-20% of mkdocs-material users. This would make it a good case for adding this as an extra. Whoever needs that functionality could install mkdocs[chinese] (or whatever name we decide to expose this feature as). And yes, I can understand that this would be breaking change.

This is exactly the argument that is a non-argument. 10-20% is a lot, and yes, Material is very popular among Chinese users, since it is one of the few SSGs that supports Chinese search to a reasonable degree, albeit it's still not perfect. Chinese is the second most spoken language in the world, so there's quite a lot of users out there. We do not want to make it anymore complicated, and as I mentioned, it would mean a breaking change which we're currently not planning for.

For this reason, we're currently not considering this change. I hope for your understanding.

[kamilkrzyskow]

Hello @ssbarnea,
have you debugged where does uv install the packages and what executable does python3 actually run? Or what are the visible site-packages during Python execution?

I tested Windows 10 + Python 3.13 + venv + uv and it works for me.
I tested Alpine Linux + Python 3.11 + venv + uv and it works for me.

In the other link you gave this example:

> uv pip install regex==2024.9.11
> python3 -c "import regex._regex as _regex"
ModuleNotFoundError: No module named 'regex._regex'

> uv pip uninstall regex
> pip install regex==2024.9.11
> python3 -c "import regex._regex as _regex"
success

This gave me the idea that perhaps uv and python3 use different Python instances, classic Python shenanigans 😅

(test-uv) :~$ uv pip tree --verbose
DEBUG uv 0.4.30
DEBUG Searching for default Python interpreter in system path
DEBUG Found `cpython-3.11.9-linux-x86_64-musl` at `/home/frog/test-uv/bin/python3` (active virtual environment)
Using Python 3.11.9 environment at test-uv
mkdocs-material v9.5.44
# ...

(test-uv) :~$ python3 -c "import sys; print(*sys.path, sep='\n'); print(sys.executable)"

/usr/lib/python311.zip
/usr/lib/python3.11
/usr/lib/python3.11/lib-dynload
/home/frog/test-uv/lib/python3.11/site-packages
/home/frog/test-uv/bin/python3

Make sure the paths match between commands and there is no weird path to another version.
Also check these commands:

$ uv pip show regex
Location: /home/frog/test-uv/lib/python3.11/site-packages
$ python3 -m pip show regex
Location: /home/frog/test-uv/lib/python3.11/site-packages

$ ls -la /home/frog/test-uv/lib/python3.11/site-packages/regex
total 2368
drwxr-sr-x    3 frog     frog          4096 Nov  7 04:42 .
drwxr-sr-x   67 frog     frog          4096 Nov  7 04:42 ..
-rw-r--r--    2 frog     frog            65 Nov  7 04:42 __init__.py
drwxr-sr-x    2 frog     frog          4096 Nov  7 04:42 __pycache__
-rwxr-xr-x    2 frog     frog       2003488 Nov  7 04:42 _regex.cpython-311-x86_64-linux-musl.so #Compiled regex._regex file unique for each OS variant
-rw-r--r--    2 frog     frog        141028 Nov  7 04:42 _regex_core.py
-rw-r--r--    2 frog     frog         32683 Nov  7 04:42 regex.py
-rw-r--r--    2 frog     frog        222040 Nov  7 04:42 test_regex.py

---

I didn't read much of it but here I see version 3.9 at the start of the code block and the traceback errors paths are from python3.8 🤔 Very strange indeed 🔍

[kamilkrzyskow]

This was tracked down to be an issue with uv, with pinning the python version, resulting in the installation of incorrect wheel packages.

• uv pip install regex reports installation success but library cannot be imported (pip works) astral-sh/uv#8865 (comment)

I had a hunch we were dealing with a mixed bag of issues and the error in the CI run confirmed it:

• https://github.com/ansible/tox-ansible/actions/runs/11705926832/job/32601848111?pr=389#step:9:91

  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/material/plugins/search/plugin.py", line 24, in <module>
    import regex as re
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/regex.py", line 417, in <module>
    import regex._regex_core as _regex_core
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ModuleNotFoundError: No module named 'regex._regex'

The regex module was found site-packages/regex but the import chain errors out at import regex._regex as _regex.
I assumed before, that the compiled module was missing altogether, but it turned out it's a version mismatch for the compiled Python version.

Crazy edge case shenanigans 🤯

[facelessuser]

Looks like Occam's razor turned out to be correct in this case. Problem installing package with uv? It's probably a problem with uv 🙂.