ocrd_all native installation fails due to tensorflow #235
Comments
|
That's a known "feature". You need Python 3.7. See https://github.com/OCR-D/ocrd-website/wiki/OCR-D-on-Debian-and-Ubuntu for more information. Or you can restrict your installation to processors without older Tensorflow. |
|
@kba, @bertsky, I think this scenario will happen more often as all recent Linux distributions have newer versions of Python. Maybe we should detect that early in the Makefile and either terminate with a prominent message or continue after such message, but omit everything which requires older versions of Tensorflow. |
We've had that discussion in #147 – where no-one came up with a concrete proposal yet – but what's the point in just throwing a more clever error message, if what we really want is to actually support these platforms? IMO we should invest the time to solve delegation to thin Docker layers instead. (We could even have an automatism which only delegates to Docker if the native recipe failed.) |
|
It seems we could avoid at least some of these problems by relying on the TF 1.15 wheels prebuilt by Nvidia. There may be more. (For example, see here or here or here or here.) Or we do permit ourselves falling for conda – at least for the "sub-venvs" ... |
|
Technically we could add our own OCR-D Python repository and provide missing packages there, something like https://digi.bib.uni-mannheim.de/pypi/. But maintaining that for Tensorflow would be a huge burden. |
That's actually quite straightforward: We don't need to download stuff from https://github.com/NVIDIA/tensorflow, just add the Nvidia server to the list of indexes searched by pip locally, which can be done by: After that, you can do: And get a fresh Python 3.8 compatible TF 1.15.5 – alas, under the wrong distribution name. All our modules depend on either You should think this is an easy task that should be supported by pip/wheel out of the box. Turns out: it isn't! I only found this recipe to create alias packages. In addition, the following seems to work (amazingly): We could easily write a fixture for all our TF1 sub-venvs using this in the makefile. Opinions anyone? |
Agreed. TF builds bind lots of CPU and network resources. If the above Nvidia recipe does not work, we could still try with any of the other prebuilt wheels linked above (which already come under the correct distribution name), and add their URLs to our index via |
pip install nvidia-pyindexDid not know that, sweet. pip download --no-deps nvidia-tensorflow
for name in nvidia_tensorflow-*.whl; do name=${name%.whl}; done
python -m wheel unpack $name.whl
for name in nvidia_tensorflow-*/; do name=${name%/}; done
newname=${name/nvidia_tensorflow/tensorflow_gpu}
sed -i s/nvidia_tensorflow/tensorflow_gpu/ $name/$name.dist-info/METADATA
sed -i s/nvidia_tensorflow/tensorflow_gpu/ $name/$name.dist-info/RECORD
sed -i s/nvidia_tensorflow/tensorflow_gpu/ $name/tensorflow_core/tools/pip_package/setup.py
mv $name/$name.dist-info $name/$newname.dist-info
mv $name/$name.data $name/$newname.data
python -m wheel pack $name
pip install $newname*.whlIt's a hack, sure, but a sensible one I think and better than having to change the |
|
Yes. And among the 4 other links given above, only this had a recent TF1 – but the nvidia repo offers more options. |
|
Note: I edited the above recipe (also in your verbatim copy) to make them work automatically. |
|
Alas, |
Sadly, that wheel, too, depends on GLIBC 2.29. |
see here
see here Not even Anaconda provides the combination TF<2 Python>3.7: |
|
And the combination TF=~2.1 Python>3.7 (which we need for ocrd_anybaseocr) is even more difficult to get by. |
It works now. But it comes with the same caveat as the multi-CUDA solution: you need to update your host system's driver. So at least for TF 1.15 (i.e. our $(pip_install_tf1nvidia)to all recipes involving that venv (just before the normal SHELL = /bin/bash
define pip_install_tf1nvidia =
. $(ACTIVATE_VENV) && if ! $(PIP) show -q tensorflow-gpu; then \
$(PIP) install nvidia-pyindex && \
pushd $$(mktemp -d) && \
$(PIP) download --no-deps nvidia-tensorflow && \
for name in nvidia_tensorflow-*.whl; do name=$${name%.whl}; done && \
$(PYTHON) -m wheel unpack $$name.whl && \
for name in nvidia_tensorflow-*/; do name=$${name%/}; done && \
newname=$${name/nvidia_tensorflow/tensorflow_gpu} &&\
sed -i s/nvidia_tensorflow/tensorflow_gpu/g $$name/$$name.dist-info/METADATA && \
sed -i s/nvidia_tensorflow/tensorflow_gpu/g $$name/$$name.dist-info/RECORD && \
sed -i s/nvidia_tensorflow/tensorflow_gpu/g $$name/tensorflow_core/tools/pip_package/setup.py && \
pushd $$name && for path in $$name*; do mv $$path $${path/$$name/$$newname}; done && popd && \
$(PYTHON) -m wheel pack $$name && \
$(PIP) install $$newname*.whl && popd && rm -fr $$OLDPWD; fi
endef(or in a separate shell script) |
|
I have a working solution to integrate this into Docker build, too. (By passing But we still lack a prebuild for this TF 2.1:
@kba, do you remember why you restricted ocrd_anybaseocr to |
I think because the loading the pretrained models led to |
Unfortunately, I cannot find any discussion/mention of this. |
See OCR-D/ocrd_anybaseocr#89 for a solution in ocrd_anybaseocr and #289 for a general solution including all of the above. |
|
This issue was solved with #289 for Python 3.8, but for Python 3.9 and Python 3.10 there is still the same problem. For those and any newer Python version it is no longer possible to provide a |
Like #219, when installing ocrd_all natively on ubuntu 20.04 and python 3.8.5,
make allfails and returns the following error (full stacktrace below):full log :
The text was updated successfully, but these errors were encountered: