Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable To Install Tensorflow on macOS Big Sur #3566

Closed
3 tasks done
timkofu opened this issue Jan 12, 2021 · 13 comments
Closed
3 tasks done

Unable To Install Tensorflow on macOS Big Sur #3566

timkofu opened this issue Jan 12, 2021 · 13 comments
Labels
kind/bug Something isn't working as expected

Comments

@timkofu
Copy link

timkofu commented Jan 12, 2021

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: macOS Big Sur 11.1
  • Poetry version: 1.1.4

Issue

poetry add tensorflow fails in a fresh Python 3.8.7 .venv with the final exception distutils.errors.CompileError: command 'clang' failed with exit status 1.

pip install tensorflow fails with the error Could not find a version that satisfies the requirement tensorflow (from versions: none).

I upgraded pip and setuptools to the latest releases pip install -U pip setuptools and rerun poetry add; It failed with the same exception, and the following error:

ERROR: Command errored out with exit status 1: /private/tmp/test_poetry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"'; __file__='"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-record-be6ffsck/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/test_poetry/.venv/include/site/python3.8/grpcio Check the logs for full command output.

However, pip install tensorflow now works:

python -c "import tensorflow;print(tensorflow.__version__)"
2.4.0
@timkofu timkofu added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Jan 12, 2021
@old-adapdr
Copy link

@timkofu this is a platform specific error that doesn't affect poetry.

Most scientific packages broke with BigSur (Numpy, Pandas, Tensorflow) as there is breaking changes both for the intel and Apple silicon based MacBooks.

To get any of the aformentioned packages running again you'll have to revert to Catalina, run it containerized/vm or wait until fixes have made it downstream for Apple's breaking changes.

@old-adapdr
Copy link

old-adapdr commented Jan 12, 2021

You can confirm this yourself in a minute or two by:

  • Creating a new virtual environment python -m virtualenv testenv && source testenv/bin/activate
  • Installing your affected packages pip install numpy tensorflow pandas etc
  • Try to run a file that just imports the affected packages, if it still breaks its 100% for sure on your system 👍

(added this comment as I see you're running multiple versions of python. Python2 might never have been updated, for that it's important to remember Python2 is a dead and no longer maintained language .)

@timkofu
Copy link
Author

timkofu commented Jan 12, 2021

@alexpdr In my last few sentences, I showed that it does work with an updated pip and setuptools, but not with poetry. Im current working with plain activate .venv/bin/activate and deactivate. Pandas, Tensorflow, Seaborn et al are all working well with the latest pip (which I demonstrated with the Tensorflow import in the Tensorflow case).

Also, I'm only running Python 3.8.7 on this venv, no Python 2. The last code is printing the Tensorflow version, showing it imported successfully with no errors when installed via pip.

@old-adapdr
Copy link

----> grpc/grpc#24677
You might be able to find additional help here, but I'd resubmit the error message on the framework/tool that produces the error (which here is grpc)

ERROR: Command errored out with exit status 1: /private/tmp/test_poetry/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"'; file='"'"'/private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-req-build-mqwpjztf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/22/yt1nh2rd27x1pjfv0zvq0c9w0000gn/T/pip-record-be6ffsck/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/test_poetry/.venv/include/site/python3.8/grpcio Check the logs for full command output.

@timkofu
Copy link
Author

timkofu commented Jan 12, 2021

Had seen that thread before reporting this; it's where I got the idea that using (the latest) plain pip would solve the DS stack (and other) install issues. Specifically this comment. "It's working fine with pip, have you tried that?" will probably be the first comment.

The activate .venv/bin/activate and deactivate (and manually managing the prod and dev deps) workflow is working well for us, but Poetry is such a great tool, and we like reporting issues as soon as we find them, so we can all keep having great DX.

@finswimmer
Copy link
Member

Which versions of pip and setuptools are installed in the venv created by poetry?

@loganjhennessy
Copy link

loganjhennessy commented Jan 12, 2021

I came across the same issue when attempting to install google-cloud-bigquery which also depends on grpcio.

There is a lot going on here, but the offending line in grpcio is https://github.com/grpc/grpc/blob/master/setup.py#L358.

The return value from sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') has always been a str (e.g. 10.14, 10.15, etc.) but in Big Sur, it is possible to get a MACOSX_DEPLOYMENT_TARGET value of just 11. Config values that look like integers are interpreted as integers by cpython, see python/cpython#23622 and python/cpython#23556. So on Big Sur, sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') returns an int. The subsequent call to pkg_resources.parse_version() fails and generates the aforementioned error.

I was able to get this to work by downgrading my Python distribution to 3.7 since, for whatever reason, sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') returns 11.0, which gets passed through as a str.

I don't know exactly where the fix for this should be made. It is frustrating that the value gets cast to an int. One solution would be to ask that the return value from that call in grpcio be cast to str before being passed to pkg_resources.parse_version().

By the way, I'm using Brew for this. Someone mentioned hardcoding the value in the Brew formula as a workaround (see python/cpython#23556 (comment)) although I don't think this is a good long-term solution.

@timkofu
Copy link
Author

timkofu commented Jan 12, 2021

@finswimmer with the venv created with poetry env use ~/.pyenv/versions/3.8.7/bin/python:

>>> import pip, setuptools
>>> pip.__version__
'20.2.2'
>>> setuptools.__version__
'49.6.0'

@nacaclanga
Copy link

Wheel installation was fixed in packaging (pypa/packaging#319) and the newer version is used in pip (pypa/pip#9138), but not in poetry or poetry-core. I suspect that this kind of issue could be fixed by providing a new version of poetry-core, where the vendored packaging dependency is updated to the latest release and then fix the requirements of poetry itself.

@damienrj
Copy link

I think this would also solve the issue with needing pip 20.3.0> to be able to download wheels.
#3591

@bajohn
Copy link

bajohn commented Feb 8, 2021

I had a similar issue as described by @loganjhennessy , regarding google-cloud-bigquery being unable to install grpcio.

A quick fix that worked for me was running

export SYSTEM_VERSION_COMPAT=1

Afterwards, clang errors were no longer thrown and I was able to install via pipenv. Not sure if this fix applies to the original issue but hope it helps!

@timkofu
Copy link
Author

timkofu commented Mar 4, 2021

This is now working perfectly with release 1.1.5.

@timkofu timkofu closed this as completed Mar 4, 2021
@abn abn removed the status/triage This issue needs to be triaged label Mar 3, 2022
Copy link

github-actions bot commented Mar 2, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected
Projects
None yet
Development

No branches or pull requests

8 participants