I think you should deliver both extensions for Python2 as well as for Python3.

Best way would be to compile against Homebrew's Python3. This way you can just deliver the Postgresapp with an out of the box working Python2 and if someone wants to use Python3, he just have to do "brew install python3" and everything is up and running.

To be honest, I'm a little lazy myself. I could build plpythin3u separately. But if you do, life's much easier. ;)

We've been working on this topic a bit.

We can't simply link with Homebrew's Python3, because we have no idea what Python version they use, and they might change the default version of Python 3. There is no binary compatibility with the different minor versions (eg. if you compile PostgreSQL with Python 3.8 it won't work with Python 3.7 or Python 3.9).

However, we've discovered that the Python 3 offers an optional stable ABI. If PostgreSQL would use that ABI, then it wouldn't matter which version of Python 3 the user has installed. I have been able to patch PostgreSQL to use the stable ABI, and I will submit my patch to the PostgreSQL project.

But there's only one problem left: I don't want to require Homebrew. Postgres.app is designed for a wide audience, and I don't know how many of them use homebrew. I want our users to be able to use Python 3 without having to learn how to use a package manager.

I believe that most people who use Python3 on macOS installed it using one of the following methods:

1. Xcode
2. Developer Tools
3. Homebrew
4. Official Installer from python.org
5. MacPorts
6. Maybe install from source?
7. (any others??)

If somehow possible, I want Postgres.app to be compatible with all of them. This means we would probably need to set some environment variable to tell plpython.so where to look for Python, and add some way for the user to select which Python Postgres.app uses.

In case anybody feels like testing, here's a build of Postgres.app with PostgreSQL 13 and Python 3. It requires the official Python 3.8 from python.org to be installed.

https://eggerapps-downloads.s3.eu-west-1.amazonaws.com/Postgres-dev-13.dmg

Are you envisioning including a prompt the user would receive when they first open the app that asks which Python install to use?

Are you envisioning including a prompt the user would receive when they first open the app that asks which Python install to use?

That could be confusing for Postgres.app users who don't use Python.

Our idea was to try to autodetect if the user has Python 3 installed, and set environment variables accordingly. (Eg. check for official Python, if not installed check Homebrew, if that isn't installed check for Xcode, ...)

There could be a preference setting that allows you to override the autodetected setting, or set it manually when Postgres.app doesn't find it. Thinking about this, I wonder if a global setting is sufficient, or if we need one setting per server (so you could have servers with different Pythons).

Anyway, the idea would be that we set the variables DYLD_LIBRARY_PATH to tell postgres which library to load, and PYTHONHOME to tell Python where to look for modules etc. Unfortunately we couldn't get DYLD_LIBRARY_PATH to work yet.

The following doesn't work:

DYLD_LIBRARY_PATH=/Path/To/Python /Applications/Postgres.app/Contents/Versions/13/bin/postgres -D "/Users/USER/Library/Application Support/Postgres/var-13"

For some reason the DYLD_LIBRARY_PATH variable has no effect. postgres has the correct entitlements, but maybe the env var gets lost when forking or something. We need to investigate this.

Another potential issue is that sometimes the framework is called "Python" (eg. the python.org installer) and sometimes it's "Python3" (eg. Xcode).

For now we've been able to get a proof of concept working with symbolic links (we were able to run a test function with Homebrew, Xcode, and official Python). So it looks like it should work in principle, but we still need to figure out how to get it all working automatically, and ideally without requiring the user to type sudo.

For some reason the DYLD_LIBRARY_PATH variable has no effect. postgres has the correct entitlements, but maybe the env var gets lost when forking or something. We need to investigate this.

Isn't this the runtime protection of SIP that prevents inheriting dynamic linker related environment variables when forking? The same reason why make check fails to start the tmp-installed binaries when not doing a make install first?

So, the approach with symlinks seems to be more promising to me. Out of curiosity - what is requiring root rights on that route?

@tbussmann Yes, it sounds like that is exactly what's going on. I think that the SIP runtime protection blocks the DYLD_* variables from being passed on to postgres worker processes. I thought that giving postgres the com.apple.security.cs.allow-dyld-environment-variables entitlement would be enough, but when the postmaster doesn't pass the variable on to workers there's no point.

So here are all the approaches that I can think of that we could use to solve this problem

1. Use DYLD_LIBRARY_PATH or similar: Doesn't work because of SIP
2. Use install_name_tool on the users computer to modify plpython3.so to point to the correct library: Doesn't work because it would invalidate the code signature
3. Use install_name_tool before code signing to point to a specific path (eg. /Library/Frameworks/Python.framework/Versions/latest/Python), and then on the users computer put a symlink to the real Python3 at that path: works, but requires admin privilege, unless we find some shared path that we can write to that doesn't require admin privileges.
4. Change the way that plpython3.so is linked, so that we can determine the location of Python dynamically at runtime instead of compile time, and manually use dlopen to link with Python. It should be possible to do that, but I don't know how complicated that would be. It would be seamless and it wouldn't require admin privilege, and we could even print a nice error message if Python 3 was missing.
5. Create a PostgreSQL extension that's loaded with preload_libraries that changes the environment of the postgres process itself. We could set DYLD_* variables from inside the process, before plpython3.so is loaded. Advantage: No change to plpython needed, would also be pretty seamless.

Can't we use approach 3 with a location within our application bundle like /Applications/Postgres.app/Contents/Versions/13/lib/myPython3 that later can be symlinked to the actual lib with the user's permissions?

Can't we use approach 3 with a location within our application bundle like /Applications/Postgres.app/Contents/Versions/13/lib/myPython3 that later can be symlinked to the actual lib with the user's permissions?

That would probably break the code signature of the Postgres.app bundle. It used to be no problem if it happened after first launch, but macOS has gotten stricter and started checking the code signature more frequently. I recall Apple saying in one of last years WWDC videos that going forward bundle modifications will only be allowed if they keep the code signature intact.

I think the easiest way for us to support Python3 would be to do the following:

1. Link with Python 3.8 from python.org. If user has that installed it works out of the box.

2. Tell people if they want to use a different Python to put a symlink in /Library/Frameworks/Python.framework/Versions/3.8/Python

3. Allow people to set environment variables so they can set $PYTHONHOME (needed eg. when you use python3 from Xcode and renamed Xcode to something else, eg. Xcode-11.app)

One other question: Is this to enable people to use Python 3 with PL/Python, or is there another use case? (I'm author of a PostgreSQL book and recommend installing PostgresApp, so I just want to anticipate changes I may need to make in the book.)

@anthonydb Yes! Right now there's only plpython2u, and I hope that once PostgreSQL 13 is released this will allow users to use plpython3u as well.

With todays release of PostgreSQL 13, we now finally include Python 3 support. However, since macOS does not include Python 3, you now need to manually install Python 3.8 from Python.org. I've documented the necessary steps here: https://postgresapp.com/documentation/plpython.html

If you try to activate plpython3u in Postgres.app without first installing Python 3.8, you'll get the following error message:

ERROR:  could not load library "/Applications/Postgres.app/Contents/Versions/13/lib/postgresql/plpython3.so": dlopen(/Applications/Postgres.app/Contents/Versions/13/lib/postgresql/plpython3.so, 10): Library not loaded: /Library/Frameworks/Python.framework/Versions/3.8/Python
  Referenced from: /Applications/Postgres.app/Contents/Versions/13/lib/postgresql/plpython3.so
  Reason: image not found

That's not a very good user experience, but I don't really know how to improve this. I hope people read the docs or at least find this issue when they google the error message. If would welcome suggestions how to improve this situation. Since a lot of people have trouble using extensions, maybe we should add some more docs about extensions in a prominent location of the home page?

If you don't want to use the official Python.org installer, it may be possible to create a symlink at /Library/Frameworks/Python.framework/Versions/3.8/Python to another python, but it needs to be the same major Python version, and since it's a bit tricky to get right, it's not something that I want to officially support. (I have been working on porting PL/Python to the stable C API which would relax the version requirement, but that's still work in progress and would need to be merged by PostgreSQL folks first)

PostgreSQL 13 in Postgres.app no longer includes Python 2.7. Building PostgreSQL with 2.7 and 3 would make the build process a lot more complicated, and since Python 2.7 reached end-of-life almost a year ago I think ending support for it seems reasonable to me. If you absolutely need 2.7 support, you can continue using Postgres.app with PostgreSQL 12.