Hi,

I took some inspiration from your patch set. :)

• The patch files 0005-add-emscripten-host.patch and 0006-fix-Py_Sigset_Converter.patch should no longer be needed. I did not apply your patches directly, but I gave you credit.
• The patch remove-duplicate-symbols-from-cfield.c.patch is causing issues on Windows, https://bugs.python.org/issue45898 .
• Patch 0001-disable-set_inheritable.patch does not explain why the change is necessary. I would like to understand the reason for the patch first before we land it in upstream.
• I created a CONFIG_SITE override, which is IMHO a cleaner approach than your pyconfig.undefs.h.
• Eventually we will be able to replace the custom Setup.local with https://github.com/python/cpython/blob/main/Modules/Setup.stdlib.in and full dependency detection in configure. There are still some modules missing from the file, though.

I've created a repo with my build scripts to build CPython (for now for emscripten but I hope to add work on WASI eventually...): https://github.com/ethanhs/python-wasm (based on @tiran's config.site and my own attempts).

I am happy to say I've successfully gotten CPython master running in the browser with emscripten!

The main trick up my sleeve is emcc -o python.html Programs/python.o libpython3.11d.a -ldl -lm --preload-file ., which loads the local build tree and the JS Emscripten system shims.

Lots of things don't work though, like help() :P

My hope is we eventually centralize all of this WASM build knowledge somewhere so that Pyodide can focus more on the "build the scientific stack" bit and the "build CPython for WASM/WASI" becomes a shared problem.

But obviously getting this all working upstream is the first step. 😅

Sounds great @tiran ! Thanks for writing this summary.

Patch 0001-disable-set_inheritable.patch does not explain why the change is necessary. I would like to understand the reason for the patch first before we land it in upstream.

It was added when the project started in 2018, with little documentation, difficult to determine exactly why now. Currently removing it works fine #2003 so it's probably something that was necessary for previous versions of Emecripten or CPython.

I've created a repo with my build scripts to build CPython

Great work @ethanhs !

My hope is we eventually centralize all of this WASM build knowledge somewhere so that Pyodide can focus more on the "build the scientific stack" bit and the "build CPython for WASM/WASI" becomes a shared problem.

Looking forward to this @brettcannon ! For information, currently, we are not necessarily focusing only on scientific computing. Other areas which constitute most of the Pyodide code base, are:

• Python/JavaScript type translations / FFI. This is maybe less relevant for WASI, but when running in the browser one needs some way to communicate between the Python interpreter and the web page where it's running.
• Building packages with C extensions. This is relevant for scientific computing but not only. Currently, we are using this hack which works surprisingly well, but clearly, a better cross-compilation story would be preferrable (Cross-compile packages #819). This is also related to distutils deprecation and its replacement.
• Trying to compensate for the limitations of the WebAssembly VM in the browser which might mean re-implementing some of the stdlib functionality with Web APIs (e.g. Write http.client in terms of Web APIs #140 which in turn requires to somehow convert async JS calls into sync Python calls Synchronous IO #1503). This is also related to passing the CPython and packages test suites. There is lots of work that could be done there.
• There are still open questions for package distribution. Right now we use the emscripten package format + pure Python wheels from PyPi with micropip. One could argue that WASM is just another build target, that one could do binary wheel Add support for wasm32 platform pypi/warehouse#10416 . This would certainly be a step forward, however, there are also lots of specifics in fetching files for a web-page as opposed to for a local install. For instance, currently, to reduce download size we are,
  • extracting tests and re-exporting them as a separate package. So there are for instance, numpy and numpy-test packages.
  • (optionally) unvendoring parts of stdlib as a separate package. Currently only distutils.
  • there will likely be minimized builds in the near future Use python-minifier to minify the Python stdlib for saving about 2.4 MB #1970

Some of those topics would likely be relevant for anyone trying to run CPython in the browser, and we can certainly try to make these parts more modular and independent so that they can be more easily re-used or adapted.
It would be good to find some way to to collaborate on these subjects as well.

@rth My comment about focusing on the "build the scientific stack" was meant to imply all the work you do for general building; sorry if it came off as me shortchanging all the work you have done! 😅

One thing we will have to discuss is where is the line in terms of what CPython can/will provide out-of-the-box and what will be left to projects like Pyodide to provide themselves? For instance, packaging is not a CPython core team concern; that's left to the community. And so all of your bullet points about Pyodide's work on packaging and building extensions definitely hold as something important this project provides.

As for the JS FFI and trying to make more parts of the stdlib work via web APIs, that's an open question as to where that work would be best to live (and we should probably not try to answer until we decide how serious we all are about centralizing this work and where the centralized location should be).

Thanks for starting this discussion @brettcannon !

One thing we will have to discuss is where is the line in terms of what CPython can/will provide out-of-the-box and what will be left to projects like Pyodide to provide themselves?

Certainly.

As for the JS FFI and trying to make more parts of the stdlib work via web APIs, that's an open question as to where that work would be best to live

Right now the JS/Python type translations is at the core of Pyodide. A good part of our API for Python and Javascript either exposes it or has a very strong coupling with it. Indeed, accessing JS from Python is a pre-requisite for finding workarounds for some of the stdlib modules. Such as using the browser event loop for asyncio or possibly in the future using WebWorkers for multiprocessing. It does make sense that better wasm support is done upstream, but as for the JS/Python type conversion API it's very specific to emscripten and the version used for the build and I imagine it would not be integrated into CPython? Even if it could, it's still actively developed, there are edge cases that need frequent fixes, and having a separate package, with faster iteration and the lighter process would still be preferable IMO. Now if it is a separate package, one can access it as,

# from Python
from pyodide import js

or

# From JS
pyodide.runPython(...)

as part of the Pyodide Python and JS packages. We have talked about it with @hoodmane and we don't really see the point of moving the JS/Python type translation code into a different project. It would pretty much mean renaming the Pyodide package to something else. Particularly in the context where it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure, where they could take CPython and the Pyodide core package and build it with a conda build setup.

We are very open to making things more modular in Pyodide #2014, or anything else that could facilitate the work you are doing though.

I imagine it would not be integrated into CPython?

Probably not. It's more of a question of whether it would be a Pyodide sort of thing or some common project we all contribute to. I basically don't know where everyone wants to draw the line on where Pyodide starts/stops compared to CPython itself and compared to some hypothetical project could sit between Pyodide and CPython.

it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure

Is that being discussed anywhere publicly? I'm curious how that would look due to some conda packages sometimes requiring arbitrary shell script execution in an activated conda environment which isn't exactly a browser feature. 😉

it's very likely there will an effort in 2022 to add WASM builds based on the conda-forge infrastructure

Might there instead be the possibility of having designated wasm wheel build targets?

https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/

And then have that wasm build target supported within something like the following:

https://github.com/pypa/cibuildwheel

A lot of the ecosystem is using cibuildwheel so this would mean a large number of wheels would then be distributed and downloadable from pypi in the wasm target.

[Edit]
I gave it a little more thought and I realised that many libraries have been needing patches and work arounds, so something like cibuildwheel won't be a magic bullet.

Personally as a package maintainer, having to support both pypi distribution and conda-forge distribution can be quite the headache. So as such I dropped conda-forge distribution of my scientific library quite some time ago
[/Edit]

It's more of a question of whether it would be a Pyodide sort of thing or some common project we all contribute to. I basically don't know where everyone wants to draw the line on where Pyodide starts/stops compared to CPython itself and compared to some hypothetical project could sit between Pyodide and CPython.

I agree it's good to have this discussion and if you want we can organize a call to talk about it.

[possible] WASM builds based on the conda-forge infrastructure

Is that being discussed anywhere publicly?

It's been an ongoing subject for a while at conda/conda#7619 but @SylvainCorlay might have more recent information.

Might there instead be the possibility of having designated wasm wheel build targets?

Definitely, these efforts are not mutually exclusive. We are currently exploring that possibility #655, pypi/warehouse#10416 and at present, it seems likely that we might use it, at least to replace the emscripten package format. Wheels do have limitations, particularly for packaging non Python dependencies, but on the conda for WASM packaging side there are also open questions #795

I agree it's good to have this discussion and if you want we can organize a call to talk about it.

Sure! Probably @ethanhs and @tiran would be good to have from the CPython side as well as me.

We are currently exploring that possibility

As the author of packaging.tags I can sponsor the PEP at least if someone with the technical knowledge can (co-)author the PEP, but there are a lot of technical questions to work through. For instance, would the platform need to be versioned due to potential breakage from changes in emscripten? Do we have a solid grounding and definition of how to import a module compiled for WASM where if some other WASM build came about they would be able to use the wheel? Does it makes sense to reuse cp as the interpreter type or would we need a new wasm interpreter type, e.g. cp310 versus wasm310? What does the build tool support look like (I know Pyodide has a solution for setuptools, but is it robust enough for most setuptools projects to rely on? What about e.g. Meson?)? Is there some ABI we need to concern ourselves with? Basically, how do you make sure the wheel tag triple each make sense in the face of WASM wheels?

We can and should use the cp311 tag. After all the extension is compiled for CPython 3.11 API and ABI. wasm32 is kind of a CPU/machine architecture and emscripten or wasi are the operating system. When you compile a C extension for WASM, then you target a CPython API/ABI, the wasm32 architecture and either emscripten or wasi as the underlying operating system and libc. I'm stretching the definition of OS a bit, though. If you think about it, then emscripten kind of provides the abstraction layer for memory allocation, I/O and other stuff.

A binary wheel for wasm32-emscripten could have a tag like spam-4.0-cp311-abi3-emscripten_wasm32.whl. Or perhaps better emscripten3_wasm32 for emsdk 3.0.0. I'm not sure of emscripten builds are backwards compatible.

A binary wheel for wasm32-emscripten could have a tag like spam-4.0-cp311-abi3-emscripten_wasm32.whl. Or perhaps better emscripten3_wasm32 for emsdk 3.0.0. I'm not sure of emscripten builds are backwards compatible.

I agree with this. I am working on moving Pyodide over to using wheels here:
#2027

I can sponsor the PEP at least if someone with the technical knowledge can (co-)author the PEP, but there are a lot of technical questions to work through.

I would be happy to author or coauthor the PEP. Not sure whether I have the technical knowledge yet, but I may be the closest to having it.

For instance, would the platform need to be versioned due to potential breakage from changes in emscripten?

Yes, I think the platform ought to be versioned. But I am thinking about having a modified ABI for dealing with function pointer casting by adding a custom llvm pass. I think this ABI will have to change occasionally in a way that is backwards but not forwards compatible (old wheels work with new interpreter, new wheels don't work with old interpreter).

Do we have a solid grounding and definition of how to import a module compiled for WASM where if some other WASM build came about they would be able to use the wheel?

I think we should keep emscripten in the target. In principle, if the other platform is able to produce function calls into emscripten-ABI wasm functions, and has it's own implementations for all the needed imports, and in particular has it's own system for dynamic linking, all compatible with emscripten, then they would be able to work together.

But if some platform is completely compatible with emscripten, it could just use the emscripten platform tag or accept emscripten-tagged wheels. In any case, I think it will be a while before e.g., WASI would ever be able to load emscripten wheels. And why wouldn't they just build separate WASI wheels?

How are the compiler runtime libraries (like libc++) and libc handled? Are they statically linked or dynamically linked? If the latter, are users expected to have these somewhere or should these be packaged as well?

libc and libc++ are dynamically linked by emscripten.
https://emscripten.org/docs/getting_started/FAQ.html#how-do-i-link-against-system-libraries-like-sdl-boost-etc

We can also package dynamic dependencies, for instance we do this with CLAPACK.

It's been an ongoing subject for a while at conda/conda#7619 but @SylvainCorlay might have more recent information.

Yes, QuantStack will start working on making emscripten-built conda/mamba packages in January.

A binary wheel for wasm32-emscripten could have a tag like spam-4.0-cp311-abi3-emscripten_wasm32.whl. Or perhaps better emscripten3_wasm32 for emsdk 3.0.0. I'm not sure of emscripten builds are backwards compatible.

+1 for this as well. It is my understanding that Emscripten does not have a stable ABI, but I don't know about WASI-libc. If they don't have a stable ABI, may I recommend that we tie Emscripten/WASI version to Python version somewhat like is done on Windows?

I would be happy to author or coauthor the PEP. Not sure whether I have the technical knowledge yet, but I may be the closest to having it.

I would suggest reading over the manylinux specs, which I probably should do too :) I think those are probably the closest to what we want.

I think we should keep emscripten in the target

+1 on this, emscripten and WASI are not compatible. WASI in particular isn't even meant to be POSIX.

I think it will be a while before e.g., WASI would ever be able to load emscripten wheels

I'm not sure if this will ever happen as I don't expect WASI to support dynamic linking (though I guess you could build on top of WASI to add this).

I don't expect WASI to support dynamic linking

They are absolutely planning support for dynamic linking:
https://github.com/WebAssembly/module-linking/blob/main/design/proposals/module-linking/Explainer.md

They are absolutely planning support for dynamic linking:

Aha, I forgot about that proposal. But that method looks to be incompatible with Emscripten's method of dynamic linking (well Emscripten kinda has 2, but whatever). Maybe some day Emscripten will transition to that, but I am also not sure how module linking would work with threads etc.

I am thinking about having a modified ABI for dealing with function pointer casting by adding a custom llvm pass. I think this ABI will have to change occasionally in a way that is backwards but not forwards compatible (old wheels work with new interpreter, new wheels don't work with old interpreter).

That might complicate things as that would be introducing a new ABI for CPython in wheels (if I'm understanding correctly). That might necessitate a change in interpreter types, but honestly no one has proposed a new ABI since wheels were created. And is the custom pass just for building the interpreter, or would everyone have to use that custom pass to be compatible (and thus how to make sure everyone building a wheel had that custom pass)?

That might necessitate a change in interpreter types

I don't think it should -- the compiler will take care of it using existing source code.

is the custom pass just for building the interpreter, or would everyone have to use that custom pass to be compatible (and thus how to make sure everyone building a wheel had that custom pass)?

It is not yet clear because I haven't gotten the design far enough. My hope would be that we could upstream this into llvm or failing that into emscripten so that everyone would use it. It will be a significant improvement to the wasm ABI I think -- at least in the Python ecosystem we have major trouble with fpcasts.

The main issue is that we would encode extra signature information into the upper bits of function pointers, so those bits need to be masked out before calling the function pointer. If a wheel is receiving a function pointer from the CPython or a wheel built with this fpcast API and it wants to call that function pointer, the pointer needs to have its signature bits cleared first. This could be fixed up with a binaryen pass on the completed wasm binary, so if a wheel was built without the ABI and needs to be used with it, it's not necessary to rebuild the wheel, sort of like the way auditwheel works.

On the other hand, wheels with the fpcast ABI just won't work with interpreters built without it. But interpreters built without the fpcast ABI are going to have worse capabilities in general than ones with it.

@SylvainCorlay do you have more details somewhere about what packages you are planning to build and how you are going about this?

I agree it's good to have this discussion and if you want we can organize a call to talk about it.

Sure! Probably @ethanhs and @tiran would be good to have from the CPython side as well as me.

@brettcannon OK, maybe let's try to setup a call in early January to talk about how we could collaborate?
Here is a doodle link to find the best time. Let me know if none of those slots work.
Also draft of the agenda https://hackmd.io/2JjM9FiZQBufKpIiQgGVHQ feel free to edit it.

@rth Done! Now we wait to see what @ethanhs and @tiran put in...

Why do you need _posixsubprocess? To make import subprocess work?

base_events.py has a top level suprocess import and our interpreter imports asyncio on startup.

configure disables pwd module and several other modules on Emscripten.

I'm not seeing this in 3.11.0a6. I have to manually remove pwd from Setup.bootstrap.

Why do you need _posixsubprocess? To make import subprocess work?

base_events.py has a top level suprocess import and our interpreter imports asyncio on startup.

python/cpython#32224 will fix import subprocess on Emscripten and WASI without the _posixsubprocess module.

I'm not seeing this in 3.11.0a6. I have to manually remove pwd from Setup.bootstrap.

I might have landed the change after 3.11.0a6. It's in HEAD of main. In the mean time you can disable the pwd module in Setup.local:

*disabled**
pwd

I might have landed the change after 3.11.0a6. It's in HEAD of main. In the mean time you can disable the pwd module in Setup.local:

When I tried that in the past, it looked like disabling a module in a Setup.local doesn't affect Module/Setup (#1883 (comment)), so we ended up with patching. Was there some update on that? or maybe I was doing it in the wrong way.

I'll add a branch building against the alpha so we can have a better reference point for this conversation.

When I tried that in the past, it looked like disabling a module in a Setup.local doesn't affect Module/Setup (#1883 (comment)), so we ended up with patching. Was there some update on that? or maybe I was doing it in the wrong way.

It's a new feature, https://bugs.python.org/issue46023

What if I want to disable a bunch of things? Do I put disabled over each? e.g.

*disabled*
module1
*disabled*
module2
...

Or it looks like I only have to put disabled once?

Or it looks like I only have to put disabled once?

just once is sufficient

For reference, I pushed my 3.11.0a6 branch here:
https://github.com/pyodide/pyodide/tree/python-3.11.0a6

You should be able to replace pyconfig.undefs.h with a CONFIG_SITE override. The overrides vars are the same as the vars in config.cache.

CONFIG_SITE=Tools/wasm/config.site-wasm32-emscripten \
    emconfigure ./configure -C ...
``

Emscripten builds have test modules disabled. You can enable the test modules with --enable-test-modules configure flag. Or you can add them to your Setup.local.

*shared*
_testbuffer _testbuffer.c
_testinternalcapi _testinternalcapi.c
_testcapi _testcapimodule.c
_testimportmultiple _testimportmultiple.c
_testmultiphase _testmultiphase.c
_ctypes_test _ctypes/_ctypes_test.c

The build system is opinionated. So far we have two build flavors --with-emscripten-target=[browser|node] and no support for side modules. I can extend the build system if you tell me what you need.

We want to target both browser and node at the same time -- in practice this doesn't seem too hard, though there is some tricky runtime feature detection required. We want to build the test modules and ssl as side modules. It would be good to work with Emscripten on improved shared library support. It's a bit of a drag that it just ignores -shared.

CONFIG_SITE=Tools/wasm/config.site-wasm32-emscripten

How do we override ac_cv_func_dlopen=no? We really don't want that. Also dup3 works now though this isn't super important. We should go back and check at some point whether there are still issues with Emscripten filesystem openat, etc commands.

You could copy the file into your project, make adjustments, and we merge your fixes back into CPython later.

Most at functions now work except for symlinkat, python/cpython#32238

I've been meaning to review which of the features we actually need to be disabling. But there are just so many things to do... Thanks for checking.

In the core test suite I'm hitting failures from faulthandler:

  File "/lib/python3.11/test/libregrtest/main.py", line 662, in main
    faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: unable to start watchdog thread

Can fix with:

import faulthandler
faulthandler.dump_traceback_later = lambda *args, **kwargs: None

In the core test suite I'm hitting failures from faulthandler:

  File "/lib/python3.11/test/libregrtest/main.py", line 662, in main
    faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: unable to start watchdog thread

Does python/cpython#32243 solve your problem?

I suspect so.

python/cpython#32253 is an attempt to add dynamic linking for WASM:

>>> import xxlimited
>>> xxlimited   
<module 'xxlimited' from '/python-wasm/cpython/builddir/emscripten-node/build/lib.emscripten-wasm32-3.11/xxlimited.cpython-311-wasm32-emscripten.so'>

I'm getting compile errors on tot from:
https://github.com/python/cpython/blob/6066739ff7794e54c98c08b953a699cbc961cd28/Parser/tokenizer.c#L2091

Parser/tokenizer.c:2093:5: error: use of undeclared identifier 'cookie_io_functions_t'
    cookie_io_functions_t io_cb = {borrow_read, NULL, NULL, NULL};
    ^
Parser/tokenizer.c:2095:12: error: implicit declaration of function 'fopencookie' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    return fopencookie(b.cookie, "r", io_cb);
           ^
Parser/tokenizer.c:2095:39: error: use of undeclared identifier 'io_cb'
    return fopencookie(b.cookie, "r", io_cb);

Note also that Emscripten's dup will be getting faster soon: emscripten-core/emscripten#14808

Probably emscripten added fopencookie in a version more recent than 2.0.27. We are stuck on 2.0.27 for now because of a bug with dynamic linking in firefox. emscripten-core/emscripten#16538

Fixed in python/cpython#32266

On tip of tree the majority of tests that we had to manually xfail or filter out are now automatically being filtered. Thanks guys!

The following are a list of tests that we want to skip / xfail / filter out that still aren't being filtered on the cpython main branch:

can't start new thread

Tons of failures RuntimeError: can't start new thread:

• test_bz2.BZ2FileTest.testThreading
• test_context.ContextTest.test_context_threads_1
• test_email.test_email.TestMiscellaneous.test_make_msgid_collisions
• test_enum.OldTestIntFlag.test_unique_composite
• test_functools.TestCachedProperty.test_threaded
• test_functools.TestLRUC.test_lru_cache_threaded
• test_gc.GCTests.test_trashcan_threads
• test_hashlib.HashLibTestCase.test_threaded_hashing
• test_importlib.test_locks.Source_ModuleLockAsRLockTests.test_thread_leak
• test_importlib.test_threaded_import
• ... etc

Sockets

Most sockets tests automatically are skipped now but not all. These tests fail on socket.listen():

• test_support.TestSupport.test_HOST
• test_support.TestSupport.test_bind_port
• test_support.TestSupport.test_find_unused_port

Missing bytecode

test_compileall and test_pydoc seem to fail when config.write_bytecode = false;

Missing modules:

A lot of problems with missing dbm. test_sysconfig and test__osx_support fail with missing _osx_support

test_copyreg missing dbm
test_ensurepip ModuleNotFoundError: No module named 'ensurepip'
test_venv: ModuleNotFoundError: No module named 'ensurepip'

Misc

unittest.test.test_break.TestBreakSignalIgnored.testTwoResults fails with

    os.kill(os.getpid(), signal.SIGINT)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 63] Operation not permitted

• bpo-40280: Detect missing threading on WASM platforms (GH-32352) python/cpython#32352 fixes the threading and socket related issues
• for dbm, ensurepip and _osx_support I suggest that you remove the test files test_dbm, test__osx_support, test_ensurepip, and test_venv.

Thanks!

I suggest that you remove the test files test_dbm, test__osx_support, test_ensurepip, and test_venv.

Yeah that's a reasonable solution. I'm not sure why test_sysconfig is trying to import _osx_support, most likely that is some misconfiguration on my side though.

@rth @tiran I think this can be closed as completed? We can open new more specific issues in the future. Thanks so much for your work on this @tiran! This is going to dramatically reduce the maintenance burden for updating Python.

Agreed. Closing. Thanks, everyone for the discussion and your work on this!