build a0 (2021-04-19 'PyPy')

WinPython-3.7.10.0a0 with PyPy-3.7 v7.4.3

What shall work:

• IDLE,
• IDLEX,
• WinPython Command Prompt,
• WinPython Interpreter.exe,
• WinPython Powershell Prompt
• the scripts:
  • make_winpython_movable (by necessity of WinPython)
  • make_winpython_fix.bat
• and so, it seems ... movable like any WinPython.

Areas of particular interest for testers::

• well well, certainly lot of bugs and speed to discover,
• help PyPy and WinPython project to iron them out
• the fact that the demo page of SQLite_bro is not working exactly like for cpython is a first issue to dig

Next build a1:

• SQLite3.35
• patch SQLite_bro (L763) to avoid multi-empty tab / keep same effect (not sure it's a 'bug' of PyPy per say)

Included Package:

• WinPython
• sqlite_bro
• pip-20.1.0dev

Location: https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.7.10.0/alphas/

speed measures

• using sqlite_bro:

pydef py_sin(s):
    "sinus function : example loading module, handling input/output as strings"
    import math as py_math
    return ("%s" %  py_math.sin(s*1));
WITH RECURSIVE
  cnt(x) AS (
     SELECT 1
     UNION ALL
     SELECT x+1 FROM cnt
      LIMIT 3000000
  )

select sum(py_sin(x))  from cnt  

to get the same number of tabs we need to replace in SQLite_bro:
https://github.com/stonebig/sqlite_bro/blob/dcbbd810d0cea2be40c5c56d02e312be350a81dd/sqlite_bro/sqlite_bro.py#L739

if cur.description is not None:
per
if cur.description is not None and len(cur.description)>0: #pypy needs the second test

• mandelbrot (when necessary wheels in place)

VSCode does the sudoku with pypy3.7-v7.3.4-win64

• echo %time% & pypy3 C:\WinP\bd37\budotpypy\wpy64-37100\notebooks\docs\Solvers_Raymond_Hettinger\python\sudoku_norvig_for_comparison_of_complexity.py
• echo %time% & python C:\WinP\bd37\budotpypy\wpy64-37100\notebooks\docs\Solvers_Raymond_Hettinger\python\sudoku_norvig_for_comparison_of_complexity.py

speed_basic.py test

• echo %time% & pypy3 C:\WinP\bd37\budotpypy\wpy64-37100\notebooks\speed_basic.py
• echo %time% & python C:\WinP\bd37\budotpypy\wpy64-37100\notebooks\speed_basic.py

import time

start_time = time.time()

total = 0
for i in range(1, 10000):
    for j in range(1, 10000):
        total += i + j

print(f"The result is {total}")

end_time = time.time()
print(f"It took {end_time-start_time:.2f} seconds to compute")

PyPy build a1 (2021-04-20 'SQLite-0.35.5')

change from build a0:

• SQLite-0.35.5 (math functions included)
• sqlite-bro-0.9.2 fixes to become compatible with PyPy
• a few clean-ups

Areas of particular interest for testers:

• sqlite_bro-0.9.2 is much more compatible with PyPy (fullly ?)
• well:
  • the pleasure of discovering the mythical PyPy 'JIT' Python
  • contributing bug hunting and fixing in PyPy for Windows

Next build a2:

• modify build system, so it's no more manual work
• wait the 64bit binary wheel problem is solved (so PyPy 7.3.5, expected in two weeks)
• hope is PyPy 7.3.5 will include:
  • the windows 64 bit wheel naming problem fix,
  • SQLite-0.35.5 with math and json functions activated, (or may patch)
  • with a strech of luck, the 'import' performance fix. (not expected)

Included Package:

• WinPython
• sqlite_bro-0.9.2
• pip-20.1.0dev

Location: https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.7.10.0/alphas/

PyPy build a2 (2021-04-24 'automated build')

Change from build a1:

• Infrastructure work:
  • standard automation ... and so back to the old SQLite shipped per PyPy-7.3.4
  • sha3-256 added
  • unchecked; "associate" function, or future Qt integration

Location: https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.7.10.0/alphas/

Areas of particular interest for testers:

• it works as any other WinPython, except you must type "pypy3" instead of python
• for experts knowing what they do:
  • check the associate function, (Winpython in the windows menu)
  • look ahead on Qt integration.

WinPython 3.7.10.0dotPyPy

The following packages are included in WinPython-64bit v3.7.10.0dotPyPy a2.

Tools

Python packages

looking at Cython CI, it seems Pypy-3.7 is (8m 10 s/ 5m 18s) 35% slower than Python-3.7 on this exercise ... room for improvement

cython source wheel is compatible with PyPy, on more package, but fails to compile statsmodels extensions. so flight domain is currently constrained per:

• jupyter_core needing pywin32
• failing to build wheels for statsmodels, pywinpty
• pyside6 wheel (on the road from Qt, but not there).

on PyPy3 side:

• "import package speed-up" is only coming for 7.3.6, (to check if performance with sqlite test improves widely)
• hpy talk or article at pycon 2021 is not yet available.
• python-3.8 compat is slow to come

may 23rd morning status;

• works: numpy, matplotlib, pandas, seaborn, scipy, lmfit, cython, sympy, holoviews, bokeh
• packages : 408 included (working well or not) out of 520
• last night improvement:
  • wheels for statsmodels, pywinpty provided per cgohlke, but statsmodels has a practice incompatible with PyPy3 statsmodels/statsmodels#7460)
  • switch to a fresher numpy+mkl solved the "import pandas" (was old) issue

Flight domain is currently constrained per:

• "Jupyter":
  • PROBLEM NARROWED: jupyter_core needing pywin32 => cgohlke did "empty" pywin32 wheel, suggests a solution is easy to replace the 3 missing functions.
  • PyPy3 is missing a python-3.5 (windows) change: https://foss.heptapod.net/pypy/pypy/-/issues/3469
• "science":
  • statsmodels needs to change that particular practice on docstring (statsmodels/statsmodels#7460)
• "Qt":
  • pyside6 wheel on the alpha road from Qt Company, but not there yet.
• "Deep Learning":
  • out of scope for now, need to have the rest flying.

• "science":
  • SOLVED: statsmodels bug resolved per this patch: statsmodels/statsmodels#7462

• "Jupyter":
  • SOLVED by Christoph Gohlke: a ccfi bug in pyzmq + jupyter_core to patch jupyter/jupyter_core#226 (comment)

Flight domain is currently constrained per:

• in "Jupyter":
  • shapely: in need for a few geograhic packages cartopy and ipyleaflet
  • orjson: missing to ipycanvas
• "science":
  • numba: missing to fastparquet
  • pyside6 wheel on the alpha road from Qt Company, but not there yet.
  • reportlab
• "Deep Learning":
  • out of scope for now, need to have the rest flying.
• "Qt": no PySide6-6.1 yet

milestone 1 "credibility" is now reached "WinPython 2021-03: PyPy-7.3.5 for Python3.7+ VSCode + Jupyterlab + Cython, on 64 bit only".

Next milestone 2 is about more natural "performance" vs cPython (as PyPy can perform, but you need often to tweak your code)

Hey @stonebig, what's the best place to report errors on these builds? Is it to the individual libraries? I can get matplotlib to crash the interpreter with the following script. It works with cPython but not Pypy. Changing the backend to Agg instead of the default TkAgg also works. I usually use Qt5Agg and plan to use that once Qt is integrated in.

"""Pypy crashes on plot closure."""

import unittest

#import matplotlib
#matplotlib.use('Agg')
import matplotlib.pyplot as plt
import numpy as np

class Test_plots(unittest.TestCase):
    def setUp(self):
        self.x = np.arange(5)
        self.y = 10 * self.x
        self.figs = None

    def test_plotting1(self):
        fig = plt.figure()
        plt.plot(self.x, self.y)
        fig2 = plt.figure()
        plt.plot(self.y, self.x)
        self.figs = [fig, fig2]

    def test_plotting2(self):
        fig = plt.figure()
        plt.plot(self.x, self.y)
        fig2 = plt.figure()
        plt.plot(self.y, self.x)
        self.figs = [fig, fig2]

    def tearDown(self):
        if self.figs is not None:
            for fig in self.figs:
                plt.close(fig)

if __name__ == '__main__':
    unittest.main(exit=False)

You may report ithe problem on pypy with the code and crash report, asking if it's a pypy bug, a way of writing not tolerated per pypy, or an incompatibility from matplotlib or tk

Do it also on stackoverflow

hum, launched this way, it doesn't blow up with Winpython64-3.7.10.0PyPycodb4.exe

this way:

Updated Flight domain seems now currently constrained per:

• in "Jupyter":
  • cartopy: missing to geoviews
  • orjson: missing to ipycanvas
  • a pyproj binary compatibility bug ??
• "science":
  • numba: missing to fastparquet
  • pyside6 wheel on the alpha road from Qt Company, but not there yet.
• "Deep Learning":
  • out of scope for now, need to have the rest flying, ep numba.
• Cpython eco-system compatibility:
  • Python-3.8 compatibility to get type-ast, mypy,
  • hpy Cython compatibility to get same performance
• VSCode-python better compatiblity, as not super friendly with pypy3 per default

Updated Flight domain (13% not possible packages) seems now currently constrained per:

• in "Jupyter":
  • a pyproj binary compatibility bug ??
• "science":
  • numba: forecast is it will remain missing till the python-3.11 (june 2022 ?)
  • pyside6 wheel on the alpha road from Qt Company, guess it will need PyPy-3.8 (sept 2021 ?)
  • solvers may need a pure python version: ecos, qldl, qpsolvers, pycosat, cvxopt , osqp, nlopt
• "Deep Learning":
  • out of scope for now, need PyPy to have the rest flying, ep numba. (dec 2022 ?)
• Cpython eco-system compatibility:
  • Python-3.8 compatibility to get type-ast, mypy, (alpha around september)
• VSCode-python better compatiblity, as not super friendly with pypy3 per default

Performance (-10x to 30x) is constrained per:

• PyPy3 team slowly removing performance problems on standard python code. Next improvement = PyPy-7.3.6 (sept 2022)
• asking cpython library to not force systematically Cython or C-packages (like ipycanvas not forcing orjson anymore).
• hpy Cython compatibility to get same performance

Update of PyPy progress, testing on PyPy3-3.8 nightly of July 29th:

• MyPy is working, if you comment the gc.set_threshold() line (purely non-functional, used to gain 4% on cPython)
• Black is installable, because MyPy is now installable.
• the Sqlite test is performed in 30 seconds in PyPy3-3.8 vs 85 seconds in PyPy-7.3.6 and 13 seconds in cPython (suspecting the "import" caching improvement backported from cPython3 code)

... not bad

PyPy Nightly for Python3.8 build a0 (2021-08-12, from PyPy3-3.8 nightly of July 29th:)

Infrastructure work:

• toolchain creation
• Location: https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.8.10.1/alphas/

next step:

• correct release version (.1 vs .0)
• integrate an msvc_runtime
• push on the interesting new bits:
  • mypy
  • ...

Flight domain, as of August 28th, is currently constrained per:

• "Jupyter" ecosystem:
  • official support is coming (or CI tested) with Jupyter-core-4.8
• "science":
  • numba missing
• "Deep Learning":
  • people working on PyTorch support (even if only for Conda/Linux)
• "Qt":
  • pyside6 wheel on the alpha road from Qt Company, but not there yet.
  • Spyder support of Pyside6 arriving in a few weeks : The idea is to merge first the support for PySide6 (hopefully in a couple of weeks) and then add the one for PyQt6.
  • Matplotlib-3.5.0 announced for September 14th
• Python-3.8
  • mypy/black: basic compatibility in PyPy python 3.8, work continues on the 'ast' part of it:
    08:51 we are down to 213 failures, 35 errors on the lib-python tests, 24 of them are test_dis and test_ast
• internal weaknesses of PyPy:
  • some bad slowness have been fixed: slowness reduced per 50% on SQlite test,
  • some historic weakness in JIT being worked on
  • not on git nor github: some work on it, may not be a lost cause.

... a little late to 'dreamed' roadmap speed, but momentum is growing

Speed-Center: (2021-12-04 version of PyPy 3.8)

the Nbody experiment became good in PyPy with type freezing
at CGO ’20, February 22–26, 2020, San Diego, CA, USA

Sqlite_Bro test: performance of 'import' and SQlite interactions

pydef py_sin(s):
    "sinus function : example loading module, handling input/output as strings"
    import math as py_math
    return ("%s" %  py_math.sin(s*1));
WITH RECURSIVE
  cnt(x) AS (
     SELECT 1
     UNION ALL
     SELECT x+1 FROM cnt
      LIMIT 3000000
  )

select sum(py_sin(x))  from cnt 

PyPy Nightly for Python3.8 build a0b (2021-08-28, from PyPy3-3.8 nightly of August 28th:)

Changes from Winpython64-3.8.10.0dotPyPyNighta0.exe :

• Infrastructure:
  • adds msvc_runtime
  • PyPy nightly version:

     Python 3.8.10 (3e3aadd1e1d0, Aug 28 2021, 00:05:42)
[PyPy 7.3.6-alpha0 with MSC v.1929 64 bit (AMD64)] on win32

• PyPy nightly has corrected nearly all its Tcl/tk bugs,
• latest version has:
  • a further 20% speed-up in SQlite test, now only 2x slower than cpython vs 6x slower on 3.7
  • still a 20% regression sudoku_norvig_for_comparison_of_complexity.py (hard1)

next step:

• gently push on the apparent speed regression test of PyPy 3.8 nightly
• check for Qt compatibility for PyPy

PyPy v7.3.6 rc is very soon (unless waiting for Hpy-0.0.3)
https://foss.heptapod.net/pypy/pypy/-/blob/eed30f5374249b5652278e3b3b83de694b693712/pypy/doc/release-v7.3.6.rst

Pythonnet is waiting Hpy to mature to do the compatibility effort pythonnet/pythonnet#330 (comment)

2021-09-18 news:

• down to 60 missing packages (11%):
  • 15 related to Qt (but ctismer of Qt/PySide, and stackless pope, working hard on it)
  • 5 needing PyPy-3.8 (mypy, black, ...)

testing cpython-3.11.0a2: not yet a truly visible speed-up at first try.

testing Pyjion: not yet a truly visible speed-up at first try on the sudoku example.

on "elegant classic" code, for now:

• PyPy gets the same 2x speed-up as cython,
• the other fail to improve anything

Educated guess is:

• working hard on JIT only is not enough to solve the Python "slowness" on executing existing code,
• a minuscule touch on the code should bring a 5x larger improvement,
• designing both JIT improvement and a PEP-JIT-performance-enabler-touch should result on:
  • 2x less heroic effort on JIT,
  • 10x better performance result,
  • no split in the community on what that "minuscule" changes shall be, so we have a "Tweaked Classic Elegant" way to write Fast Python Code, and preferably only one.

Something like the data API standard but for JIT ?

december 11th news:

• cpython-3.11.0a3 improves and goes up to cpython-3.9.0 speed on old PC, claim is 19% speed-up (halfway to 50% target), if you knon how to read https://speed.python.org/
• the same shall be true for Pyjion-1.2.1

January 2nd:

• new in PyPy-3.8: opencv, mypy, papermill, ... maybe numpy-1.22.0
• still missing from PyPy-3.7: osqp, gdal, pandas-1.4

January 22nd:

• new in PyPy-3.8: pandas-1.3.5 with fix, gdal, osqp

February 27th, 2022:

• PyPy-3.8-7.3.8 with Python-3.8 official compatibility
• Python-3.9-7.3.8 beta, seems on track for a Q2 official release (moving from 3.7 to 3.9 in 1 year would be awesome)