https://buildbot.python.org/all/#/builders/302/builds/338

FAIL: test_nextafter (test.test_math.MathTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aixtools/buildarea/3.9.aixtools-aix-power6/build/Lib/test/test_math.py", line 1968, in test_nextafter
    self.assertIsNaN(math.nextafter(NAN, 1.0))
  File "/home/aixtools/buildarea/3.9.aixtools-aix-power6/build/Lib/test/test_math.py", line 2015, in assertIsNaN
    self.fail("Expected a NaN, got {!r}.".format(value))
AssertionError: Expected a NaN, got 1.0.

The test:

        # NaN
        self.assertIsNaN(math.nextafter(NAN, 1.0))   # <=== HERE
        self.assertIsNaN(math.nextafter(1.0, NAN))
        self.assertIsNaN(math.nextafter(NAN, NAN))

The Linux manual page says: "If x or y is a NaN, a NaN is returned."
https://man7.org/linux/man-pages/man3/nextafter.3.html

But it seems like the AIX libc doesn't implement this rule. Should we implement this rule in Python on AIX?

The strange thing is that it worked previously. test.python of build 338:

platform.platform: AIX-2-00F9C1964C00-powerpc-32bit
sysconfig[HOST_GNU_TYPE]: powerpc-ibm-aix7.2.4.0
platform.architecture: 32bit

The latest green build is built 347. test.pythoninfo of build 347:

platform.architecture: 32bit
platform.platform: AIX-2-00F9C1964C00-powerpc-32bit
sysconfig[HOST_GNU_TYPE]: powerpc-ibm-aix7.2.0.0

Was the machine updated two days ago (2020-11-09), between build 338 and build 347?

If AIX were one of our officially supported platforms, then yes, I'd say that we should add a workaround to handle special cases ourselves, similarly to what we already do for a number of math module functions (like math.pow, for example).

But given that it's only a "best effort" platform, I'm not convinced that it's worth the effort or the extra complication in the codebase.

-0 from me, I guess.

Is there any reasonable channel for reporting the issue upstream?

My worry is that I'm getting emails about AIX buildbot failures. I see different options:

* Skip the test on AIX
* Fix nextafter() on AIX
* Turn off AIX buildbot email notifications
* Remove thE AIX buildbot

> My worry is that I'm getting emails about AIX buildbot failures.

That sounds more like a process problem than a CPython codebase one. The ideal would be that the machinery sending those notifications can be configured to ignore known failures when deciding whether to send email. Is that remotely feasible? (I have zero familiarity with the buildbot machinery.)

Skipping the test on AIX sounds like a reasonable option, but I kinda *want* IBM developers running the Python test suite on AIX to see those failures, in the hope that they might then be motivated to push for a fix to the relevant AIX bug. :-)

> That sounds more like a process problem than a CPython codebase one. The ideal would be that the machinery sending those notifications can be configured to ignore known failures when deciding whether to send email. Is that remotely feasible? (I have zero familiarity with the buildbot machinery.)

If a test fails all the time and not randomly, a single email is sent at the first failure.

I'm annoyed by test_threading which crash randomly on AIX: https://bugs.python.org/issue40068 It's a known issue, I already fixed 3/4 of the issue, but I didn't fix the remaining part.

In the past, I already disabled AIX email notifications simply because there was nobody to fix issues, and so emails were just spam.


> but I kinda *want* IBM developers running the Python test suite on AIX to see those failures, in the hope that they might then be motivated to push for a fix to the relevant AIX bug. :-)

Well, that would be great.

I'm not sure if Michael Felt is still working on supporting AIX in Python. David Edelsohn might help.

nextafter is a known problem on AIX.  I believe that it is being addressed in newer releases of AIX.

Michael and I are helping the IBM AIX Open Source team to increase their attention on Python, but things only move so fast.

I have been experimenting with different hardware and AIX versions.

When building on AIX 5.3 - and the oldest libraries - test_math passes.

When I run the test on POWER8, using either xlc or gcc test_math fails with just one element of the test.

When I run the test on POWER6 I get many more errors - that I never had before. These are all after OS updates (I was not going to build for AIX 5.3 any more).

An idea I have now - that may explain the sudden change in behavior is if the libraries have been optimized to always use the DFP (decimal floating point) internally - for what, from the application perspective - is the normal - no HW acceleration for FP - interface.

I know there are ways to 'discover' this, but I'll need to write some tests so that I can see - if linking to different libraries actuates DFP performance counters yes and no.

At this point - this feels like the a potential explanation.

I investigated another problem with nextafter() in 2015 and opened an internal IBM AIX PMR.  At the time it was not using decimal float code.

The earlier problem was the handling of -0.0.  At the time, the code was hand-written assembly language that did not check for IEEE floating point corner cases.

> The earlier problem was the handling of -0.0.  At the time, the code was hand-written assembly language that did not check for IEEE floating point corner cases.

I'm quite happy that my hand written tests detect bugs in nextafter() implementations ;-)

How can we fix the buildbot? Add #ifdef in mathmodule.c to implement the special cases, but only on AIX? Skip the test?

There seems to be a lot of interaction of OS level and compiler used.

* Waiting for the next bot run to get a different compiler.

+++
AIX 6.1.6 and older libraries - no test errors reported

AIX 7.1.4 and newer libraries - when using the binary built on 6.1.6 (AIX 6.1 TL6) - no error

AIX 7.1.4, using xlc-v13.1.2 (try and buy version), same AIX level as the bot (7.1 TL4 SP8) - no error

Back to the bot: AIX 7.1 TL4 SP8 and gcc-4.7.4 - strange errors. See, e.g., https://buildbot.python.org/all/#/builders/302/builds/373/steps/5/logs/stdio with additional errors such as:

======================================================================
FAIL: testHypotAccuracy (test.test_math.MathTests) (hx='0x1.89d8c423ea0c6p+29', hy='0x1.d35dcfe902bc3p+29', x=825956484.4892814, y=980138493.1263355)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_math.py", line 867, in testHypotAccuracy
    self.assertEqual(hypot(x, y), z)
AssertionError: 1281747081.1271062 != 1281747081.127106

Waiting for bot run 374 - to see if the results change when the compiler changes.

I'll try moving the bot to another system - as the system the bot is on is more than just the bot. Maybe there are side-effects coming in, unexpectedly, from other sources. ** The two systems just mentioned are fresh installs.

> Back to the bot: AIX 7.1 TL4 SP8 and gcc-4.7.4

The latest GCC version is GCC 10. Is it still relevant to test GCC 4.7 released 8 years ago? (Well, I'm not sure that the C compiler explains all issues.)

I believe that Michael was trying to probe under what circumstances the failure appears.  But, not GCC 4.7 is not relevant.

Yes, just probing, the version of gcc is irrelevant.

What I do believe is important is that bot run 374, 375 and 376 passed - On AIX 7.1 TL4 SP8.

The failure starting with 377 is an undefined variable.

"./Modules/posixmodule.c", line 15146.53: 1506-045 (S) Undeclared identifier SPLICE_F_MOVE.
"./Modules/posixmodule.c", line 15147.57: 1506-045 (S) Undeclared identifier SPLICE_F_NONBLOCK.
"./Modules/posixmodule.c", line 15148.53: 1506-045 (S) Undeclared identifier SPLICE_F_MORE.

So, let's put this on hold. I'll get a new environment built up specific for testing python on AIX. Hopefully by Friday.

> "./Modules/posixmodule.c", line 15146.53: 1506-045 (S) Undeclared identifier SPLICE_F_MOVE.

This is unrelated: https://bugs.python.org/issue41625#msg381259 Please continue the discussion this SPLICE there.

[Victor]

> How can we fix the buildbot? Add #ifdef in mathmodule.c to implement the special cases, but only on AIX? Skip the test?

I'm not super-keen on using #ifdefs to implement the special-case handling _just_ for AIX: that opens the door to a labyrinth of #ifdef'ery working around various different problems on various different platforms. If we're going to handle special cases ourselves, let's do it for all platforms.

But I'd also be fine with skipping the test (just on AIX, of course) for now. If we can also find a way to remind ourselves to revisit once the upstream bug has been fixed, so much the better.