gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222

ogbiggles · 2025-04-30T19:28:52Z

When shutdown is called with wait=False, the executor thread keeps running even after the ProcessPoolExecutor's state is reset. The executor then tries to replenish the worker processes pool resulting in an error and a potential hang when it comes across a worker that has died. Fixed the issue by having _adjust_process_count() return without doing anything if the ProcessPoolExecutor's state has been reset.

Added unit tests to validate two scenarios:
max_workers < num_tasks (exception)
max_workers > num_tasks (exception + hang)

Issue: ProcessPoolExecutor raises exception or hangs the process when shutdown(wait=False) and worker throws exception #132969

…sk exited abnormally When shutdown is called with wait=False, the executor thread keeps running even after the ProcessPoolExecutor's state is reset. The executor then tries to replenish the worker processes pool resulting in an error and a potential hang when it comes across a worker that has died. Fixed the issue by having _adjust_process_count() return without doing anything if the ProcessPoolExecutor's state has been reset. Added unit tests to validate two scenarios: max_workers < num_tasks (exception) max_workers > num_tasks (exception + hang)

bedevere-app · 2025-04-30T19:28:57Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

… into origin/main Skip forkserver-dependent test on Windows

python-cla-bot · 2025-04-30T21:11:28Z

All commit authors signed the Contributor License Agreement.

encukou · 2025-05-05T15:04:43Z

@hugovk: If this is indeed a blocker for the beta, +1 for me -- it shouldn't break anything.
For a proper review, I'd need more time to familiarize myself with this part of the code. (@gpshead, should I?)

But, as week-old issue that affects 3.12+, I think it's OK to defer it.

@itamaro: Did you intend this to block the beta?

itamaro · 2025-05-05T15:28:58Z

Did you intend this to block the beta?

Should have said it here, not only in Discord :)
I wasn't sure whether this should block the release, so added the tag to make sure someone at least takes a look, considering beta 1 is imminent!

hugovk · 2025-05-05T15:50:18Z

Thanks all, it's been looked at and sounds like it can wait, so let's defer it.

Lib/concurrent/futures/process.py

Lib/test/test_concurrent_futures/test_shutdown.py

Misc/NEWS.d/next/Library/2025-04-30-19-32-18.gh-issue-132969.EagQ3G.rst

…use @unittest.skipIf annotation to skip test on win32, add some more comments

… into origin/main

…s by swiching to "spawn"

Misc/NEWS.d/next/Library/2025-04-30-19-32-18.gh-issue-132969.EagQ3G.rst

…s on windows

…t_method is "fork" or ("forkserver" and windows)

Lib/test/test_concurrent_futures/test_shutdown.py

…ntext() as the mp_context

…ten message

…wn(wait=False) to prevent resource leak when running CI

Lib/test/test_concurrent_futures/test_shutdown.py

…er" if not windows

YvesDup · 2025-05-13T12:51:09Z

LGTM

ogbiggles · 2025-05-13T17:32:32Z

LGTM

Thank you @YvesDup for the thorough review of my first PR, much appreciated.

picnixz

I'm not very fond of a long NEWS, especially with the "A combination of ...", but it's the full CHANGELOG, so why not.

picnixz · 2025-05-15T12:50:00Z

Lib/concurrent/futures/process.py

+        # gh-132969: avoid error if shutdown(wait=False) is called and state is reset
+        # and leaving the executor still running


Suggested change

# gh-132969: avoid error if shutdown(wait=False) is called and state is reset

# and leaving the executor still running

# gh-132969: avoid leaving the executor running after

# calling shutdown(wait=False) and state is reset.

I think this is what you wanted to say right?

picnixz · 2025-05-15T12:50:46Z

Lib/test/test_concurrent_futures/test_shutdown.py

+        f1 = executor.submit(ProcessPoolShutdownTest._good_task_gh_132969, 1)
+        f2 = executor.submit(ProcessPoolShutdownTest._failing_task_gh_132969, 2)
+        f3 = executor.submit(ProcessPoolShutdownTest._good_task_gh_132969, 3)
+        result:int = 0


Suggested change

result:int = 0

result = 0

picnixz · 2025-05-15T12:53:28Z

Misc/NEWS.d/next/Library/2025-04-30-19-32-18.gh-issue-132969.EagQ3G.rst

@@ -0,0 +1 @@
+Prevent the ProcessPoolExecutor executor thread, which remains running when shutdown is called with wait=False, from attempting to adjust the pool's worker processes after the object state has already been reset during shutdown. A combination of conditions, including a worker process having terminated abormally, resulted in an exception and a potential hang when the still-running executor thread attempted to replace dead workers within the pool.


Suggested change

Prevent the ProcessPoolExecutor executor thread, which remains running when shutdown is called with wait=False, from attempting to adjust the pool's worker processes after the object state has already been reset during shutdown. A combination of conditions, including a worker process having terminated abormally, resulted in an exception and a potential hang when the still-running executor thread attempted to replace dead workers within the pool.

Prevent the :class:`~concurrent.futures.ProcessPoolExecutor` executor thread,

which remains running when :meth:`shutdown(wait=False)

<concurrent.future.ProcessPoolExecutor.shutdown>`, from

attempting to adjust the pool's worker processes after the object state has already been reset during shutdown.

A combination of conditions, including a worker process having terminated abormally,

resulted in an exception and a potential hang when the still-running executor thread

attempted to replace dead workers within the pool.

YvesDup · 2025-05-22T19:39:27Z

@ogbiggles Would you apply the last suggestions ?

bedevere-app bot added the awaiting review label Apr 30, 2025

bedevere-app bot mentioned this pull request Apr 30, 2025

ProcessPoolExecutor raises exception or hangs the process when shutdown(wait=False) and worker throws exception #132969

Open

📜🤖 Added by blurb_it.

1774445

ogbiggles mentioned this pull request Apr 30, 2025

[3.12] gh-132969: Fix exception/hang shutdown(wait=False) and a task exited abnormally #133220

Closed

blurb-it bot and others added 4 commits April 30, 2025 16:48

pythongh-132969 Skip forkserver-dependent test on Windows

0938eeb

Merge branch 'origin/main' of github.com:ogbiggles/cpython-fix-132969…

dbc0ba8

… into origin/main Skip forkserver-dependent test on Windows

pythongh-132969 Skip forkserver dependent tests on Windows

99ab321

Merge branch 'main' into origin/main

fe3650c

ZeroIntensity added the topic-multiprocessing label May 4, 2025