bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

asvetlov · 2019-06-24T15:30:15Z

The resurrection of #13630
Now without dangling threads in tests :)

https://bugs.python.org/issue35621

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

asvetlov · 2019-06-24T15:38:22Z

I tested it with manual adding 1-second timeout in spawned waiter threads to emulate slow test machine.
After making sure that ./python -Werror -m test -v --fail-env-changed test_asyncio doesn't produce an error I removed the sleep.

asvetlov · 2019-06-24T15:43:13Z

Lib/asyncio/unix_events.py

@@ -1035,11 +1052,6 @@ def __exit__(self, a, b, c):
    def add_child_handler(self, pid, callback, *args):
        assert self._forks, "Must use the context manager"

-        if self._loop is None:


The check is replaced by watcher.is_active() call in subprocess transport

1st1

LGTM overall. Thanks for pushing this through, Andrew.

Doc/library/asyncio-policy.rst

Doc/library/asyncio-subprocess.rst

Lib/asyncio/unix_events.py

Co-Authored-By: Yury Selivanov <yury@magic.io>

asvetlov · 2019-06-30T09:00:21Z

Thanks for review!

miss-islington · 2019-06-30T09:55:02Z

Thanks @asvetlov for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

bedevere-bot · 2019-06-30T09:55:16Z

GH-14484 is a backport of this pull request to the 3.8 branch.

…ted in non-main thread (pythonGH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

…ted in non-main thread (pythonGH-14344)

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>

PR #5862 Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344 * https://docs.python.org/3/library/asyncio-policy.html#asyncio.MultiLoopChildWatcher Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com> (cherry picked from commit 7080a8b)

asvetlov added 7 commits Jun 3, 2019

Close pid waiters on watcher.close()

3695efd

Revert "Revert "bpo-35621: Support running subprocesses in asyncio wh…

3559581

…en loop is executed in non-main thread (python#13630)" (pythonGH-13793)" This reverts commit 9535aff.

Merge branch 'dangling-threads-in-watcher' into threaded-waiter

b18e6b9

Merge remote-tracking branch 'upstream/master' into threaded-waiter

1bc1d8e

Work on

ef1e21c

Merge branch 'master' into threaded-waiter

c982e4a

Fix dangling thread warnings in tests

6912ec7

asvetlov requested review from 1st1 and gpshead as code owners Jun 24, 2019

the-knights-who-say-ni added the CLA signed label Jun 24, 2019

bedevere-bot added the awaiting core review label Jun 24, 2019

asvetlov commented Jun 24, 2019

View changes

Update documentation

efb19c0

asvetlov added expert-asyncio needs backport to 3.8 labels Jun 24, 2019

Add docstrings

eb07403

1st1 approved these changes Jun 30, 2019

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jun 30, 2019

asvetlov and others added 8 commits Jun 30, 2019

Update Doc/library/asyncio-policy.rst

6aa3357

Co-Authored-By: Yury Selivanov <yury@magic.io>

Fix doc text

25d9ac2

Update Doc/library/asyncio-policy.rst

ae3a706

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst

8baf48c

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Doc/library/asyncio-policy.rst

6e6a783

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py

1e1281a

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update Lib/asyncio/unix_events.py

936aeb0

Co-Authored-By: Yury Selivanov <yury@magic.io>

Update docs

cc5a752

asvetlov merged commit 0d671c0 into python:master Jun 30, 2019

bedevere-bot removed the awaiting merge label Jun 30, 2019

asvetlov deleted the threaded-waiter branch Jun 30, 2019

bedevere-bot removed the needs backport to 3.8 label Jun 30, 2019

miss-islington added a commit that referenced this pull request Jun 30, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

bf8cb31

…ted in non-main thread (GH-14344) (cherry picked from commit 0d671c0) Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-35621: Support running subprocesses in asyncio when loop is execu…

af4d7c9

…ted in non-main thread (pythonGH-14344)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-35621: Support running subprocesses in asyncio when loop is execu…

b340eb2

…ted in non-main thread (pythonGH-14344)

webknjaz mentioned this pull request Jan 25, 2021

pytest-xdist breaks asyncio-based code expecting to be run in the main thread pytest-dev/pytest-xdist#620

Open

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

a7f5a2a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

7b7a324

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz mentioned this pull request Jan 25, 2021

Re-introduce pytest-xdist in supported envs aio-libs/aiohttp#5431

Draft

5 tasks

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Jan 25, 2021

Use ThreadedChildWatcher in tests where available

e3df8bc

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 21, 2021

Use ThreadedChildWatcher in tests where available

0df32d5

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

webknjaz added a commit to aio-libs/aiohttp that referenced this pull request Feb 24, 2021

Use ThreadedChildWatcher in tests where available

bea959a

Refs: * pytest-dev/pytest-xdist#620 * https://stackoverflow.com/a/58614689/595220 * https://bugs.python.org/issue35621 * python/cpython#14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot

asvetlov commented Jun 24, 2019

asvetlov Jun 24, 2019

1st1 left a comment

asvetlov commented Jun 30, 2019

miss-islington commented Jun 30, 2019

bedevere-bot commented Jun 30, 2019

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

Conversation

asvetlov commented Jun 24, 2019 • edited by bedevere-bot

asvetlov commented Jun 24, 2019

asvetlov Jun 24, 2019

Choose a reason for hiding this comment

1st1 left a comment

asvetlov commented Jun 30, 2019

miss-islington commented Jun 30, 2019

bedevere-bot commented Jun 30, 2019

asvetlov commented Jun 24, 2019 •

edited by bedevere-bot