bpo-31061: fix crash in asyncio speedup module #2966
Conversation
thehesiod
changed the title
|
this is my first cpython PR, let me know what else needs to be done to get this into any other releases it's required in. |
Modules/_asynciomodule.c
Outdated
| if (Future_CheckExact(fut)) { | ||
| /* When fut is subclass of Future, finalizer is called from | ||
| * subtype_dealloc. | ||
| */ | ||
| PyObject_GC_Track(self); |
There was a problem hiding this comment.
Is this really needed?
There was a problem hiding this comment.
sorry, you're right. it's needed.
There was a problem hiding this comment.
I'm just following what was done in https://bugs.python.org/issue26617. I think it's worth someone grepping through the whole codebase, I saw several other places that maybe needed this. It seems like every dealloc should have a PyObject_GC_UnTrack at the top?
There was a problem hiding this comment.
Maybe, noddy4 example should do it at first :)
There was a problem hiding this comment.
btw how do we get this into 3.6.3 as well?
There was a problem hiding this comment.
It seems like every dealloc should have a PyObject_GC_UnTrack at the top?
When the type has Py_TPFLAGS_HAVE_GC.
lru_cache_dealloc in _functoolsmodule seems need it.
There was a problem hiding this comment.
I filed an issue for that. http://bugs.python.org/issue31095
There was a problem hiding this comment.
k, should I remove the LRU change I added here?
There was a problem hiding this comment.
Yes, please. asyncio speedup module is introduced from Python 3.6.
It make easy to backport.
|
Since |
|
Oh, sorry, macro version API broke Windows build and you used public APIs to fix it. |
Modules/_asynciomodule.c
Outdated
| @@ -962,14 +962,18 @@ FutureObj_dealloc(PyObject *self) | |||
| { | |||
| FutureObj *fut = (FutureObj *)self; | |||
|
|
|||
| PyObject_GC_UnTrack(self); | |||
|
|
|||
| if (Future_CheckExact(fut)) { | |||
There was a problem hiding this comment.
Please move PyObject_GC_UnTrack after this if block, and remove Track and Untrack in the block.
There was a problem hiding this comment.
but that would mean this (34fae03) is wrong too then no?
because it first does an UnTrack: 34fae03#diff-c3cf251f16d5a03a9e7d4639f2d6f998L1113
With the same Track/Untrack pattern here:
34fae03#diff-c3cf251f16d5a03a9e7d4639f2d6f998R1129
There was a problem hiding this comment.
current code:
UnTrack()
Track()
PyObject_CallFinalizerFromDealloc()
UnTrack()
I proposed:
PyObject_CallFinalizerFromDealloc()
UnTrack()
34fae03 does other things between UnTrack() and Track()
But this code does only Future_CheckExact() and it is safe.
There was a problem hiding this comment.
k, I'm claiming ignorance since I don't know what these macros achieve :) I'd personally feel safer if we had a test case, do you know how to create a testcase to cause this to crash? Mine takes a day to reproduce in a production environment :(
There was a problem hiding this comment.
CPython's circular reference GC uses doubly linked list to track object which can be member of circular reference.
PyObject_GC_UnTrack() removes object from the list, and PyObject_GC_Track() insert the object to the list.
Despite very undetarministic multithreading, I can cause SEGV quickly with this code:
import asyncio
import gc
gc.set_debug(gc.DEBUG_STATS)
class Evil:
def __del__(self):
gc.collect()
def main():
f = asyncio.Future()
f.set_result(Evil())
for i in range(100):
main()
There was a problem hiding this comment.
nice! I'll add testcases and add your suggestions, thanks so much for all the help!
There was a problem hiding this comment.
k, added FutureObj testcase, have a good test case for the TaskObj? Not quite sure how to trigger it.
|
Could you add news entry? |
|
added news and reverted LRU change |
|
should we put as many fixes as we can in this PR, or create another PR with as many as we can? I need the defaultdict one too...and maybe even others as who knows what third party libraries do. I feel like we've opened a pandoras box of crashers :) |
|
I already created #2974. |
Include/objimpl.h
Outdated
|
|
||
| /* | ||
| See description in Doc/c-api/gcsupport.rst | ||
| */ |
There was a problem hiding this comment.
Please remove it.
We can reach the document easily with searching API name.
methane
changed the title
|
The latest revision LGTM. But can you use |
|
@1st1 should I remove the one in NEWS? Also was hoping I could add a testcase for Task but couldn't figure out how to repro it, especially given Task inherits from Future so I don't want to trigger the Future's crash |
Misc/NEWS
Outdated
| @@ -10,6 +10,8 @@ What's New in Python 3.7.0 alpha 1? | |||
| Core and Builtins | |||
| ----------------- | |||
|
|
|||
| - bpo-31061: Fixed a crash when using asyncio and threads. | |||
There was a problem hiding this comment.
Last nit: the change in Misc/NEWS needs to be reverted, we need only blurb file.
Yes, please!
I'm trying to come up with a test. |
|
The following test crashes unpatched CPython: import gc
def test_task_del_collect(self):
async def coro():
return Evil()
class Evil:
def __del__(self):
gc.collect()
for i in range(100):
task = self.new_task(self.loop, coro())
self.loop.run_until_complete(task) |
|
The test also fails if we fix only Task, but not Future. |
|
so I had something similar but it doesn't crash: import gc
import asyncio
class Evil:
def __del__(self):
gc.collect()
async def run():
return Evil()
loop = asyncio.get_event_loop()
for i in range(100):
loop.run_until_complete(asyncio.Task(run()))
# f = asyncio.Future()
# f.set_result(Evil()) |
|
ah, figured it out, had to have multiple tasks pending in the run loop. |
Lib/test/test_asyncio/test_tasks.py
Outdated
| def run(): | ||
| return Evil() | ||
|
|
||
| yield from asyncio.gather(*[asyncio.Task(run()) for _ in range(100)]) |
There was a problem hiding this comment.
This test won't run. yield from makes test_task_del_collect() a generator and test suite doesn't know how to run them.
Please change to
self.loop.run_until_complete(
asyncio.gather(*[asyncio.Task(run()) for _ in range(100)]))
|
ya realized as I was going to work, this should fix it |
Lib/test/test_asyncio/test_tasks.py
Outdated
| @@ -4,6 +4,7 @@ | |||
| import contextlib | |||
| import functools | |||
| import io | |||
| import gc | |||
There was a problem hiding this comment.
Another nit: 'gc' import should be before 'functools' -- we sort imports alphabetically in CPython.
There was a problem hiding this comment.
may be worth running a linter on CI to catch these
There was a problem hiding this comment.
I agree, good idea. BTW help is always welcome by python/core-workflow. You're an awesome contributor, we'd love you to continue! :)
There was a problem hiding this comment.
you're too kind :) Will keep that in mind if we run into any other cpython issues.
|
this set of PRs is a massive win for Python. It would be nice if a core dev could look and find a way to nip it at the bud to avoid these hacky calls, as shown they're highly error prone. Who knows how many third party modules are missing them too. |
(cherry picked from commit de34cbe)
|
GH-2984 is a backport of this pull request to the 3.6 branch. |
|
Do we have C Task/Future in 3.5? |
|
No. asyncio speedup is introduced at 3.6 |
https://bugs.python.org/issue31061
follows what was done in: https://bugs.python.org/issue26617
fallout in: http://bugs.python.org/issue31095
The text was updated successfully, but these errors were encountered: