Skip to content

/ loky

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH add ResourceTracker to improve ressource clean up #204

Merged
merged 23 commits into from Jun 6, 2019
Merged

ENH add ResourceTracker to improve ressource clean up #204

merged 23 commits into from Jun 6, 2019

Conversation

@pierreglaser
Copy link
Collaborator

@pierreglaser pierreglaser commented Mar 19, 2019

This PR aims at extending the current SemaphoreTracker to make it able to track more generic named resources, such as folders, and potentially memmap and shared memory.

Many small changes in files because of the name refactoring. The main change is the registering/unregistering API extension: when one wants to track a new resource, one must (along with the resource name), specify the resource type. This is necessary for the RessourceTracker to know which cleanup function to use for each resource before shutdown.

Base branch: https://github.com/tomMoral/loky/pull/202
@ogrisel
Copy link
Collaborator

@ogrisel ogrisel commented Mar 20, 2019

French typo everywhere: ressource => resource :)
@ogrisel ogrisel changed the title [WIP] ENH: move to a more generic RessourceTracker [WIP] ENH: move to a more generic ResourceTracker Mar 21, 2019
@ogrisel
ogrisel approved these changes Mar 21, 2019
View changes
Copy link
Collaborator

@ogrisel ogrisel left a comment

A batch of comments. Overall this LGTM as well (there are duplicated comments with the tacker_pid bugfix PR.
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
tests/test_resource_tracker.py Show resolved Hide resolved
@pierreglaser
Copy link
Collaborator Author

@pierreglaser pierreglaser commented Mar 21, 2019 •
edited

I need to somehow rebase with #202.
@pierreglaser pierreglaser force-pushed the pierreglaser:ressource-tracker branch from ac477db to 2cbbc88 Mar 21, 2019
@tomMoral
tomMoral reviewed Mar 24, 2019
View changes
Copy link
Collaborator

@tomMoral tomMoral left a comment

A first batch of comments. This is going in the right direction :)
loky/backend/popen_loky_posix.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/spawn.py Outdated Show resolved Hide resolved
tests/_executor_mixin.py Show resolved Hide resolved
tests/test_resource_tracker.py Outdated Show resolved Hide resolved
tests/test_resource_tracker.py Outdated Show resolved Hide resolved
tests/test_resource_tracker.py Show resolved Hide resolved
@pierreglaser pierreglaser force-pushed the pierreglaser:ressource-tracker branch 2 times, most recently from d9af7d9 to cc754f7 Apr 5, 2019
@pierreglaser
Copy link
Collaborator Author

@pierreglaser pierreglaser commented Apr 11, 2019

@tomMoral I addressed most of the comments from your previous review.

There is one thing though apparently, spawn_passfd (used to create the resource tracker) is POSIX only. For this reason, the resource_tracker tests are failing on Windows.

Recent support has been added to enable files descriptors to child processes in windows, (post windows 7, and post python3.7). Should we dive further into it? Or stick to POSIX-only support?
@pierreglaser pierreglaser changed the title [WIP] ENH: move to a more generic ResourceTracker ENH: move to a more generic ResourceTracker Apr 11, 2019
@pierreglaser pierreglaser changed the title ENH: move to a more generic ResourceTracker [WIP] ENH: move to a more generic ResourceTracker May 10, 2019
@pierreglaser pierreglaser mentioned this pull request May 29, 2019
Merged
@pierreglaser pierreglaser force-pushed the pierreglaser:ressource-tracker branch from 36da509 to 88eeccc Jun 4, 2019
@pierreglaser pierreglaser changed the title [WIP] ENH: move to a more generic ResourceTracker ENH: move to a more generic ResourceTracker Jun 5, 2019
@tomMoral
tomMoral reviewed Jun 5, 2019
View changes
Copy link
Collaborator

@tomMoral tomMoral left a comment

A (large) batch of comments, many are nitpicks. This looks very good, thanks @pierreglaser for the hard work! I will do a second pass in the afternoon if possible.
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
loky/backend/resource_tracker.py Show resolved Hide resolved
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
tests/test_resource_tracker.py Show resolved Hide resolved
tests/test_resource_tracker.py Outdated Show resolved Hide resolved
tests/test_resource_tracker.py Show resolved Hide resolved
loky/backend/popen_loky_win32.py Show resolved Hide resolved
loky/backend/popen_loky_win32.py Show resolved Hide resolved
@pierreglaser
Copy link
Collaborator Author

@pierreglaser pierreglaser commented Jun 5, 2019

OK @tomMoral I addressed most of your comments. I am somewhat hesitant about rewriting the probe handling because I'd rather be coherent with what we did in python/cpython#13222, but if you feel strongly about this I'll change this code.
pierreglaser added 14 commits Mar 12, 2019
@pierreglaser
CLN spurious commented line
5e956ad
@pierreglaser
MNT refactor semaphore_tracker->ressource_tracker
75612ef
@pierreglaser
TST activate relevant tests on windows
17136cc
@pierreglaser
CLN fix small rebasing mistake
d4d8b44
@pierreglaser
FIX correct pytest syntax
a722bda
@pierreglaser
windows ressource_tracker (with all fds inherited)
f710ff8
@pierreglaser
ENH implement resource_tracker on windows
0a1407c
@pierreglaser
FIX fix AttributeError on linux
484dc33
@pierreglaser
CLN tweaks
dae5041
@pierreglaser
TST test resource-tracker on windows
03cfe2d
@pierreglaser
FIX remove unecessary interaction with pytest
4023527
@pierreglaser
TST skip ResourceTracker tests on python2
e11e3c4
@pierreglaser
MNT implement resource_tracker (win32, python 2)
a729dc2
@pierreglaser
CLN various cleanups
221c246
pierreglaser and others added 6 commits May 29, 2019
@pierreglaser
CLN more cleanups
64ee4a1
@pierreglaser
CI try to fix PyPy failures
48c4f8b
@pierreglaser
MNT use noop for PROBE
856c4df
@pierreglaser @tomMoral
Update loky/backend/resource_tracker.py
95a5138 
Co-Authored-By: Thomas Moreau <thomas.moreau.2010@gmail.com>
@pierreglaser
address review comments
c75aa3d
@pierreglaser
CLN typo
Loading status checks…
36092f9
@pierreglaser pierreglaser force-pushed the pierreglaser:ressource-tracker branch from 115554f to 36092f9 Jun 5, 2019
@ogrisel
ogrisel reviewed Jun 5, 2019
View changes
loky/backend/resource_tracker.py Outdated Show resolved Hide resolved
@tomMoral
Copy link
Collaborator

@tomMoral tomMoral commented Jun 6, 2019

I am somewhat hesitant about rewriting the probe handling because I'd rather be coherent with what we did in python/cpython#13222, but if you feel strongly about this I'll change this code.

I understand but we already have some slight differences so I think it would be fine. The question is more whether it is useful or not. I think it allows you to avoid having an empty set 'noop': set() and this makes more sense to me but I am not feeling this strongly about it so if you do not like it, I am fine with the current implementation.
@tomMoral
tomMoral approved these changes Jun 6, 2019
View changes
Copy link
Collaborator

@tomMoral tomMoral left a comment •
edited

LGTM! Nice work @pierreglaser
I have just a last comment. If you don't want to get rid of the noop, I am fine with merging this as is.
loky/backend/popen_loky_win32.py Show resolved Hide resolved
pierreglaser added 3 commits Jun 6, 2019
@pierreglaser
CLN no probe, friendlier user messages
Loading status checks…
03accef
@pierreglaser
MNT mention discussion about windows handles inheritance
Loading status checks…
5e641d0
@pierreglaser
DOC fix comment
Loading status checks…
77dd9d8
@tomMoral tomMoral changed the title ENH: move to a more generic ResourceTracker ENH add ResourceTracker to improve ressource clean up Jun 6, 2019
@tomMoral tomMoral merged commit 6beefdb into joblib:master Jun 6, 2019
4 checks passed
4 checks passed
codecov/patch 91.73% of diff hit (target 85%)
Details
codecov/project 89.94% (+0.05%) compared to 048faf8
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel

@tomMoral tomMoral

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
@pierreglaser @ogrisel @tomMoral