Skip to content

/ cupy

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import `using_allocator` in `cupy.cuda` #2951

Merged
merged 5 commits into from Feb 18, 2020
Merged

Import `using_allocator` in `cupy.cuda` #2951

merged 5 commits into from Feb 18, 2020

Conversation

@jakirkham
Copy link
Member

jakirkham commented Jan 13, 2020

cc @emcastillo
@emcastillo emcastillo self-assigned this Jan 14, 2020
@emcastillo
Copy link
Member

emcastillo commented Jan 14, 2020

depends on #2952
@emcastillo emcastillo added this to the v8.0.0a1 milestone Jan 14, 2020
@niboshi
Copy link
Member

niboshi commented Jan 14, 2020

Why does it depend on #2952?
@emcastillo
Copy link
Member

emcastillo commented Jan 14, 2020 •
edited

They share a commit, @jakirkham should make them independent or rebase this one after #2952 is merged
@niboshi
Copy link
Member

niboshi commented Jan 14, 2020

It seems they don't share a commit and I think we can merge this PR and discard the other one.
@emcastillo
Copy link
Member

emcastillo commented Jan 14, 2020 •
edited

Document using_allocator function commit is duplicated in both, although it has a different hash.
I agree with closing #2952 and merging this one instead
@jakirkham
Copy link
Member Author

jakirkham commented Jan 14, 2020

PR ( #2952 ) could be backported to v7. This PR cannot be backported.
@niboshi
Copy link
Member

niboshi commented Jan 14, 2020

Document using_allocator function commit is duplicated in both, although it has a different hash.

And the content is different, too.

@jakirkham
I think it's opposite: this PR can be backported and #2952 shouldn't.
cupy.cuda.memory.using_allocator is a "wrong" signature and users shouldn't use it.
Fortunately this signature is not documented yet. If it became documented in v7 (by backporting #2952) and this PR would be merged (to v8), v7 and v8 would be incompatible.

This PR breaks existing code that use cupy.cuda.memory.using_allocator, but we can't avoid that anyway. Maybe we can say it's their responsibility because it's not documented.
@jakirkham
Copy link
Member Author

jakirkham commented Jan 14, 2020

Ok happy to defer 🙂
@jakirkham jakirkham mentioned this pull request Jan 14, 2020
Closed
@emcastillo
Copy link
Member

emcastillo commented Jan 14, 2020

I just got the flow of the two prs😅
sorry for the missunderstanding
@jakirkham
Copy link
Member Author

jakirkham commented Jan 15, 2020

No worries. Sorry for not being clearer.
@emcastillo
Copy link
Member

emcastillo commented Jan 15, 2020

Might be the sphinx error related to the function being a context manager?
@jakirkham
Copy link
Member Author

jakirkham commented Jan 25, 2020

It looks like Travis CI is using Python 3.5.6. Is that needed or can we update to a newer Python version?
@jakirkham
Copy link
Member Author

jakirkham commented Jan 27, 2020

My guess is using contextmanager in Cython results in something that Sphinx is unable to parse. This issue ( cython/cython#1750 ) and this blogpost seem relevant here. What do we think about just moving the context manager into a pure Python file?
@emcastillo
Copy link
Member

emcastillo commented Jan 27, 2020

That approach should be fine.
@jakirkham jakirkham force-pushed the jakirkham:imp_using_allocator branch from bcfd395 to 86abf0d Feb 4, 2020
@jakirkham
Copy link
Member Author

jakirkham commented Feb 4, 2020

Have pushed a few changes. These rewrite using_allocator to use only get_allocator and set_allocator (to make it possible to move the function). Then have pulled this code into cupy.cuda's __init__.py. This seems to fix the Sphinx error. However as I've rewritten the function, it would be good to run this on Jenkins and also get a review. Please let me know what you think 🙂
@emcastillo
Copy link
Member

emcastillo commented Feb 7, 2020

Jenkins, test this please
@pfn-ci-bot
Copy link
Collaborator

pfn-ci-bot commented Feb 7, 2020

Successfully created a job for commit 86abf0d:
@chainer-ci
Copy link
Collaborator

chainer-ci commented Feb 7, 2020

Jenkins CI test (for commit 86abf0d, target branch master) failed with status FAILURE.
@kmaehashi kmaehashi removed the cat:test label Feb 10, 2020
@jakirkham
Copy link
Member Author

jakirkham commented Feb 13, 2020

Had missed updating the tests. This seems to fix them except for one test that is checking for a ValueError to be raised by set_allocator when called within a using_allocator context. Will try to work out some changes to fix this.
@jakirkham jakirkham force-pushed the jakirkham:imp_using_allocator branch from 0d9282e to 1f0e295 Feb 13, 2020
jakirkham added 3 commits Jan 13, 2020
@jakirkham
Import `using_allocator` in `cupy.cuda`
b6a7379
@jakirkham
Document `using_allocator` function
2972015
@jakirkham
Make `using_allocator` a private generator
fdd5fd1 
As wrapping the Cython `using_allocator` with `contextmanager` is
problematic for documentation, make room for a Python wrapper that can
be a `contextmanager`. Do this by dropping the `contextmanager` bit from
the Cython implementation, mark it as private, and finally stop
importing it in `cupy.cuda`. Can follow this by wrapping it at the
Python level.
jakirkham added 2 commits Feb 13, 2020
@jakirkham
Create Python `using_allocator` wrapper
051b5e0 
Provide a `contextmanager` at the Python level by simply calling the
Cython generator in a loop and `yield`ing it's contents. This should
only ever be one `yield`, but it does ensure the correct behavior with
respect to setup and teardown. As there is no special error handling in
the Python function, skip this part. Make a note in the docs that this
is a wrapper and why it exists.
@jakirkham
Update `using_allocator` tests to right path
Loading status checks…
e12cc54
@jakirkham jakirkham force-pushed the jakirkham:imp_using_allocator branch from 1f0e295 to e12cc54 Feb 13, 2020
@jakirkham
Copy link
Member Author

jakirkham commented Feb 13, 2020

Ok, simplified things a fair bit. Expect this should work better. Could use another run on Jenkins when you have a chance @emcastillo 🙂
@jakirkham
Copy link
Member Author

jakirkham commented Feb 13, 2020

Should add this worked for me locally as well. 😄
@emcastillo emcastillo modified the milestones: v8.0.0a1, v8.0.0b1 Feb 14, 2020
@emcastillo
Copy link
Member

emcastillo commented Feb 18, 2020

Jenkins, test this please
@pfn-ci-bot
Copy link
Collaborator

pfn-ci-bot commented Feb 18, 2020

Successfully created a job for commit e12cc54:
@chainer-ci
Copy link
Collaborator

chainer-ci commented Feb 18, 2020

Jenkins CI test (for commit e12cc54, target branch master) failed with status FAILURE.
@jakirkham
Copy link
Member Author

jakirkham commented Feb 18, 2020

The Jenkins status says the build failed, but if I look at the dashboard everything is passing. Am I missing something?

Screen Shot 2020-02-17 at 7 38 23 PM
@emcastillo
Copy link
Member

emcastillo commented Feb 18, 2020

Jenkins went nuts, lets try it again.
Jenkins, test this please.
@pfn-ci-bot
Copy link
Collaborator

pfn-ci-bot commented Feb 18, 2020

Successfully created a job for commit e12cc54:
@jakirkham
Copy link
Member Author

jakirkham commented Feb 18, 2020

No worries. Thanks for the help @emcastillo! 😄 Would also be interested to get your take on this approach 😉
@chainer-ci
Copy link
Collaborator

chainer-ci commented Feb 18, 2020

Jenkins CI test (for commit e12cc54, target branch master) succeeded!
@emcastillo
emcastillo approved these changes Feb 18, 2020
View changes
@mergify mergify bot merged commit c704320 into cupy:master Feb 18, 2020
11 checks passed
11 checks passed
Jenkins Build finished.
Details
Summary 1 rule matches
Details
codecov/patch 100% of diff hit (target 0%)
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls First build on imp_using_allocator at 79.001%
Details
pfn-public-ci/cupy.py2.cv.examples test succeeded
Details
pfn-public-ci/cupy.py2.cv.gpu test succeeded
Details
pfn-public-ci/cupy.py3.amd test succeeded
Details
pfn-public-ci/cupy.py3.cv.examples test succeeded
Details
pfn-public-ci/cupy.py3.cv.gpu test succeeded
Details
emcastillo added a commit to emcastillo/cupy that referenced this pull request Feb 18, 2020
@mergify @emcastillo
Merge pull request cupy#2951 from jakirkham/imp_using_allocator
f710539 
Import `using_allocator` in `cupy.cuda`
@emcastillo emcastillo mentioned this pull request Feb 18, 2020
Merged
@jakirkham jakirkham deleted the jakirkham:imp_using_allocator branch Feb 18, 2020
@niboshi niboshi removed the no-compat label Feb 18, 2020
@jakirkham jakirkham mentioned this pull request Feb 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emcastillo emcastillo

Assignees

@emcastillo emcastillo

Labels
cat:enhancement st:test-and-merge to-be-backported
Projects
None yet
Milestone
v8.0.0b1
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
@jakirkham @emcastillo @niboshi @pfn-ci-bot @chainer-ci @kmaehashi
You can’t perform that action at this time.