Skip to content

/ sparse

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add numba extension support for COO objects #307

Merged
merged 4 commits into from Jan 20, 2020
Merged

ENH: Add numba extension support for COO objects #307

merged 4 commits into from Jan 20, 2020

Conversation

@eric-wieser
Copy link
Contributor

eric-wieser commented Jan 17, 2020 •
edited

Over in pygae/clifford, we're finding we want to be able to write optimized loops using arr.coords and arr.data.

This aims to be the minimum amount needed to make that work.

Marking as draft since this has no tests

cc @stuartarchibald, @luk-f-a
@eric-wieser
eric-wieser reviewed Jan 17, 2020
View changes
sparse/_coo/numba_extension.py Outdated Show resolved Hide resolved
@eric-wieser
eric-wieser reviewed Jan 17, 2020
View changes
sparse/_coo/numba_extension.py Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Jan 17, 2020 •
edited

Codecov Report

Merging #307 into master will increase coverage by 0.42%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #307      +/-   ##
==========================================
+ Coverage    93.5%   93.93%   +0.42%     
==========================================
  Files          17       19       +2     
  Lines        2140     2308     +168     
==========================================
+ Hits         2001     2168     +167     
- Misses        139      140       +1
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 17, 2020

I'm willing to put this in, provided we find a reviewer. I'm afraid my experience with Numba isn't enough to review this PR.
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 17, 2020

Or there are thorough tests -- More than willing to learn as I go!
@lgtm-com
Copy link

lgtm-com bot commented Jan 17, 2020

This pull request introduces 3 alerts when merging 9f20773 into 9e1213a - view on LGTM.com

new alerts:

  • 2 for Unused import
  • 1 for Wrong number of arguments in a class instantiation
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch 2 times, most recently from 0e20f78 to 9210ba7 Jan 17, 2020
@lgtm-com
Copy link

lgtm-com bot commented Jan 17, 2020

This pull request introduces 1 alert when merging 9210ba7 into 9e1213a - view on LGTM.com

new alerts:

  • 1 for Unused import
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch 2 times, most recently from 9089f3d to 667cd34 Jan 17, 2020
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 17, 2020

Or there are thorough tests

Added some tests that should hopefully cover everything here
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch from 667cd34 to bc79529 Jan 17, 2020
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 17, 2020

Could you perform a quick memory test and see if creating a humongous number of objects doesn't blow up memory? Doesn't need to be in the code.

Also, adding the fill_value would be nice.
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 17, 2020

Also, adding the fill_value would be nice.

Perhaps selfish, but since my application leaves that as 0, I'd prefer to leave that to a follow-up. Hopefully with something like this patch in place, it will be easier for others to work out how to add properties.
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch 2 times, most recently from ab77f64 to bfba121 Jan 17, 2020
@eric-wieser eric-wieser marked this pull request as ready for review Jan 17, 2020
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch from bfba121 to ba686f0 Jan 17, 2020
@eric-wieser
eric-wieser reviewed Jan 17, 2020
View changes
sparse/_coo/numba_extension.py Outdated Show resolved Hide resolved
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 17, 2020

Could you perform a quick memory test and see if creating a humongous number of objects doesn't blow up memory?

I don't have a good feel for what such a test would look like.
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 17, 2020

I don't have a good feel for what such a test would look like.

Just something that boxes and unboxes random COO objects in a loop.
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 17, 2020

Perhaps selfish, but since my application leaves that as 0, I'd prefer to leave that to a follow-up. Hopefully with something like this patch in place, it will be easier for others to work out how to add properties.

It is, but there's a correctness problem here. At least check if it's zero?
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 17, 2020 •
edited

It is, but there's a correctness problem here.

Ugh, that's unfortunate. Probably easier to add it than to work out how to insert an exception somewhere. (edit: done)
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch 2 times, most recently from 0e2bd10 to 8cb6821 Jan 17, 2020
@eric-wieser
ENH: Add numba extension support for COO objects
Loading status checks…
9dca6c0
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch from 8cb6821 to 9dca6c0 Jan 17, 2020
@eric-wieser
eric-wieser reviewed Jan 17, 2020
View changes
sparse/_coo/numba_extension.py Show resolved Hide resolved
@hameerabbasi
hameerabbasi reviewed Jan 18, 2020
View changes
sparse/_coo/numba_extension.py Outdated Show resolved Hide resolved
@hameerabbasi
hameerabbasi reviewed Jan 20, 2020
View changes
sparse/_coo/numba_extension.py Show resolved Hide resolved
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 20, 2020

You need to run black to fix the format checks. 😄 Otherwise this is good to go.
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch from c443814 to b419c38 Jan 20, 2020
@hameerabbasi
hameerabbasi reviewed Jan 20, 2020
View changes
sparse/_coo/numba_extension.py Show resolved Hide resolved
eric-wieser added 3 commits Jan 20, 2020
@eric-wieser
BUG: Handle errors correctly in `box` and `unbox`
e390515
@eric-wieser
MAINT: Use contextlib.ExitStack to flatten all the nesting
a156ffc
@eric-wieser
MAINT: Add a context manager to encapsulate early returns
Loading status checks…
7e8545c
@eric-wieser eric-wieser force-pushed the eric-wieser:numba-extension branch from b419c38 to 7e8545c Jan 20, 2020
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 20, 2020

You need to run black to fix the format checks. 😄 Otherwise this is good to go.

Decided that a better choice was to avoid having such long expressions - updated to not need any line-wrapping.
@hameerabbasi
hameerabbasi reviewed Jan 20, 2020
View changes
sparse/_coo/numba_extension.py Show resolved Hide resolved
@eric-wieser
Copy link
Contributor Author

eric-wieser commented Jan 20, 2020

@hameerabbasi, let me know if you want anything else here, I'm losing track.
@hameerabbasi hameerabbasi merged commit 40138e0 into pydata:master Jan 20, 2020
5 checks passed
5 checks passed
Documentation Build #20200120.4 succeeded
Details
LGTM analysis: Python No new or fixed alerts
Details
Tests #20200120.4 succeeded
Details
codecov/patch 100% of diff hit (target 93.5%)
Details
codecov/project 93.93% (+0.42%) compared to 9e1213a
Details
@hameerabbasi
Copy link
Collaborator

hameerabbasi commented Jan 20, 2020

Thanks, @eric-wieser!
@eric-wieser eric-wieser mentioned this pull request Jan 23, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stuartarchibald stuartarchibald

@hameerabbasi hameerabbasi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
@eric-wieser @hameerabbasi @stuartarchibald
You can’t perform that action at this time.