Skip to content

BUG: f2py: make callbacks threadsafe#16519

Merged
charris merged 2 commits into
numpy:masterfrom
pv:f2py-threadsafe-cb
Jun 7, 2020
Merged

BUG: f2py: make callbacks threadsafe#16519
charris merged 2 commits into
numpy:masterfrom
pv:f2py-threadsafe-cb

Conversation

@pv
Copy link
Copy Markdown
Member

@pv pv commented Jun 7, 2020

Make f2py Python callback functions threadsafe by putting the associated globals to thread-local storage. Use compiler TLS if possible (implemented for gcc & msvc), and fall back to Python-provided TLS otherwise.

Fixes gh-16513

Comment thread numpy/f2py/cb_rules.py Outdated
Comment thread numpy/f2py/cb_rules.py Outdated
Comment thread numpy/f2py/cb_rules.py Outdated
Comment thread numpy/f2py/cb_rules.py Outdated
Comment thread numpy/f2py/cb_rules.py Outdated
@charris
Copy link
Copy Markdown
Member

charris commented Jun 7, 2020

The formatting cleanup is nice. I wouldn't complain if you made it a separate PR and put the rest of this on top after it goes in. There is still more cleanup to do, but I think others can take care of it.

@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020

Style changes in gh-16520 separately.
Will rebase once that's merged.

@pv pv force-pushed the f2py-threadsafe-cb branch 3 times, most recently from cfe6340 to 1ba1dbe Compare June 7, 2020 18:23
@pv pv force-pushed the f2py-threadsafe-cb branch from 1ba1dbe to d7b3177 Compare June 7, 2020 18:27
@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020

Rebased.

@charris
Copy link
Copy Markdown
Member

charris commented Jun 7, 2020

@pv IIRC, swapping was the main reason we could not use the newer interfaces that are in the include files. If you have any ideas for an public swap procedure I'd like to hear them.

@melissawm There are lots of opportunity for additional code beautification here, much of it whitespace :)

@sturlamolden
Copy link
Copy Markdown
Contributor

With C11 or C++11 we can use thread_local macro from the standard library, defined in threads.h. That will work on any standards compliant C or C++ compiler, even if The non-standard __thread and __declspec(thread) extensions are not defined.

@charris
Copy link
Copy Markdown
Member

charris commented Jun 7, 2020

Let's put this in, thanks Pauli. I'm a bit undecided on backporting, I'd rather keep changes to a minimum for the 1.19 release.

@charris charris added the 09 - Backport-Candidate PRs tagged should be backported label Jun 7, 2020
@charris charris merged commit 581eab3 into numpy:master Jun 7, 2020
@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020

@charris: hmm, TBH I'm not sure what you mean by swapping and what are the newer interfaces here.

@charris
Copy link
Copy Markdown
Member

charris commented Jun 7, 2020

I'm talking about the inline functions that replaced macros in order to hide ndarray internals. I'm not sure that is what this is about, but I recall data pointers being swapped somewhere in f2py and that kept us from an easy path to using

#define NPY_NO_DEPRECATED_API NPY_API_VERSION

Now that I think about it, it would be good to have a release note for this, maybe under improvements.

@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020

Ok, added release note and Sturla's suggestion in #16524

@seberg
Copy link
Copy Markdown
Member

seberg commented Jun 7, 2020

Sorry, just curious... Is this approach fully safe with restoring always the same state? I noticed there is a longjmp in there. And we still have some annoying issues in the fft code with longjmp not being restored.

It seems to me however, that we may have to make it a doubly-linked list to really always correctly restore things if multiple threads finish in random order? But maybe I just need to think about it longer...

@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020 via email

@pv
Copy link
Copy Markdown
Member Author

pv commented Jun 7, 2020 via email

@seberg
Copy link
Copy Markdown
Member

seberg commented Jun 8, 2020

Ah, nvm. totally different situation since its on the stack here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: f2py produces wrapper code that are not threadsafe

4 participants