Add benchmarks for ctypes function call overhead #197
Conversation
|
Wanted to add a note: This is definitely a "microbenchmark" that isn't a great example of a real-world workload. |
Mostly LGTM. There are some slight changes we should consider though, particularly using runner.bench_time_func() for micro benchmarks.
pyperformance/venv.py
Outdated
| benchmark_dir = os.path.dirname(bench.metafile) | ||
| if os.path.isfile(os.path.join(benchmark_dir, "setup.py")): | ||
| requirements._add(benchmark_dir) |
There was a problem hiding this comment.
+1 to adding support for a per-benchmark setup.py. It makes sense.
If we are going to explicitly support it then it should be documented in doc/custom_benchmarks.rst.
There was a problem hiding this comment.
Also, as an official feature, it would probably make sense to add something like Benchmark.setup_py (set if the file exists). At the least, the code here would change:
| benchmark_dir = os.path.dirname(bench.metafile) | |
| if os.path.isfile(os.path.join(benchmark_dir, "setup.py")): | |
| requirements._add(benchmark_dir) | |
| if bench.setup_py: | |
| requirements._add(benchmark_dir) |
If fact, it would make sense to update Requirements.from_benchmarks() to use it, which would make these lines unnecessary.
There was a problem hiding this comment.
Taking this further, we should also consider adding some entry to the benchmark's metadata (and thus pyproject.toml) to explicitly indicate the setup.py should be used.
(If we want the presence of setup.py to imply it should be installed by default, then maybe we add an opt-out indicator instead. That can be done later if needed, but the level of effort right now is small.)
| void_foo_void() | ||
| int_foo_int(1) | ||
| void_foo_int(1) | ||
| void_foo_int_int(1, 2) | ||
| void_foo_int_int_int(1, 2, 3) | ||
| void_foo_int_int_int_int(1, 2, 3, 4) | ||
| void_foo_constchar(b"bytes") |
There was a problem hiding this comment.
The real benefit of micro-benchmarks is that it narrows down where performance regressions might be. With that in mind, would these different signatures have enough independent potential for regression that it would it make sense to have a separate benchmark for each? Would it be worth bothering even if they did?
pyperformance/data-files/benchmarks/bm_ctypes/run_benchmark_argtypes.py
Outdated
Show resolved
Hide resolved
Thanks for the changes. I just have a couple more suggestions.
pyperformance/venv.py
Outdated
| if bench.setup_py: | ||
| self._add(os.path.dirname(bench.metafile)) |
There was a problem hiding this comment.
| if bench.setup_py: | |
| self._add(os.path.dirname(bench.metafile)) | |
| if bench.setup_py: | |
| self._add(bench.setup_py) |
There was a problem hiding this comment.
I'm not sure I get this one. setup_py is not a boolean to indicate there is one, and then you pass the directory it contains to pip (not the file itself). Maybe renaming this flag to needs_build would be more accurate (since that how pip see it -- this directory has something that needs to be built, and setup.py is just an implementation detail...)
There was a problem hiding this comment.
Sorry for the confusion. My point is that we should record the directory name on the Benchmark object rather than using os.path.dirname() here. We could call it something like setup_py_dir if you prefer.
doc/custom_benchmarks.rst
Outdated
| @@ -333,6 +333,7 @@ tool.extra_opts [str] X | |||
| tool.inherits file | |||
| tool.runscript file X | |||
| tool.datadir file X | |||
| tool.setup_py bool | |||
There was a problem hiding this comment.
setup_py might be too ambiguous. Consider something like:
| tool.setup_py bool | |
| tool.install_setup bool |
I can also imagine someone wanting to use a different filename that setup.py (e.g. if they have multiple different setup scripts). So:
| tool.setup_py bool | |
| tool.install_setup bool | |
| tool.setupscript str |
The default for setupscript would be 'setup.py'.
There was a problem hiding this comment.
If adding "setupscript" seems premature, we can skip it. (I'm just trying to keep custom benchmarks in mind. Any friction in writing a benchmark can mean the difference between someone doing it or not.)
There was a problem hiding this comment.
Yeah -- in addition, I don't think pip makes it possible. You don't tell it "here's a setup.py", you tell it "here's a source directory to install, and it knows that setup.py is one of the ways to build it..."
pyperformance/_benchmark.py
Outdated
| @property | ||
| def setup_py(self): | ||
| return self._get_metadata_value('setup_py', False) |
There was a problem hiding this comment.
Let's return the filename (and check if the file exists) here:
| @property | |
| def setup_py(self): | |
| return self._get_metadata_value('setup_py', False) | |
| @property | |
| def setup_py(self): | |
| try: | |
| return self._setup_py | |
| except AttributeError: | |
| self._setup_py = None | |
| if self._get_metadata_value('install_setup', True): | |
| filename = self._get_metadata_value('setupscript', 'setup.py') | |
| rootdir = os.path.dirname(self.metafile) | |
| #rootdir = self._get_rootdir() | |
| filename = os.path.join(rootdir, filename) | |
| if os.path.isdir(filename): | |
| filename = os.path.join(filename, 'setup.py') | |
| if os.path.exists(filename): | |
| self._setup_py = filename | |
| return self._setup_py |
Also, I'm thinking it might be best for the default "install_setup" to be True. So, by default, the setup script is installed if it is there. That seems like the more likely user expectation.
There was a problem hiding this comment.
See above for why the filename may not be necessary.
But no harm in changing the default to True (I think).
This additionally adds support for building C extensions as part of creating a benchmark's virtual environment, since that didn't seem to be possible prior to this change.
See faster-cpython/ideas#370 for some background discussion.