Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase the precision of the summary in trace module #112192

Closed
gaogaotiantian opened this issue Nov 17, 2023 · 7 comments
Closed

Increase the precision of the summary in trace module #112192

gaogaotiantian opened this issue Nov 17, 2023 · 7 comments
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@gaogaotiantian
Copy link
Member

gaogaotiantian commented Nov 17, 2023 •
edited by bedevere-app bot
Loading

Feature or enhancement

Proposal:

With the current trace module, you'll have a summary like this if you enable summary

lines   cov%   module   (path)
  154    24%   _weakrefset   (/home/gaogaotiantian/programs/mycpython/Lib/_weakrefset.py)
 1293     0%   ast   (/home/gaogaotiantian/programs/mycpython/Lib/ast.py)
  160     5%   bz2   (/home/gaogaotiantian/programs/mycpython/Lib/bz2.py)
  133    39%   cProfile   (/home/gaogaotiantian/programs/mycpython/Lib/cProfile.py)
  903     7%   collections.__init__   (/home/gaogaotiantian/programs/mycpython/Lib/collections/__init__.py)

I propose to increase the precision of cov% to one decimal place so you'll have something like

lines    cov%   module   (path)
  154   24.7%   _weakrefset   (/home/gaogaotiantian/programs/mycpython/Lib/_weakrefset.py)
 1293    0.2%   ast   (/home/gaogaotiantian/programs/mycpython/Lib/ast.py)
  160    5.0%   bz2   (/home/gaogaotiantian/programs/mycpython/Lib/bz2.py)
  133   39.8%   cProfile   (/home/gaogaotiantian/programs/mycpython/Lib/cProfile.py)
  903    7.8%   collections.__init__   (/home/gaogaotiantian/programs/mycpython/Lib/collections/__init__.py)

This may seem a bit non-sense at the first glance, but it's related to a very practical question - how do we know our coverage improved after some new test cases? With integer percentage, we have a 1% resolution, which means we can't tell if there's a one line improvement if the file is larger than 100 lines. For a file that has 1000 lines, the resolution is 10 lines! It's not trivial!

By introducing only a single digit, we improved the resolution to 0.1%, which is arbitray I know. But, in practical, most Python modules have lines between 100 and 1000 lines.

Check a quick "wc -l Lib/*.py" result. Remember, this is without comments so the real line number is smaller 147 Lib/__future__.py 16 Lib/__hello__.py 108 Lib/_aix_support.py 1173 Lib/_collections_abc.py 251 Lib/_compat_pickle.py 162 Lib/_compression.py 396 Lib/_markupbase.py 335 Lib/_opcode_metadata.py 574 Lib/_osx_support.py 147 Lib/_py_abc.py 2649 Lib/_pydatetime.py 6425 Lib/_pydecimal.py 2683 Lib/_pyio.py 285 Lib/_pylong.py 103 Lib/_sitebuiltins.py 566 Lib/_strptime.py 242 Lib/_threading_local.py 205 Lib/_weakrefset.py 188 Lib/abc.py 17 Lib/antigravity.py 2656 Lib/argparse.py 1836 Lib/ast.py 584 Lib/base64.py 900 Lib/bdb.py 118 Lib/bisect.py 344 Lib/bz2.py 195 Lib/cProfile.py 796 Lib/calendar.py 400 Lib/cmd.py 371 Lib/code.py 1132 Lib/codecs.py 151 Lib/codeop.py 166 Lib/colorsys.py 469 Lib/compileall.py 1283 Lib/configparser.py 814 Lib/contextlib.py 4 Lib/contextvars.py 305 Lib/copy.py 217 Lib/copyreg.py 451 Lib/csv.py 1592 Lib/dataclasses.py 9 Lib/datetime.py 11 Lib/decimal.py 2056 Lib/difflib.py 933 Lib/dis.py 2856 Lib/doctest.py 2052 Lib/enum.py 313 Lib/filecmp.py 442 Lib/fileinput.py 192 Lib/fnmatch.py 988 Lib/fractions.py 959 Lib/ftplib.py 1026 Lib/functools.py 167 Lib/genericpath.py 215 Lib/getopt.py 185 Lib/getpass.py 657 Lib/gettext.py 311 Lib/glob.py 250 Lib/graphlib.py 694 Lib/gzip.py 253 Lib/hashlib.py 603 Lib/heapq.py 219 Lib/hmac.py 1632 Lib/imaplib.py 3451 Lib/inspect.py 99 Lib/io.py 2341 Lib/ipaddress.py 64 Lib/keyword.py 191 Lib/linecache.py 1742 Lib/locale.py 356 Lib/lzma.py 2198 Lib/mailbox.py 645 Lib/mimetypes.py 666 Lib/modulefinder.py 192 Lib/netrc.py 901 Lib/ntpath.py 81 Lib/nturl2path.py 418 Lib/numbers.py 112 Lib/opcode.py 467 Lib/operator.py 1681 Lib/optparse.py 1152 Lib/os.py 1637 Lib/pathlib.py 2236 Lib/pdb.py 1814 Lib/pickle.py 2890 Lib/pickletools.py 529 Lib/pkgutil.py 1357 Lib/platform.py 911 Lib/plistlib.py 466 Lib/poplib.py 581 Lib/posixpath.py 655 Lib/pprint.py 615 Lib/profile.py 778 Lib/pstats.py 211 Lib/pty.py 212 Lib/py_compile.py 314 Lib/pyclbr.py 2851 Lib/pydoc.py 326 Lib/queue.py 237 Lib/quopri.py 1000 Lib/random.py 194 Lib/reprlib.py 219 Lib/rlcompleter.py 318 Lib/runpy.py 167 Lib/sched.py 71 Lib/secrets.py 603 Lib/selectors.py 250 Lib/shelve.py 345 Lib/shlex.py 1581 Lib/shutil.py 92 Lib/signal.py 671 Lib/site.py 1109 Lib/smtplib.py 967 Lib/socket.py 858 Lib/socketserver.py 7 Lib/sre_compile.py 7 Lib/sre_constants.py 7 Lib/sre_parse.py 1507 Lib/ssl.py 201 Lib/stat.py 1480 Lib/statistics.py 309 Lib/string.py 272 Lib/stringprep.py 15 Lib/struct.py 2209 Lib/subprocess.py 374 Lib/symtable.py 340 Lib/tabnanny.py 2904 Lib/tarfile.py 925 Lib/tempfile.py 497 Lib/textwrap.py 28 Lib/this.py 1733 Lib/threading.py 381 Lib/timeit.py 140 Lib/token.py 545 Lib/tokenize.py 747 Lib/trace.py 1227 Lib/traceback.py 560 Lib/tracemalloc.py 76 Lib/tty.py 4176 Lib/turtle.py 341 Lib/types.py 3432 Lib/typing.py 793 Lib/uuid.py 595 Lib/warnings.py 663 Lib/wave.py 674 Lib/weakref.py 620 Lib/webbrowser.py 206 Lib/zipapp.py 724 Lib/zipimport.py

As we can obviously tell, most of the files have line numbers between 100 and 1000. Even for files that have more than 1000 lines, they are normally less than 3000 lines. Again, the "executable lines" are less because wc counts empty lines and comments.

So, 0.1% is arbitrary, but I think it's the sweet spot for coverage report - it's not too verbose, but it can almost always show coverage changes for common python modules.

But, this is not backward compatible right? Yes, it's not. However, the summary is mainly used for human to read, not for machines. So I don't think there are a lot of code relying on the output of the summary. I believe most of the users would be happier than disturbed by the new resolution.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs
@gaogaotiantian gaogaotiantian added the type-feature A feature request or enhancement label Nov 17, 2023
@arhadthedev arhadthedev added performance Performance or resource usage stdlib Python modules in the Lib dir labels Nov 18, 2023
@gaogaotiantian
Copy link
Member Author

Hi, @ambv , you have just worked with the coverage with the full test suite. Do you think a summary with an extra digit help? Could you share some thoughts about this? Thanks!

@rruuaanng
Copy link
Contributor

rruuaanng commented Nov 18, 2024 •
edited
Loading

I think this is great, if we make a change to the code, even if it only has an effect of %0.1, this won't be visible at all, for example %30.1, the 0.1 has been wiped out by the %d. I'll commit a PR : ) @gaogaotiantian

@bedevere-app bedevere-app bot mentioned this issue Nov 18, 2024
Merged
@gaogaotiantian
Copy link
Member Author

The problem of this issue is never the lack of implementation. This is a breaking change so we need more support from the core dev team. However, the module is not in active develop state so few people are interested in changing it. That's why this is blocked.

@rruuaanng
Copy link
Contributor

rruuaanng commented Nov 18, 2024 •
edited
Loading

Ah, in that case, we can promote it together. We can discuss it in PR. I think this demand is very good. I will try to absorb suggestions from other core developers on this event :)

@gaogaotiantian
Copy link
Member Author

gaogaotiantian commented Nov 18, 2024 •
edited
Loading

Also next time, it would be better to ask for the OP whether they want to submit a PR first (unless of course, the OP has left the discussion or obviously not interested). From the prototype result someone may be able to infer that I already had an implementation.

It's okay to work on something stale or blocked for a while, but it's also important to understand why it was there. Feature requests are more difficult to land than obvious bug fixes. If you want to work on CPython (and maybe build reputation), bugs might be better starters.

As for this specific feature request, I'd like to see it happen, but I also understand this is not a trivial thing. We probably need a couple core devs to look at this and that may not be available anytime soon (that's actually the reason I left this request the first place). Anyway, good luck!

@vstinner
Copy link
Member

But, this is not backward compatible right? Yes, it's not. However, the summary is mainly used for human to read, not for machines. So I don't think there are a lot of code relying on the output of the summary. I believe most of the users would be happier than disturbed by the new resolution.

IMO it's ok to change the output format in Python 3.14.

@rruuaanng
Copy link
Contributor

But, this is not backward compatible right? Yes, it's not. However, the summary is mainly used for human to read, not for machines. So I don't think there are a lot of code relying on the output of the summary. I believe most of the users would be happier than disturbed by the new resolution.

IMO it's ok to change the output format in Python 3.14.

What do you think of this proposal? It shows the changes accurately, and it seems to be a good idea.

erlend-aasland pushed a commit that referenced this issue Dec 3, 2024
@rruuaanng
gh-112192: Increase the trace module coverage precision to one decimal (
12397a5 
#126972)
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 8, 2025
@rruuaanng @srinivasreddy
pythongh-112192: Increase the trace module coverage precision to one …
fbf712f 
…decimal (python#126972)
ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025
@rruuaanng @ebonnal
pythongh-112192: Increase the trace module coverage precision to one …
ea50a2e 
…decimal (python#126972)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vstinner @arhadthedev @gaogaotiantian @erlend-aasland @rruuaanng