Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upbpo-29854: Fix segfault in call_readline() #728
Conversation
This comment has been minimized.
This comment has been minimized.
the-knights-who-say-ni
commented
Mar 19, 2017
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA. This is necessary for legal reasons before we can look at your contribution. Please follow these steps to help rectify the issue:
Thanks again to your contribution and we look forward to looking at it! |
This comment has been minimized.
This comment has been minimized.
mention-bot
commented
Mar 19, 2017
|
@nirs, thanks for your PR! By analyzing the history of the files in this pull request, we identified @vadmium, @loewis and @ronaldoussoren to be potential reviewers. |
This comment has been minimized.
This comment has been minimized.
|
I signed the CLA few years ago, but my github user name was missing in bpo. |
| line = (const char *)history_get(length)->line; | ||
| else | ||
| hist_ent = history_get(length); | ||
| line = (const char *)hist_ent ? hist_ent->line : ""; |
This comment has been minimized.
This comment has been minimized.
vadmium
Mar 23, 2017
Member
Why do you cast hist_ent? The original cast was added in Git revision 2525dc8, though I have doubts about the reason [avoiding a compiler warning when assigning (char *) to (const char *)]. It is better to avoid casts if possible; they can mask real errors and warnings.
This comment has been minimized.
This comment has been minimized.
serhiy-storchaka
Mar 23, 2017
Member
The cast is not needed. A compiler shouldn't emit a warning when assigning char * to const char *.
And more, I have doubts about the priority of the cast and the trinary operator. Does the above code is equivalent to (const char *)(hist_ent ? hist_ent->line : "") or to ((const char *)hist_ent) ? hist_ent->line : ""?
This comment has been minimized.
This comment has been minimized.
nirs
Mar 23, 2017
Author
Contributor
I think the correct cast would be:
hist_ent ? (const char *)hist_ent->line : "";
But the cast is probably not needed, I kept it only to minimize changes which are not required to fix this issue.
| else | ||
| hist_ent = history_get(length); | ||
| line = (const char *)hist_ent ? hist_ent->line : ""; | ||
| } else |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
nirs
Mar 24, 2017
Author
Contributor
I tried to keep the current code style of this module, see for example write_history_file, read_history_file, read_init_file, setup_readline.
This comment has been minimized.
This comment has been minimized.
|
Changes in version 2:
|
This comment has been minimized.
This comment has been minimized.
|
Version 3 adds the missing NEWS entry. |
| import readline | ||
| import sys | ||
| history_file = "{}" |
This comment has been minimized.
This comment has been minimized.
vadmium
May 20, 2017
•
Member
Might be safer to use ascii or {!a} (repr or {!r} on Python 2), in case the path has special characters; e.g. double quotes are allowed on Unix, Windows user profiles may be non-ASCII. Or you could change directory to the temp directory in the child process, or pass it as a CLI argument.
This comment has been minimized.
This comment has been minimized.
nirs
May 21, 2017
Author
Contributor
The path is a temporary path, so it should not have quotes. I think a better way would be to use command line arguments or environment variables instead.
|
|
||
| script = """ | ||
| import readline | ||
| import sys |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| inputrc = os.path.join(temp_dir, "inputrc") | ||
| with io.open(inputrc, "wb") as f: | ||
| f.write(b"set history-size %d\n" % history_size) | ||
| env = os.environ.copy() |
This comment has been minimized.
This comment has been minimized.
vadmium
May 20, 2017
Member
I think it would be safer and clearer to make the copy with env = dict(os.environ). Os.environ.copy is undocumented.
This comment has been minimized.
This comment has been minimized.
| self.assertEqual(len(lines), history_size) | ||
| self.assertEqual(lines[-1].strip(), b"last input") | ||
| finally: | ||
| shutil.rmtree(temp_dir) |
This comment has been minimized.
This comment has been minimized.
berkerpeksag
May 20, 2017
Member
Can't we use self.addCleanup(shutil.rmtree, temp_dir) instead of this try..finally block?
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| temp_dir = tempfile.mkdtemp() | ||
| try: | ||
| inputrc = os.path.join(temp_dir, "inputrc") | ||
| with io.open(inputrc, "wb") as f: |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
serhiy-storchaka
Jun 16, 2017
Member
I don't think there is a significant difference between Python 2 builtin open() and io.open() here.
This comment has been minimized.
This comment has been minimized.
| readline.read_history_file(history_file) | ||
| input() | ||
| readline.write_history_file(history_file) | ||
| """.format(history_file) |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| @@ -322,6 +322,9 @@ Core and Builtins | |||
| Extension Modules | |||
| ----------------- | |||
|
|
|||
| - bpo-29854: Fix segfault in readline when using readline's history-size | |||
| option. | |||
This comment has been minimized.
This comment has been minimized.
berkerpeksag
May 20, 2017
Member
Please add "Patch by Nir Soffer." (and add two spaces after the full stop)
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Changes in version 4:
|
This comment has been minimized.
This comment has been minimized.
|
Version 4 is ready for 25 days, it would be nice if someone can take a look :-) |
|
The last version of the PR looks good to me, thank you! I still prefer to use |
This comment has been minimized.
This comment has been minimized.
|
LGTM if use builtin |
This comment has been minimized.
This comment has been minimized.
|
Changes in version 5:
|
| temp_dir = tempfile.mkdtemp() | ||
| try: | ||
| inputrc = os.path.join(temp_dir, "inputrc") | ||
| with io.open(inputrc, "wb") as f: |
This comment has been minimized.
This comment has been minimized.
serhiy-storchaka
Jun 16, 2017
Member
I don't think there is a significant difference between Python 2 builtin open() and io.open() here.
| try: | ||
| inputrc = os.path.join(temp_dir, "inputrc") | ||
| with io.open(inputrc, "wb") as f: | ||
| f.write(b"set history-size %d\n" % history_size) |
This comment has been minimized.
This comment has been minimized.
serhiy-storchaka
Jun 16, 2017
Member
This is the first time I see using bytes formatting in tests! Since bytes formatting is supported in Python 3.5 this is okay.
| self.assertEqual(len(lines), history_size) | ||
| self.assertEqual(lines[-1].strip(), b"last input") | ||
| finally: | ||
| shutil.rmtree(temp_dir) |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@serhiy-storchaka, I saw your comment about using test.support.temp_dir after I uploaded version 5. See nirs@45931f5 using it. I can squash it into the tests patch or send another pull request later if you like. |
This comment has been minimized.
This comment has been minimized.
|
@nirs, sorry, I added my comments few days ago, but forgot to publish them. When I approved the PR, they became visible. I still think that it would be better to use |
This comment has been minimized.
This comment has been minimized.
|
@serhiy-storchaka are you waiting for another change? |
This comment has been minimized.
This comment has been minimized.
|
I think we need to move the NEWS entry into I liked nirs@45931f5. Could you integrate it into this PR? I can do the merging if Serhiy is busy with other PRs. Thanks! (and sorry for taking too long to merge this.) |
This comment has been minimized.
This comment has been minimized.
No, I just waited for other reviewers in the case they have other comments. I left this on Berker. |
This comment has been minimized.
This comment has been minimized.
|
@berkerpeksag, ok, I will squash nirs/cpython@45931f5 into the test patch, and update the news item as you suggest. |
This enable testing custom readline configuration using the INPUTRC environment variable, or passing arguments to the child process in a clean way.
readline segfaults on input() if the number of items in the history file is equal or more to history size * 2. This issue affects only GNU readline. When using libedit emulation system history size option does not work.
This comment has been minimized.
This comment has been minimized.
|
@berkerpeksag are you sure blurb is ready? see python/core-workflow#153 |
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but python should not segfault even if readline returns unexpected value.
This comment has been minimized.
This comment has been minimized.
|
Changes in version 3:
@berkerpeksag, @serhiy-storchaka please review. |
This comment has been minimized.
This comment has been minimized.
|
Thanks! |
This comment has been minimized.
This comment has been minimized.
|
@berkerpeksag, @serhiy-storchaka, @vadmium thanks for reviewing! Do you want me to backport this to older versions? |
This comment has been minimized.
This comment has been minimized.
|
@nirs it would be better to fix the test failures reported at https://bugs.python.org/issue29854#msg297876 and https://bugs.python.org/issue29854#msg297877 before doing the backports. Do you have some time to take a look at those failures? |
This comment has been minimized.
This comment has been minimized.
|
@berkerpeksag sure, I'll take a look. |
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work.
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work.
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work.
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work. This is a backport of the actual fix from master without the test, since the test depends on new run_pty() helper which is not available in 2.7.
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work.
If history-length is set in .inputrc, and the history file is double the history size (or more), history_get(N) returns NULL, and python segfaults. Fix that by checking for NULL return value. It seems that the root cause is incorrect handling of bigger history in readline, but Python should not segfault even if readline returns unexpected value. This issue affects only GNU readline. When using libedit emulation system history size option does not work. This is a backport of the actual fix from master without the test, since the test depends on new run_pty() helper which is not available in 2.7.
nirs commentedMar 19, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.
It seems that the root cause is incorrect handling of bigger history in
readline, but python should not segfault even if readline returns
unexpected value.