Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-29854: Fix segfault in call_readline() #728

Merged
merged 3 commits into from Jul 7, 2017

Conversation

@nirs
Copy link
Contributor

nirs commented Mar 19, 2017

If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but python should not segfault even if readline returns
unexpected value.

@the-knights-who-say-ni

This comment has been minimized.

Copy link

the-knights-who-say-ni commented Mar 19, 2017

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA. This is necessary for legal reasons before we can look at your contribution. Please follow these steps to help rectify the issue:

  1. If you don't have an account on b.p.o, please create one
  2. Make sure your GitHub username is listed in "Your Details" at b.p.o
  3. If you have not already done so, please sign the PSF contributor agreement. The "bugs.python.org username " requested by the form is the "Login name" field under "Your Details".
  4. If you just signed the CLA, please wait at least one US business day and then check "Your Details" on bugs.python.org to see if your account has been marked as having signed the CLA (the delay is due to a person having to manually check your signed CLA)
  5. Reply here saying you have completed the above steps

Thanks again to your contribution and we look forward to looking at it!

@mention-bot

This comment has been minimized.

Copy link

mention-bot commented Mar 19, 2017

@nirs, thanks for your PR! By analyzing the history of the files in this pull request, we identified @vadmium, @loewis and @ronaldoussoren to be potential reviewers.

@nirs nirs force-pushed the nirs:readline-segfault branch from 08fa519 to 9f071fd Mar 19, 2017
@nirs nirs changed the title bpo-29854: Fix segtault in call_readline() bpo-29854: Fix segfault in call_readline() Mar 19, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Mar 19, 2017

I signed the CLA few years ago, but my github user name was missing in bpo.

line = (const char *)history_get(length)->line;
else
hist_ent = history_get(length);
line = (const char *)hist_ent ? hist_ent->line : "";

This comment has been minimized.

Copy link
@vadmium

vadmium Mar 23, 2017

Member

Why do you cast hist_ent? The original cast was added in Git revision 2525dc8, though I have doubts about the reason [avoiding a compiler warning when assigning (char *) to (const char *)]. It is better to avoid casts if possible; they can mask real errors and warnings.

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Mar 23, 2017

Member

The cast is not needed. A compiler shouldn't emit a warning when assigning char * to const char *.

And more, I have doubts about the priority of the cast and the trinary operator. Does the above code is equivalent to (const char *)(hist_ent ? hist_ent->line : "") or to ((const char *)hist_ent) ? hist_ent->line : ""?

This comment has been minimized.

Copy link
@nirs

nirs Mar 23, 2017

Author Contributor

I think the correct cast would be:

hist_ent ? (const char *)hist_ent->line : "";

But the cast is probably not needed, I kept it only to minimize changes which are not required to fix this issue.

else
hist_ent = history_get(length);
line = (const char *)hist_ent ? hist_ent->line : "";
} else

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Mar 23, 2017

Member

} and else should be on separate lines.

This comment has been minimized.

Copy link
@nirs

nirs Mar 24, 2017

Author Contributor

I tried to keep the current code style of this module, see for example write_history_file, read_history_file, read_init_file, setup_readline.

@nirs nirs force-pushed the nirs:readline-segfault branch from 9f071fd to 6861217 May 19, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented May 19, 2017

Changes in version 2:

  • Add failing test (serhiy-storchaka)
  • Remove unneeded cast in readline.c (vadmium)
@nirs nirs force-pushed the nirs:readline-segfault branch from 6861217 to 2811487 May 19, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented May 19, 2017

Version 3 adds the missing NEWS entry.

import readline
import sys
history_file = "{}"

This comment has been minimized.

Copy link
@vadmium

vadmium May 20, 2017

Member

Might be safer to use ascii or {!a} (repr or {!r} on Python 2), in case the path has special characters; e.g. double quotes are allowed on Unix, Windows user profiles may be non-ASCII. Or you could change directory to the temp directory in the child process, or pass it as a CLI argument.

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

The path is a temporary path, so it should not have quotes. I think a better way would be to use command line arguments or environment variables instead.


script = """
import readline
import sys

This comment has been minimized.

Copy link
@vadmium

vadmium May 20, 2017

Member

Is this needed?

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

Leftover, will remove.

inputrc = os.path.join(temp_dir, "inputrc")
with io.open(inputrc, "wb") as f:
f.write(b"set history-size %d\n" % history_size)
env = os.environ.copy()

This comment has been minimized.

Copy link
@vadmium

vadmium May 20, 2017

Member

I think it would be safer and clearer to make the copy with env = dict(os.environ). Os.environ.copy is undocumented.

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

I agree, will update in the next version.

self.assertEqual(len(lines), history_size)
self.assertEqual(lines[-1].strip(), b"last input")
finally:
shutil.rmtree(temp_dir)

This comment has been minimized.

Copy link
@berkerpeksag

berkerpeksag May 20, 2017

Member

Can't we use self.addCleanup(shutil.rmtree, temp_dir) instead of this try..finally block?

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Jun 16, 2017

Member

Or use test.support.temp_dir() or test.support.temp_cwd().

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

Good idea, will change.

temp_dir = tempfile.mkdtemp()
try:
inputrc = os.path.join(temp_dir, "inputrc")
with io.open(inputrc, "wb") as f:

This comment has been minimized.

Copy link
@berkerpeksag

berkerpeksag May 20, 2017

Member

Did you use io.open() to make backporting to 2.7 easier?

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Jun 16, 2017

Member

I don't think there is a significant difference between Python 2 builtin open() and io.open() here.

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

Yes, code that works in both python 3 and 2 is my intent.

readline.read_history_file(history_file)
input()
readline.write_history_file(history_file)
""".format(history_file)

This comment has been minimized.

Copy link
@berkerpeksag

berkerpeksag May 20, 2017

Member

Style nit: I'd rather use an f-string here.

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

f-string does not work on python 2, creating extra work :-)

Misc/NEWS Outdated
@@ -322,6 +322,9 @@ Core and Builtins
Extension Modules
-----------------

- bpo-29854: Fix segfault in readline when using readline's history-size
option.

This comment has been minimized.

Copy link
@berkerpeksag

berkerpeksag May 20, 2017

Member

Please add "Patch by Nir Soffer." (and add two spaces after the full stop)

This comment has been minimized.

Copy link
@nirs

nirs May 21, 2017

Author Contributor

Sure.

@nirs nirs force-pushed the nirs:readline-segfault branch from 2811487 to a67438c May 21, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented May 21, 2017

Changes in version 4:

  • Avoid undocumented os.environ.copy (vadmium)
  • Remove unneeded import (vadmium)
  • Pass configuraion to child process using environment variables instead of formatting (vadmium, berkerpeksag)
  • Use addCleanup instead of try-finally (berkerpeksag)
  • Adding missing "Patch by" to news entry (berkerpeksag)
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jun 15, 2017

Version 4 is ready for 25 days, it would be nice if someone can take a look :-)

Copy link
Member

berkerpeksag left a comment

The last version of the PR looks good to me, thank you!

I still prefer to use open() instead of io.open() (we are likely get a merge conflict in Misc/NEWS file when we backport it to 2.7 so we will need to do manual edit anyway and I don't think 2.7 is important to write slightly less idiomatic code in Python 3), but I'm going to left the ultimate decision to Martin and Serhiy.

@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Jun 16, 2017

LGTM if use builtin open().

@nirs nirs force-pushed the nirs:readline-segfault branch from a67438c to d2f511d Jun 16, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jun 16, 2017

Changes in version 5:

  • Use builtin open instead of io.open (io.open is required for backporting to 2.7).
temp_dir = tempfile.mkdtemp()
try:
inputrc = os.path.join(temp_dir, "inputrc")
with io.open(inputrc, "wb") as f:

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Jun 16, 2017

Member

I don't think there is a significant difference between Python 2 builtin open() and io.open() here.

try:
inputrc = os.path.join(temp_dir, "inputrc")
with io.open(inputrc, "wb") as f:
f.write(b"set history-size %d\n" % history_size)

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Jun 16, 2017

Member

This is the first time I see using bytes formatting in tests! Since bytes formatting is supported in Python 3.5 this is okay.

self.assertEqual(len(lines), history_size)
self.assertEqual(lines[-1].strip(), b"last input")
finally:
shutil.rmtree(temp_dir)

This comment has been minimized.

Copy link
@serhiy-storchaka

serhiy-storchaka Jun 16, 2017

Member

Or use test.support.temp_dir() or test.support.temp_cwd().

@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jun 16, 2017

@serhiy-storchaka, I saw your comment about using test.support.temp_dir after I uploaded version 5.

See nirs@45931f5 using it. I can squash it into the tests patch or send another pull request later if you like.

@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Jun 17, 2017

@nirs, sorry, I added my comments few days ago, but forgot to publish them. When I approved the PR, they became visible.

I still think that it would be better to use test.support.temp_dir() (it is more robust for testing on Windows), but this is not critical, and I approved the PR in any case.

@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 2, 2017

@serhiy-storchaka are you waiting for another change?

@berkerpeksag

This comment has been minimized.

Copy link
Member

berkerpeksag commented Jul 3, 2017

I think we need to move the NEWS entry into Misc/NEWS.d/next/Library now. You can use blurb to create a NEWS file.

I liked nirs@45931f5. Could you integrate it into this PR?

I can do the merging if Serhiy is busy with other PRs. Thanks! (and sorry for taking too long to merge this.)

@serhiy-storchaka

This comment has been minimized.

Copy link
Member

serhiy-storchaka commented Jul 3, 2017

@serhiy-storchaka are you waiting for another change?

No, I just waited for other reviewers in the case they have other comments. I left this on Berker.

@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 4, 2017

@berkerpeksag, ok, I will squash nirs/cpython@45931f5 into the test patch, and update the news item as you suggest.

nirs added 2 commits May 19, 2017
This enable testing custom readline configuration using the INPUTRC
environment variable, or passing arguments to the child process in a
clean way.
readline segfaults on input() if the number of items in the history file
is equal or more to history size * 2.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 6, 2017

@berkerpeksag are you sure blurb is ready? see python/core-workflow#153

If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but python should not segfault even if readline returns
unexpected value.
@nirs nirs force-pushed the nirs:readline-segfault branch from d2f511d to 9fcda61 Jul 6, 2017
@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 6, 2017

Changes in version 3:

  • Integrate nirs/cpython@45931f5 into the tests patch (berkerpeksag)
  • move the NEWS entry into Misc/NEWS.d/next/Library (berkerpeksag)

@berkerpeksag, @serhiy-storchaka please review.

@berkerpeksag berkerpeksag merged commit fae8f4a into python:master Jul 7, 2017
3 checks passed
3 checks passed
bedevere/issue-number Issue number 29854 found.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@berkerpeksag

This comment has been minimized.

Copy link
Member

berkerpeksag commented Jul 7, 2017

Thanks!

@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 7, 2017

@berkerpeksag, @serhiy-storchaka, @vadmium thanks for reviewing!

Do you want me to backport this to older versions?

@berkerpeksag

This comment has been minimized.

Copy link
Member

berkerpeksag commented Jul 7, 2017

@nirs it would be better to fix the test failures reported at https://bugs.python.org/issue29854#msg297876 and https://bugs.python.org/issue29854#msg297877 before doing the backports. Do you have some time to take a look at those failures?

@nirs

This comment has been minimized.

Copy link
Contributor Author

nirs commented Jul 7, 2017

@berkerpeksag sure, I'll take a look.

@nirs nirs deleted the nirs:readline-segfault branch Jul 7, 2017
nirs added a commit to nirs/cpython that referenced this pull request Jul 8, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.
berkerpeksag added a commit that referenced this pull request Jul 8, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.
nirs added a commit to nirs/cpython that referenced this pull request Jul 8, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.
nirs added a commit to nirs/cpython that referenced this pull request Jul 8, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.

This is a backport of the actual fix from master without the test, since
the test depends on new run_pty() helper which is not available in 2.7.
berkerpeksag added a commit that referenced this pull request Jul 9, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.
berkerpeksag added a commit that referenced this pull request Jul 10, 2017
If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.

This is a backport of the actual fix from master without the test, since
the test depends on new run_pty() helper which is not available in 2.7.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.