gh-99127: Port syslog module to use module state. #99128

corona10 · 2022-11-05T13:27:16Z

Issue: Port syslog module to use module state #99127

corona10 · 2022-11-05T13:27:55Z

(.oss) ➜  cpython git:(gh-99127) ✗ ./python.exe -m test test_syslog -R 3:3
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 3.65 Run tests sequentially
0:00:00 load avg: 3.65 [1/1] test_syslog
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 704 ms
Tests result: SUCCESS

ronaldoussoren

Something to consider: what happens when multiple sub interpreters use the syslog extension. The current implementation always calls openlog(3) at least once before calling syslog(3), see the check in syslog_syslog_impl.

That results in interference between sub interpreters when in one of them the python code explicitly calls syslog.openlog and the other doesn't.

IMHO the check in syslog_syslog_impl is not necessary because it results in a call to openlog(3) with parameters that match the default behaviour of the syslog library. Removing it is a fairly minor behaviour change though (syslog.syslog will currently override any calls to openlog that were done in C before the first call to syslog.syslog and will stop doing that when removing the check in syslog_syslog_impl).

ronaldoussoren · 2022-11-05T13:49:29Z

Modules/syslogmodule.c

+_syslog_clear(PyObject *module)
+{
+    _syslog_state *state = get_syslog_state(module);
+    Py_CLEAR(state->S_ident_o);


Implementations of openlog(3) might just store the "ident" argument (which is why S_ident_o exists). On such implementations the Py_CLEAR call results into use-after-free when syslog() is called from C code after this point.

Probably best to call closelog(3) with some default arguments before clearing.

corona10 · 2022-11-05T14:44:20Z

Something to consider: what happens when multiple sub interpreters use the syslog extension. The current implementation always calls openlog(3) at least once before calling syslog(3), see the check in syslog_syslog_impl.

Thanks that's why I got a headache which makes me close the issue but thanks to suggest the solution.

IMHO the check in syslog_syslog_impl is not necessary because it results in a call to openlog(3) with parameters that match the default behaviour of the syslog library.

Yeah, IIUC we should call openlog(3) anyway while calling syslog.syslog right?
Please let me know if I understood it wrongly.

corona10 · 2022-11-05T15:01:08Z

Yeah, IIUC we should call openlog(3) anyway while calling syslog.syslog right?

Or did you intended implicit call of openlog(3) while calling syslog(3)?

bedevere-bot · 2022-11-06T03:27:47Z

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 952ff46 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

corona10 · 2022-11-06T14:22:22Z

Failure tests from buildbot/PPC64LE Fedora were not related to this change.

Modules/syslogmodule.c

kumaraditya303 · 2022-11-06T14:36:07Z

Modules/syslogmodule.c

+static void
+_syslog_free(void *module)
+{
+    _syslog_clear((PyObject *)module);


Suggested change

_syslog_clear((PyObject *)module);

(void)_syslog_clear((PyObject *)module);

Why casting is needed at this moment?

Some compilers like clang complain if the returned value is ignored and it is best practice to add cast to ignore the returned value.

Some compilers like clang complain if the returned value is ignored

Please let me know which clang version emits the warning this case.
FYI I am using clang 14.

Please let me know which clang version emits the warning this case.

I don't remember the version and the compiler option since I use gcc on linux not clang. There are occurrences of this in stdlib like

cpython/Modules/_functoolsmodule.c

Line 189 in 61b6c40

(void)partial_clear(pto);

Yeah but some code doesn't use the casting also.

cpython/Modules/_json.c

Line 628 in 61b6c40

scanner_clear((PyScannerObject *)self);

But let's follow the recent convention, I can't find related issues about the casting.

Casting should not be necessary here and is just clutter. Clang and GCC can warn about ignored return values, but that's only when APIs are marked with a warn_unused_result attribute (which at least some libraries use to mark functions where the result is an error indicator that must be checked).

Modules/syslogmodule.c

erlend-aasland · 2022-11-06T19:09:59Z

Unfortunately I don't have time to review this right now. Maybe @serhiy-storchaka can have a look? He recently did some work on this module.

ronaldoussoren · 2022-11-06T20:51:05Z

Yeah, IIUC we should call openlog(3) anyway while calling syslog.syslog right?
Please let me know if I understood it wrongly.

At the C level calling openlog(3) before calling syslog(3) is not necessary, although the spec is not entirely clear about this. The linux manual page does clearly mention that openlog(3) is optional, and matches what I remember from older Unix systems.

The call to openlog in syslog_syslog_impl basically resets the default system state (use the process name for the ident and log using the LOG_USER facility). Leaving that out should therefore be harmless, but this is technically a user-visible change of behaviour:

C code calls openlog(...) with same parameters
Python code calls syslog.syslog

Currently the Python code resets the values set in the first step, with my proposal it would no longer do this. I'd consider this a good change, but there's bound to be someone affected by this.

vstinner

The stateless syslog C API doesn't seem to be designed to be consumed by multiple interpreters. I'm not sure that making S_ident_o and S_log_open global varibles per interpreter is safe.

vstinner · 2022-11-07T16:34:52Z

Modules/syslogmodule.c

-static char S_log_open = 0;
+typedef struct {
+    PyObject *S_ident_o;  /*  identifier, held by openlog()  */
+    char S_log_open;


What does the S_ prefix stand for? Does it mean "state"? Maybe remove it?

vstinner · 2022-11-07T16:34:52Z

Modules/syslogmodule.c

-static char S_log_open = 0;
+typedef struct {
+    PyObject *S_ident_o;  /*  identifier, held by openlog()  */
+    char S_log_open;


Can you please add a comment to explain the purpose of the S_log_open flag? I understand that it's set to 1 after openlog() is called.

vstinner · 2022-11-07T16:34:52Z

Modules/syslogmodule.c

@@ -162,8 +170,9 @@ syslog_openlog_impl(PyObject *module, PyObject *ident, long logopt,
    }

    openlog(ident_str, logopt, facility);


This API is stateless. What happens if interpreter A calls openlog("idA", options_a, LONG_USER) and interpreter B calls openlog("idB", options_b, LOG_INFO)?

I'm not sure if it's a good idea to make the global S_log_open flag per-interpreter. See for example the recent crash on macOS: #98178. Same question for S_ident_o.

If you're confident that safe, please add a comment explaining the behavior, who owns the memory, etc.

It is safe, but the behavior might be unexpected. The only issue of safety would be how POSIX syslog keeps a pointer to "ident" that must not be deallocated before closelog() is called. I don't see where that would be a problem.

That said, moving the data to per-interpreter module state may lead to a slight change in behavior. Again, though, using syslog with multiple interpreters is already wierd.

vstinner · 2022-11-07T16:34:52Z

Modules/syslogmodule.c

+{
+    _syslog_state *state = get_syslog_state(module);
+    if (state->S_log_open) {
+        closelog();


What happens if interpreter A calls closelog() and interpreter B (same process!) still uses the Python syslog module, so after closelog() has been called?

POSIX syslog will transparently call openlog() with default values.

bedevere-bot · 2022-11-07T16:34:56Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

And if you don't make the requested changes, you will be poked with soft cushions!

ericsnowcurrently · 2022-11-07T19:39:12Z

I've posted to the issue about interpreters interfering with each other (which is not new with this PR): #99127 (comment).

vstinner · 2022-11-07T21:18:27Z

For the specific case of sub-interpreters, maybe sub-interpreters must not be allowed to call openlog() and closelog()?

But for the simple case of a single intereterpreter but loading the C extensions twice, there is a similar issue.

This problem reminds me the complicated case of the posix/nt extension being converted to multi-phase init. I got a very complicated issue about Python exit. Some applications started to crash. Oops. Python os.environ[key] = value stored a copy of "key=value" string... until Python exit. But if the application still runs after Python shutdown, reading the environment could lead to reading freed memory which leaded to a crash! See:

The fix? Let the C library handles memory for us: use setenv() function, rather than putenv()!

For me, the Python syslog module should be a thin wrapper to a glue between Python and the underlying C library: have something which handles openlog() / closelog(), keep memory around, and only frees this memory when the last interpreter is deleted. Maybe it can be a shared state with a reference counter?

syslog instance 1 calls openlog(): copy arguments in heap memory
syslog instance 2 calls openlog(): free old memory, copy new arguments
...
last syslog instance exits: call closelog(), release memory

For me, it sounds weird and dangerous to still use closelog() in another syslog instance which remains alive. Maybe it works on some operating systems. But again, see the recent crash on macOS.

vstinner · 2022-11-07T21:20:30Z

For example, the Python socket.socket module has an _io_refs reference counter: it's used to decide when sock.close() will really close the underlying "raw" socket (C _socket.socket module).

ericsnowcurrently · 2022-11-08T00:20:59Z

Yeah, it is usually best to restrict management of global resources to the main interpreter. I'd say the syslog API counts as a global resource, so I'd be good with syslog.openlog() and syslog.closelog() only working in the main interpreter. Maybe also disallow syslog.syslog() in a subinterpreter if openlog() hasn't been called yet (or maybe not).

ericsnowcurrently · 2022-11-08T00:24:43Z

FYI, it looks like logging.handlers.SysLogHandler doesn't use the syslog module, so that shouldn't factor in.

corona10 requested a review from ericsnowcurrently Nov 5, 2022

bedevere-bot mentioned this pull request Nov 5, 2022

Port syslog module to use module state #99127

Open

bedevere-bot added the awaiting core review label Nov 5, 2022

corona10 force-pushed the gh-99127 branch from 910eaea to 646c40f Compare Nov 5, 2022

corona10 requested a review from erlend-aasland Nov 5, 2022

pythongh-99127: Port syslog module to use module state.

6ebf768

corona10 force-pushed the gh-99127 branch from d23f11a to 6ebf768 Compare Nov 5, 2022

corona10 closed this Nov 5, 2022

corona10 deleted the gh-99127 branch Nov 5, 2022

ronaldoussoren reviewed Nov 5, 2022

View changes

corona10 restored the gh-99127 branch Nov 5, 2022

corona10 reopened this Nov 5, 2022

corona10 added 2 commits Nov 5, 2022

Address code review

d6955d7

Remove logic to check the openlog status while calling syslog.syslog

c916872

corona10 requested a review from ronaldoussoren Nov 5, 2022

Do not call openlog while calling syslog

6a76772

corona10 force-pushed the gh-99127 branch from 9333a09 to 6a76772 Compare Nov 5, 2022

Fix test_audit

29e1076

corona10 requested a review from vstinner Nov 6, 2022

nit

952ff46

corona10 added the 🔨 test-with-buildbots Test the PR with the buildbot fleet and report in the status section label Nov 6, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test the PR with the buildbot fleet and report in the status section label Nov 6, 2022

kumaraditya303 reviewed Nov 6, 2022

View changes

Modules/syslogmodule.c Outdated Show resolved Hide resolved

kumaraditya303 reviewed Nov 6, 2022

View changes

Modules/syslogmodule.c Outdated Show resolved Hide resolved

corona10 added 2 commits Nov 6, 2022

Address code review

e9dc30c

nit

3f02813

erlend-aasland removed their request for review Nov 6, 2022

vstinner requested changes Nov 7, 2022

View changes

bedevere-bot added awaiting changes and removed awaiting core review labels Nov 7, 2022

gh-99127: Port syslog module to use module state. #99128

gh-99127: Port syslog module to use module state. #99128

corona10 commented Nov 5, 2022 •

edited by bedevere-bot

corona10 commented Nov 5, 2022

ronaldoussoren left a comment

ronaldoussoren Nov 5, 2022

corona10 commented Nov 5, 2022 •

edited

corona10 commented Nov 5, 2022 •

edited

bedevere-bot commented Nov 6, 2022

corona10 commented Nov 6, 2022

kumaraditya303 Nov 6, 2022

corona10 Nov 6, 2022

kumaraditya303 Nov 6, 2022

corona10 Nov 6, 2022 •

edited

kumaraditya303 Nov 6, 2022

corona10 Nov 6, 2022

ronaldoussoren Nov 6, 2022

erlend-aasland commented Nov 6, 2022

ronaldoussoren commented Nov 6, 2022

vstinner left a comment

vstinner Nov 7, 2022

vstinner Nov 7, 2022

vstinner Nov 7, 2022

ericsnowcurrently Nov 7, 2022

vstinner Nov 7, 2022

ericsnowcurrently Nov 7, 2022

bedevere-bot commented Nov 7, 2022

ericsnowcurrently commented Nov 7, 2022

vstinner commented Nov 7, 2022

vstinner commented Nov 7, 2022

ericsnowcurrently commented Nov 8, 2022

ericsnowcurrently commented Nov 8, 2022

	_syslog_clear((PyObject *)module);
	(void)_syslog_clear((PyObject *)module);

		@@ -162,8 +170,9 @@ syslog_openlog_impl(PyObject module, PyObject ident, long logopt,
		}

		openlog(ident_str, logopt, facility);

gh-99127: Port syslog module to use module state. #99128

Are you sure you want to change the base?

gh-99127: Port syslog module to use module state. #99128

Conversation

corona10 commented Nov 5, 2022 • edited by bedevere-bot

corona10 commented Nov 5, 2022

ronaldoussoren left a comment

Choose a reason for hiding this comment

corona10 commented Nov 5, 2022 • edited

corona10 commented Nov 5, 2022 • edited

bedevere-bot commented Nov 6, 2022

corona10 commented Nov 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corona10 Nov 6, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erlend-aasland commented Nov 6, 2022

ronaldoussoren commented Nov 6, 2022

vstinner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Nov 7, 2022

ericsnowcurrently commented Nov 7, 2022

vstinner commented Nov 7, 2022

vstinner commented Nov 7, 2022

ericsnowcurrently commented Nov 8, 2022

ericsnowcurrently commented Nov 8, 2022

corona10 commented Nov 5, 2022 •

edited by bedevere-bot

corona10 commented Nov 5, 2022 •

edited

corona10 commented Nov 5, 2022 •

edited

corona10 Nov 6, 2022 •

edited