Various short development-process topics

Ready to give LWN a try?

With a subscription to LWN, you can stay current with what is happening in the Linux and free-software community and take advantage of subscriber-only site features. We are pleased to offer you a free trial subscription, no credit card required, so that you can see for yourself. Please, join us!

Stable updates

Kees Cook wanted to talk briefly about the testing of stable updates. There was a set of significant random-number-generator patches applied early this year that was subsequently backported to the stable updates. That was a bit controversial at the time, since such changes go beyond the normal rules for stable updates, but it was agreed in this session that it has made backporting other changes easier since then. At the time, though, these patches brought some significant regressions to the stable kernels.

In retrospect, Cook said, it was clear that these patches were not adequately tested in the stable updates. How did that happen, he asked, and how can such a situation be prevented in the future?

Greg Kroah-Hartman agreed that he should have staged those changes for longer, but also said that they had passed all of the continuous-integration (CI) tests that the stable maintainers had available to them. The random-number-generator maintainer was deeply involved in the whole process as well. Guenter Roeck said that the community's testing is far from complete at this point, and that it can take quite a while for regressions to come to light. But, he said, the situation is improving. Five years ago there was almost no CI testing at all; now things are better, and in five years they will be better yet.

Torvalds said that, even now, half of the supported architectures get no CI testing at all; Thomas Gleixner added that even 32-bit x86 systems are poorly tested at this point. Cook concluded that the systems currently in use are not adequately represented in the testing infrastructure. The first step to make things better would appear to be to determine where the worst gaps are.

Compiler versions

Christoph Hellwig wanted to talk about a 6.0 merge-window pull request that evoked a strong response (and a rejection) from Torvalds. The problem was a compiler warning that Torvalds saw but the others did not; that resulted from Torvalds running a more recent version of the compiler than anybody else. Only the newer compiler emitted that particular warning, so none of the other developers were aware of it. Perhaps, Hellwig suggested, the kernel.org front page could be augmented to show which version of the compiler Torvalds is currently using.

Torvalds answered that this episode was a bit of a mistake; he had thought that all of the warnings raised by recent GCC releases had been taken care of. In general, he said, he is not running bleeding-edge compilers. If developers want to know which version he is running, they should just assume that he has whatever "a fairly recent Fedora" is shipping. Cook suggested that perhaps Torvalds should avoid doing Fedora updates just before the merge window opens.

External test suites

Paolo Bonzini asked about running external test suites with the kernel. He has some of those that are not specific to Linux or KVM; as a result, a number of the tests are not really applicable, and they can fail on older versions of Linux. He asked whether others in the group have had similar experiences.

Mark Brown suggested maintaining a list of expected failures. Ted Ts'o said that people working on tests do not normally want to annotate which ones will fail to work on specific kernel versions. Hellwig said that there are two cases to think about. The first of those is when the functionality being tested simply does not exist on the target system; the tests should be checking for that and responding accordingly. Otherwise, he said, a test failure indicates a real problem somewhere.

Ts'o asked about cases where a known failure exists, but it has been decided that backporting the fix is not worth the effort or the risk. Gleixner suggested just documenting the situation when that happens. Alexei Starovoitov described another situation: the addition of the CAP_BPF capability broke programs using the libcap library, which aborted when it saw a capability it didn't know about. There weren't a lot of ideas for how to avoid that kind of problem.

Pre-merge CI

Dave Airlie informed the group of a change coming to the graphics subsystem aimed at getting submitters to run CI tests prior to their code being merged into the subsystem tree. Submitters are expected to do that now, but the system that implements this process is "hacky"; it is being replaced with a GitLab instance. The plan is for submitters to create a merge request in GitLab, which will cause the CI tests to be run on their proposed changes. For now, the actual merge is still happening manually.

Linus's happiness level

Torvalds started off the traditional closing session by saying that the development process is, from his point of view at least, working quite well. His biggest complaint is having to wait for fixes; he'll be told that a patch is pending, but he'll still be waiting for it a couple of -rc releases later. Alluding to the previous BPF session, he acknowledged that the networking pulls tend to be huge, but he is happy that those pull requests are now coming from a few different maintainers. Networking used to be an area he worried about; "now I just worry about Greg [Kroah-Hartman]".

He has asked some maintainers to not send large pull requests on the weekends to make his life easier; he would rather have that code in the mainline for a few days ahead of the -rc release in case any problems turn up. So now the networking requests come on Thursdays, which gives more time to shake out problems and makes the whole process work better.

Torvalds let it be known that he would much rather receive pull requests early in the merge window rather than later. That helps him to get ahead of the game and clear time in the second week of the merge window to closely examine requests that he is concerned about. His life during the merge window, he said, is "one week of intense merging" of the subsystems he trusts, and "one week of careful merging" for those he worries about or has had problems with. Anybody who wonders which subsystems fall into the latter group can find out by looking which ones do not get pulled until the second week.

After the merge window closes, he said, he gets six weeks or so of relatively low-stress life. Overall he's entirely happy to keep doing this work. Airlie asked how much information in the typical pull request is actually useful to Torvalds. The answer was "all of it". Torvalds will sometimes edit out information that appears to be intended just for him, but he encouraged maintainers to keep providing anything that might be useful. The pull requests that are generated by scripts and look like "git shortlog" output are not his favorite, though; he would rather get a report from the maintainer on what he is being asked to pull.

James Bottomley asked Torvalds to name the maintainer who sends the best pull requests; the answer was Christian Brauner, whose requests (recent example) read "like a short novella". Each request explains why the change is being made, then gets into the details of how the change was done. Steven Rostedt suggested that it would be good to have a document on how to write good pull requests.

Brauner asked about discussions that end up stalemated; he mentioned the folio discussions in particular. When does Torvalds decide to step into a stalled debate to bring things to a close? Torvalds replied that he is generally happy to let a discussion go on for a while, often because he doesn't know enough to make a judgment himself. But in the end resolving such disagreements is part of his job. He suggested that developers can always send him email when there is a problem.

As a closing note, Torvalds said that, "because of past behavior", he has a filter on his outgoing email that prevents him from sending messages with certain words in them. This doesn't create problems; he has mostly trained himself to speak more gently. But it triggers when he replies to messages from other developers where they have used one of the forbidden words. So, to make his life a little easier, he asked the community as a whole: "please don't call each other morons on the mailing lists".