A kernel panel convenes in Edinburgh

Ignore previous instructions; subscribe to LWN today

Every article on LWN.net is written by humans, for humans. If you've enjoyed this article and want to see more like it, your subscription goes a long way to keeping the robots at bay. We are offering a free one-month trial subscription (no credit card required) to get you started.

One of the most popular sessions at any LinuxCon is the kernel panel, Jim Zemlin, executive director of the Linux Foundation, said in his introduction. Another lively panel made its appearance on October 22 in Edinburgh, Scotland as part of LinuxCon Europe. The panel was ostensibly focused on "core and embedded", but predictably ranged further than that.

The panel was moderated by LWN executive editor Jonathan Corbet and had four panelists who are developers from all over the kernel. Each panelist introduced himself, with Greg Kroah-Hartman starting things off. He, of course, works for the Linux Foundation on a number of different kernel areas including USB and the driver core as well as both the stable and staging trees, which he called "the two extremes in the kernel". Will Deacon works for ARM on very low-level architecture-specific code like memory barriers and TLB handling—"stuff that most people hate". He also co-maintains the 64-bit ARM (aarch64) support in the kernel.

Sebastian Hesselbarth was invited as a "hobbyist" attendee to the Kernel Summit. He started working on the mainline kernel about a year ago. Peter Zijlstra from Red Hat rounded things out. He rattled off a number of things that he co-maintains with Ingo Molnar including the scheduler, perf, and lockdep. He also works on the memory management subsystem, IRQs, timers, and various low-level architectures "when I have to". "I get around a bit", he said.

Corbet noted that there is a huge amount of code coming in now from the mobile and embedded world mostly in the ARM tree, but that when you look at just the contributions to the core kernel (scheduler, kernel and mm directories), it is companies like Red Hat and IBM. He asked "are we able to work together and make one kernel that works for everybody?" Zijlstra said that it was a "fun exercise" to make one kernel that scaled from tiny little machines to the largest ones out there.

Power

That led Corbet to direct a question to Zijlstra about some frustrations he had heard from the ARM community about getting scheduler patches reviewed. Zijlstra said that he wants to avoid having a different scheduler for every architecture. Power-awareness is important to ARM, but it will be (or already is) important for others as well. There are several subsystems that currently work together to handle power management (CPU idle, CPU frequency, and the scheduler), but he wants to see a "coherent framework" proposed that will help solve all the problems and not just "random tweaks here and there". There is a mini-summit for power-aware scheduling later in the week, he said, where he hoped that some of the issues could be resolved.

But, as Deacon pointed out, the problem is not just for ARM and mobile, the server community is interested in better power performance as well. Zijlstra agreed but noted that it is "difficult to get a straight answer" from that community about its needs. The hardware vendors really don't want to talk to each other, he said. Deacon pointed out that it might just be that they aren't used to talking to each other and that they see it as much easier to get things done when they don't have to. Kroah-Hartman said that the reason hardware vendors start out with drivers is because they are self-contained and don't require coordination with other vendors. It is harder to change the core kernel, he said, as it should be, but companies eventually get there. Neither Intel nor IBM worked on the core a while back, but now that's a large part of what they do. Moving into the core is simply a migration that many companies eventually make.

The ARM big.LITTLE architecture is different than anything we have seen before, moving us from symmetric multiprocessing (SMP) to asymmetric, heterogeneous multiprocessing, Corbet said. He asked how that would impact the scheduler. Zijlstra was upbeat about supporting that model, saying that the scheduler already deals with SMP systems that have asymmetric loads due to realtime or interrupt processing. Big.LITTLE is different than that, certainly, but there are enough similarities that some of the code which tries to balance "fairness" by tracking the realtime and interrupt usage on the CPUs, which could map to more and less powerful processors

Kroah-Hartman isn't sure that the idea behind big.LITTLE will really pan out. There are other system components (memory, buses, etc.) that take more power and still need to stay powered up even when only a "little" CPU is running, so it's not clear that just turning off the more power-expensive processors is enough of a win. It may be an "experiment that is failing", he said. Deacon had not heard that, he said. Companies are putting big.LITTLE into products, so they believe it will bring power savings. We should see proof one way or another before too long, he said.

Large, small, and tiny

There have long been complaints that the kernel is enterprise-focused, though they have died out some over the years; does that make it hard for the embedded side, Corbet asked. Deacon said that there is a history of that in the kernel, and that many of the maintainers came about during that era, but that he, at least, didn't see a real problem with that. As the kernel evolves, so do the maintainers and the focus; it all balances out. Kroah-Hartman was quick to note that the enterprise systems of five or ten years ago were now in many people's pockets, so that those differences flatten out over time. In addition, the choices rarely come down to making one better completely at the expense of the other, Deacon said, so people air their concerns and a compromise is found.

Going the other direction, Corbet noted that Tim Bird has been concerned that Linux is leaving tiny systems behind. Most embedded devices these days are fairly capable systems with fewer memory and other constraints from years past, but ignoring the really small systems may leave Linux out of the "internet of things". He asked if the panelists were concerned about that.

Zijlstra didn't think it was a real problem as there are patches to shrink the kernel that are still accepted. In addition, a recent suggestion to remove uniprocessor (UP) support—essentially turning it into an SMP system with one processor, though with all of the extra SMP data structure overhead—was rejected as there are still plenty of single-core processors out there and Intel recently introduced more with the Quark UP line. Deacon noted that support for a tiny ARM core ("M class") was recently merged as well. As long as people are still running tiny systems and are willing to put in the work, support for the super low-end should be fairly easy to maintain, but once the use of UP systems goes away, for example, "that code will bitrot really fast", Zijlstra said. "I can't wait for that day, but I'm afraid I will have to wait a long time."

Community

Somewhat controversially, this year's Kernel Summit committee set aside summit slots for hobbyists, Corbet said, so he wanted to explore that a bit. He asked the audience how many had contributed a patch to the kernel and somewhere around one-third raised their hands; perhaps half of those indicated they had done it on their own, not as part of their job. On stage, Hesselbarth served as a representative from the hobbyist community; Corbet started by asking him why he contributed to the kernel on his own when so many around him were being paid to do so.

Hesselbarth said that he had a personal interest in tiny devices, but that he probably was not a prototypical Linux hobbyist. He is a hardware engineer with an interest in building systems-on-chip (SoCs) for devices. He writes Linux drivers as part of his job, which led him to work on the ARM SoC code on the side.

Corbet asked: "Is our community friendly?" He wondered if Hesselbarth was able to get the answers he needed on mailing lists, for example. Hesselbarth said that it depends on how you approach the mailing lists. If you start out "kindly", most on the list will accept you, will look at your code, and will correct you if you are wrong, even if they have done that many times for others in the past. Zijlstra had some advice for anyone looking for assistance: ask specific questions, and always on the mailing list. Specific questions are easier to answer and emailing him privately puts a message on "a very short path to /dev/null".

Corbet brought up the steady decline in the percentage of hobbyist developer patches over the years. Zijlstra and Deacon both thought that perhaps the absolute numbers weren't really declining that much, it was that the number of paid contributors has risen so much. Kroah-Hartman said that people who show any ability to get patches into the kernel immediately get job offers. He knows of at least five people who were doing style patches to staging drivers in their spare time, got offers, and now do kernel development full time. It is quite difficult to find kernel developers, so companies watch the lists and ask other kernel developers for leads on new employees, he said.

When everyone gets hired, though, that causes a problem as sometimes they can no longer maintain the code they were working on. Or, as Zijlstra pointed out, they get reassigned to a different project and maintenance falters. Corbet mentioned that he sat in on a Matthew Garrett talk earlier in the conference where he talked about some buggy code he had merged before getting reassigned and just kind of left it all behind.

Relatedly, Kroah-Hartman recently backed out a change when the email to the developer bounced. It is a problem, Kroah-Hartman said, because a maintainer needs to trust that someone contributing will be around to fix the code, "or I have to". He noted a "big, hairy" network change that went in a few years ago, where, literally, email to the developer started bouncing the day after it was merged. It took many months to unwind those changes, which is part of what makes it difficult to get large changes into the network subsystem today.

Device tree woes

Non-discoverable hardware and the ARM device tree solution for that was next up. Once upon a time, the devices attached to a system could be determined at run time, but that is often not the case anymore, so some kind of external description of those devices is required. For ARM, device trees are being used to do that, but there are some problems with the consistency in how those device trees are specified (i.e. the bindings). A bigger issue is whether device tree bindings constitute an unbreakable kernel ABI, which means that even more care is required before merging any device tree support. Corbet asked: "Is device tree the right answer, what have we done wrong, and how can we fix that?"

Deacon said that a full day of the ARM mini-summit (concurrent with the conference) is devoted to trying to work out those problems. Device tree has been an improvement over the older mechanisms, and it has allowed a lot more code and drivers to go into the kernel. The ABI question is a "religious" one and there is a fundamental disagreement between those who think it is an unstable interface that shouldn't be used in any products and those who think it is an unbreakable ABI that is fixed in stone. He was hopeful that some of that could be ironed out in the mini-summit.

Hesselbarth found that device tree makes things harder for hobbyists because they have to consider all of the possible ways that someone might hook up a particular IP block when designing the device tree for it. There is something of a trend to cut the static configuration data right out of the drivers and essentially paste it into a device tree, Deacon said, which is not the right way to approach creating a device tree entry. In addition, Kroah-Hartman said that it is hard for driver maintainers to decide whether to merge a driver because they are unsure if the device tree support is "correct"—or even what "correct" looks like.

Kernel ARM maintainer Russell King recently put out a detailed critique of device tree that Corbet said he hadn't had a chance to digest yet, but clearly concerned the ability of device tree to describe the complexities of today's devices. They are no longer just a single device, often, but a collection of devices connected by buses. "Is there a fundamental flaw there?", he asked.

Kroah-Hartman likened the problem to that which the Video4Linux (V4L) developers have been grappling with for years. There needs to be a way to describe devices and how they interconnect, which is a complex problem, but it has to be described somewhere. "What's wrong with making it all discoverable?", Zijlstra asked. That's what kernel developers want, but it's difficult to convince the hardware makers, Kroah-Hartman said. Zijlstra's suggestion to "take them out back" and make them "do what we want" was greeted with laughter, but lots of nodding heads as well.

Corbet noted that there was a trend toward getting the knowledge from user space, both in the V4L world with the media controller interface and in the ION memory allocator; are we getting to the point where we just can't solve those problems in the kernel, he asked. Deacon said that he doesn't think it is unsolvable, but it isn't fun to think about how to solve and it is easier to think it is someone else's problem. He said that not only is hardware configuration being pushed into user space, it is also being pushed into the firmware.

The hardware today is "plug and play" modules, Kroah-Hartman said, where hardware makers buy IP blocks from multiple vendors and hook them all up in different ways. Linux has drivers for each of the pieces, but not all the different ways they might be hooked up and communicate with each other. The hardware folks have solved the problem, but it needs to be handled on the kernel side. He returned to the discoverable hardware idea, noting that simple, static tables that could be read by the kernel would help solve the problem. Zijlstra suggested that perhaps the tools used by the hardware designers could be changed to make it easier to provide the kernel what it needed. Hesselbarth seemed a bit skeptical that Synopsys and other tool vendors would be all that interested in helping out.

Security

The conversation turned to security and whether the kernel developers were doing enough to deliver a secure kernel. Kroah-Hartman said that the community can "always do better", but that the bugs reported to the kernel security email address are fixed "as soon as we possibly can". He noted that various static code analysis tools are being run on the kernel and are finding lots of bugs that get fixed right away. Some of the creators of those tools, Julia Lawall (Coccinelle) and Dan Carpenter (Smatch) for example, have fixed more security bugs than anyone, he said. It is an area that is ripe for more research and the community is always open to better ways.

Some 4000 patches were made to the 3.0 kernel in the two years it has been maintained as a stable kernel (it reached end of life with 3.0.101, which was released on the day of the session). Corbet asked if that was reasonable. Deacon said that you have to trust that kernel developers are not adding bugs on purpose, so when they are found, they need to be fixed; "what's the alternative?", he asked. Kroah-Hartman noted that many of those patches were for things like new device IDs, but that the kernel developers are learning over time. Mistakes were made—and fixed.

Corbet asked if we could expect fewer fixes for 3.10 (which will also be supported for two years), but Kroah-Hartman said that there would likely be more. He now maintains the stable kernel as part of his job, so he has more time to find patches that need to go into the tree. Beyond that, the code base has grown. But the number of fixes going into the kernel starts to tail off significantly after a year, he said. For the stable updates released on that day, 3.0 had eight patches, while 3.10 had nearly 100. The world changes; processors speed up (which leads to new timing issues), for example. "If we stop our rate of change, then we are dead", he said, we have to keep up with the changes going on in the rest of the world.

Referring to Deacon's assertion that bugs are not being introduced on purpose, Corbet asked the panel how it is we know that. The whole maintenance system relies on trust, Deacon said, if that's missing, the whole thing breaks down. Kroah-Hartman said that if you look at the known exploits, they are all attacking some "stupid" mistake that someone (including him) has made, not some kind of introduced backdoor. People who research flaws per line of code find Linux to have lower rates than anything else, he said, so we are doing something right. Corbet pointed out that the kernel developers introduce enough bugs on their own, so there is no real need for anyone else to do so—to some chuckles across the stage.

2038

While OpenBSD has fixed its "2038 problem" (when 32-bit timestamps will wrap), Linux still has not, Corbet said, and wondered if Linux would be ready for that event. He also asked: Will there be no 32-bit processors to be affected in 2038? Deacon noted that billions of 32-bit ARM processors are shipping today, so it is hard to believe they will all be gone in 25 years. Kroah-Hartman said that Intel came out with a 486-based processor recently as well. He suggested that they could all come out of retirement to fix the problems.

Corbet said that OpenBSD broke its ABI to handle the change—something it can do because it ships its user space with the kernel—but that is not something that Linux can do. Something clever will be required to fix the problem, which suggests we should be thinking about it now. Deacon indicated that he thought an ABI break will eventually have to happen at some point. The real problem will be for devices that are being deployed now that will still be running in 2038.

That led to the last question, which is how to handle things like control groups that were added to the kernel, but were "wrong" both internally and externally. The internal problem can be fixed relatively easily, but how can we continue on without carrying tons of baggage from early mistakes at the ABI level, he asked. Zijlstra said that even "simple things" need to be written at least three times before you get them "sort of right". Other operating systems have a way to deprecate things, and some of those have been tried in Linux without success, Kroah-Hartman said. The Linux method is to write something new and wait for the old one to die, then try to sneak the code out of the kernel and wait to see if anyone screams, Zijlstra said.

On the other hand, Deacon said, if you wait for a perfect solution, you'll never get anywhere. Kroah-Hartman said that there is no model for Linux, everything it is doing is new; it is at the forefront and we are learning how to deal with these kinds of problems. "We do it all in public", unlike companies that struggle with the same things, he said. There are lots of hard, and fun, problems to be solved going forward, he said. It is a high-quality problem to have, Corbet said to general agreement.

With that, the time was up and Zemlin retook the stage. Mostly, he wanted to clear up some misinformation that he heard during the session: Kroah-Hartman would not be retired by 2038, Zemlin said—to laughter from the assembled attendees.

[Thanks to the Linux Foundation for travel assistance to Edinburgh for LinuxCon Europe.]