What's brewing in CXL

LWN.net needs you!

Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing.

CXL, he began, is a way of providing memory over the PCIe bus; it generally offers latency that is worse than accessing memory on remote NUMA nodes. The strength and challenge of CXL is that it is highly configurable, allowing, for example, the setup of interleaved memory access that improves performance. One problem is that, while the kernel can control some access to CXL memory, the system firmware wants to play with things as well. Another challenge comes with the hot-plug nature of CXL memory; if it goes away, a portion of the system's RAM can disappear.

The CXL standard is evolving quickly, and manufacturers are putting out hardware with interesting deviations from the adopted standards. The kernel is following something like the ACPI "code-first" policy with regard to these changes and documenting them in the kernel tree. The hope is to send a message to manufacturers: "somebody broke it this way, please break yours the same way".

Error handling is an evolving area as well. CXL protocol errors are reported to the kernel as PCIe internal errors, but they are handled through a side channel so that the PCIe core code does not need to deal with them. The CXL code is introducing kernel panics in its error-handling paths; that is not something that is normally appreciated in kernel code but, Williams said, the firmware is going to panic the system anyway in such situations.

Accelerator support (memory-to-memory compression, for example) is close to landing in the kernel. It turns out that accelerators are relatively simple to support. Another development area is vfio-cxl, which is a mechanism to allow exporting CXL accelerators to virtual machines.

Dynamic capacity has long been a dream for system designers, Williams said. One buys a lot of DIMMs, puts them into a box, runs a cable, and a lot of hosts can then map that memory. But then the kernel has to somehow make that memory available to user space. The plan is to use device DAX as the interface, but there are questions about how to create private nodes for dedicated memory (a topic that would return the following day). There needs to be integration with guest_memfd as well.

What is not brewing in CXL support? One non-development area is error isolation; if a CXL host bridge is lost, all associated devices will fail, and terabytes of system RAM may vanish. It is hard to envision a way in which the system can survive such an event, at least when memory is involved. Error isolation for accelerator users might be more feasible. There is also no work currently on supporting peer-to-peer operations on CXL devices, but "somebody will want it someday". Finally, CXL encryption, which he described as "another bucket of acronyms", is also not expected to be supported in the near future.

The session concluded there, with no real discussion among those present.