FUSE and io_uring

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Bernd Schubert led a session at the 2023 Linux Storage, Filesystem, Memory-Management and BPF Summit on the intersection of FUSE and io_uring. He works for DDN Storage, which is using FUSE for two network-storage products; he has found FUSE to be a bottleneck for those filesystems. That could perhaps be improved by using io_uring, which is something he has been working on and wanted to discuss.

He noted that Boaz Harrosh had developed the zero-copy user-mode filesystem (ZUFS) in 2018, but it did not go upstream in part due to concerns about its overlap with FUSE. Meanwhile, Miklos Szeredi started working on FUSE2, but that has languished as well. Schubert briefly looked at both, but the FUSE2 Git tree was hard to review since it is a single big patch rather than being broken into reviewable pieces.

Last year, he was working on atomic open operations and noticed some problems; Szeredi asked for some benchmarks, which turned out to be confusing. Multiple threads were reading from /dev/fuse, which caused the confusing results; switching to a single thread made the results consistent. He also realized that performing a polling loop before adding an operation to the wait queue greatly improved the filesystem performance.

He was also looking at an NVMe driver that his company uses, wondering why it was able to avoid the bottlenecks that he was seeing; it used io_uring, but in the "wrong" direction, from user space to the kernel, while FUSE needed to go the other way. Around that time, the IORING_OP_URING_CMD support for io_uring was added, which is being used in the ublk user-space block driver in the right direction, from the kernel to user space. That provides a model for doing something similar in FUSE, which is what Schubert has been working on.

He explained the inner workings of the work he has done to make FUSE use io_uring. There is one thread per core, each with its own ring buffer; there is a shared memory buffer with the FUSE queue ID used as the offset for user space to mmap() its region. Libfuse initiates operations with an IORING_OP_URING_CMD, which is the core idea taken from ublk. For debugging purposes, there is a mode with a single thread and ring buffer.

Amir Goldstein asked whether user space really needed to be aware of the underlying implementation. Schubert replied that his goal was to make it transparent, so that existing filesystem implementations could gain the performance benefits without having to change their code.

He is unhappy with using the queue ID to identify the user-space buffer (via the mmap() offset), but was unable to find the corresponding buffer in the kernel without it. There was some discussion of ways to get the kernel and user-space in sync on the buffer location directly. Jan Kara suggested looking in the VMA associated with the user-space virtual address to find where in kernel memory the buffer was located; he said that he could help Schubert find the right calls to make for that.

Schubert showed some performance benchmarks, but noted that he needs to find a way to keep the scheduler from migrating the application processes to other CPUs. For example, direct I/O reads showed moderate improvements for io_uring over regular FUSE, but much larger improvements when migration was disabled. Kara cautioned that CPU migration can be a problem at the start of a test like this, but may not actually be problematic over the long term; meanwhile, there are other workloads that may benefit from the migration.

But Schubert said that FUSE is particularly affected by this because most of the I/O work is being handed off to another process; once it completes, it is best if the application process is still running on that same CPU. Kara said that it is not a simple problem and that the scheduler may lack the information it needs to make the right decision. The scheduler developers are aware that there may be problems in scheduling for io_uring and are working on some solutions; he recommended that Schubert work with those developers.