macOS: libfuse2 mounting breaks with macFUSE 5.3 API changes

[oenderg]

Problem

fuser’s libfuse2 backend mounts with fuse_mount_compat25(). Starting with macFUSE 5.3.0, that function returns -1 unconditionally on Apple and Darwin fuse_kern_mount is no longer available. Consequently, the unchanged fuser hello example cannot mount.

This is separate from the asynchronous mount-readiness problem tracked in #736.

Version split

• Through macFUSE 5.2, fuse_mount_compat25() returns the /dev/macfuse descriptor through the older Darwin mount implementation.
• In 5.3.0–5.3.2, fuse_mount_compat25() is disabled and public fuse_chan_fd() returns the channel’s fd field, which is -1 for the new MFChannel-backed Darwin channel. These releases therefore provide no usable public libfuse2 fd path for this design.
• In 5.3.3, fuse_chan_fd() obtains the device descriptor from MFChannel, making public fuse_mount() + retained channel + duplicated fuse_chan_fd() viable.

An isolated test on macOS 15.7.8 arm64 with the macFUSE 5.3.3 kext backend confirmed that this public-channel sequence mounts and reads the upstream hello filesystem.

The descriptor returned by fuse_chan_fd() is borrowed. fuser must duplicate it for DevFuse while retaining the owning channel for the mount lifetime.

Teardown ownership question

When macFUSE 5.3.3 fuse_unmount() starts asynchronous teardown for a CONNECTING or mounted Darwin channel, source-level reference accounting appears to balance the callback and unmount-thread retains but not the caller’s original fuse_mount() reference.

Does fuse_unmount() consume or release that original channel reference in this path? If not, relinquishing the pointer after fuse_unmount() can leak the channel object, while unconditionally calling fuse_chan_destroy() afterwards is unsafe for paths where fuse_unmount() destroys it immediately.

Historical compatibility note

macFUSE channel lifetime is not uniform across older releases. In particular, macFUSE 4.0–4.8.3 stores an unretained channel pointer in its asynchronous mount callback context, so destroying the channel before that callback completes can cause a use-after-free. macFUSE 4.9 added callback locking and channel retain/release handling.

This means replacing the old mount path for every historical macFUSE version may be unsafe without version-specific analysis. Solving historical compatibility is not required to address the macFUSE 5.3 break.

Proposed direction

For macFUSE 5.3.3, use the public channel API, retain the channel, duplicate the borrowed fd for DevFuse, and give setup, error, and unmount paths explicit single ownership.

The 5.3.0–5.3.2 transition releases may need to be documented as unsupported unless macFUSE exposes a suitable public path.

Mount readiness remains in #736. Process-death cleanup, application-specific behavior, and private APIs such as fuse_darwin_mount are out of scope.

References

• macFUSE 5.3.0 disables fuse_mount_compat25 on Apple
• macFUSE 5.3.0 fuse_chan_fd returns the channel’s fd field
• macFUSE 5.3.3 obtains the borrowed fd from MFChannel
• MFMount descriptor ownership contract
• macFUSE 4.9 asynchronous mount race fix
• macFUSE 5.3.3 asynchronous unmount implementation
• Related readiness issue #736