ffi: load libraries from a mounted VFS

[mcollina]

The operating system's dynamic loader cannot open a library that lives in a mounted virtual file system: the reserved mount path has no real inode. Native addons already handle this in require(): the loader hands their bytes to process.dlopen(), which loads them from a private, self-cleaning image — an anonymous in-memory memfd on Linux.

This makes ffi.dlopen() and new ffi.DynamicLibrary() do the same, transparently:

const ffi = require('node:ffi');
const path = require('node:path');

// mountPoint is a mounted VFS (or a SEA useVfs mount).
const { lib, functions } = ffi.dlopen(
  path.join(mountPoint, `mylib.${ffi.suffix}`),
  { add: { arguments: ['i32', 'i32'], return: 'i32' } },
);

• A path under the reserved VFS root is detected with a prefix check; its bytes are read from the VFS and the native constructor loads them from the same kind of private image (AddonImage, moved from an anonymous namespace in node_binding.cc to node_binding.h so node_ffi.cc can reuse it). library.path keeps reporting the virtual path.
• Because the load happens inside the constructor, the image never outlives the call: it is unlinked right after uv_dlopen() on POSIX (in-memory memfd on Linux, so nothing touches the file system at all), and there is nothing left for dlclose() to clean up or reference-count. Windows retains the delete-on-close handle for the process lifetime, exactly as for addons.
• Libraries on the real file system are unaffected and load directly; the VFS machinery is only loaded for paths under the reserved root, so non-VFS users pay only a string prefix check.
• Loading from bytes requires file-system write permission for the temp directory on top of the FFI permission, mirroring dlopenBinary().

Bug fix included

Writing the test exposed a pre-existing bug, fixed in the first commit: the dlopen hook installed while a VFS is mounted always forwarded its flags parameter, so a two-argument process.dlopen() call for a real file-system path reached the original implementation with undefined as the flags. That coerces to 0, which is not a valid dlopen(2) mode, and loading any addon from the real file system failed with EINVAL while a VFS was mounted.

[pipobscure]

LETM (Looks Excellent To Me 😃 )

[pipobscure]

The only question I have is whether we want to hide the detail that the path may need materializing inside ffi.load like we did for process.dlopen. There are pro & cons for both. And maybe someone that is using FFI is the kind of person good with handling this manually. However this is also the kind of thing that's easy to forget to do (especially if you don't think of VFS as an option). So hiding it inside ffi.load would eliminate this fault category.

And since node:ffi has both dlopen and dlclose we even have a good place to put it and do reference counting. Making a library Symbol.disposable can call dlclose which would decrement the counter on the path and on the last close we can delete the file. And then just cleanup leftovers atexit.