module: resolve and instantiate loader pipeline hooks

[guybedford]

This PR provides an implementation of a hooking mechanism worked on with some early feedback from @bmeck.

Usage

The es module loading pipeline is hooked via a --loader flag to Node, which specifies a file containing a loader definition.

For example usage:

• A loader that loads all files (including .js) as es modules is provided at https://github.com/guybedford/node/blob/resolve-hook-rebased/test/fixtures/es-module-loaders/js-loader.mjs.
• A loader that provides named exports for NodeJS builtin modules is provided at https://github.com/guybedford/node/blob/resolve-hook-rebased/test/fixtures/es-module-loaders/builtin-named-exports-loader.mjs.

Motivation

This work is built to follow principles of extensibility - allowing users (framework and platform developers for this feature) to customize their own hooks for their use case needs. The core ecosystem conventions for loading modules in NodeJS will remain the ones that NodeJS sets itself by the ecosystem popularity due to interop requirements.

The goal here though is that through having the ability for users to experiment with loader features will allow the most useful and popular features to potentially be merged back into NodeJS core to extend the ecosystem conventions. So roughly the extensible manifesto thinking.

I believe enabling these use cases is important to avoid possible ecosystem fragmentation through other means.

The examples above show common use cases, and these hooks are also sufficient to implement the package.json "module" property resolution concept using a loader (nodejs/node-eps#60).

Architecture

The hook architecture here is inspired by learnings from the WhatWG loader work, while aiming to cut down the scope of hooking to a bare minimum API surface area, as can be seen from the examples.

The loader itself is given control of module resolution, of the form:

resolve (specifier: string, parentModuleUrl: string) => { url: string, format: string }

Where the "format" returned is "esm" for the ES module loader, "cjs" for the traditional cjs loader, "builtin" to refer to a NodeJS core module, "json" for json and "addon" for a Node binary addon. In future this could be extended for new module formats (ie wasm, binary ast).

This resolver follows exactly what has always been the resolve hook in the module loader specifications, except instead of just returning a string URL, it returns the URL along with the format. Other information could also possibly be returned on this object in future including for example integrity metadata.

When the format returned is "dynamic", then we delegate loading to a dynamicInstantiate hook, hooked in the same way as resolve:

async dynamicInstantiate (url: string) => ({
  exports: string[],
  execute: (exports) => {
    // get and set functions provided for pre-allocated export names
    exports.x.set('asdf');
  }
})

This API is designed to enable the above example use cases, while ensuring that loader internals are kept private, to ensure that what is hooked is only what is intended to be hooked. For example, none of ModuleWrap is exposed.

Feedback very welcome!

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

esmodules