Native Yarn Plug'n'Play module resolution#4568
Conversation
Under Yarn PnP there is no node_modules tree; a manifest (.pnp.data.json, or .pnp.cjs with the data inlined — Yarn's default) maps every (package, importer) pair to the package's on-disk location, which for npm packages is a path inside a Yarn cache .zip archive. tsc gets PnP support because Yarn ships a runtime patch for it; the native binary cannot be patched that way, so tsgo fails to resolve any dependency in a PnP project (TS2307). This implements PnP resolution natively. - internal/pnp: loads and compiles a PnP manifest (a .pnp.data.json sidecar, or the JSON inlined in a .pnp.cjs, extracted from its RAW_RUNTIME_STATE literal) and answers RESOLVE_TO_UNQUALIFIED (package identifier + importer directory -> package directory), following the Yarn PnP specification and the reference resolver's actual behavior where it diverges: the trailing-slash longest-prefix locator loop, the top-level fallback (gated, like Yarn's makeApi, so the synthetic top-level locator never falls back and workspaces are excluded via the fallbackExclusionList), unfulfilled-peer vs not-found, and aliased targets. Discovery walks up from the tsconfig/current directory for a manifest. Resolved package locations are kept in Yarn's __virtual__/<hash>/<n> space (not dereferenced): the locator table is keyed by these paths, so a file resolved under a virtual location must keep it for findLocator to identify the owning package when resolving that package's own imports. - internal/vfs/zipoverlay: a vfs.FS decorator that serves files inside Yarn cache .zip archives, reporting the archive and its interior directories as directories so the module resolver's parent-directory checks pass. It dereferences __virtual__/$$virtual paths to their backing location (a cache .zip or a plain workspace/unplugged directory) only at the point of a filesystem read, while Realpath returns virtual/zip paths as identity so program paths stay in the locator space. Non-PnP paths delegate unchanged. Inserted beneath the bundled/cached FS wrappers at the tsc, LSP, and API-server construction sites. Zip-interior lookups are case-sensitive (a documented limitation on case-insensitive hosts). - internal/module/resolver: when a PnP manifest governs the project, the nearest node_modules walk is preceded by a PnP lookup that mirrors the walk's shape (preferred TS/DTS extensions on the implementation package, then DTS in the matching @types package, then fallback JS/JSON). When PnP governs the importer but the specifier does not resolve through the manifest, the classic ancestor walk is skipped (so an undeclared "phantom" dependency is not picked up from a stray node_modules) while typeRoots still runs. When no manifest is found the branch is a no-op, so non-PnP projects are unaffected. Hermetic Go unit tests cover the manifest loader, resolution (workspace/npm/ aliased/peer/undeclared, top-level fallback and exclusion, transitive imports from a virtualized package), the __virtual__ dereference (verified against Yarn berry's VirtualFS resolveVirtual cases), and the zip overlay. Refs microsoft#460
|
We already have #1966 I don't think we need a second PR for this unless it's a group effort or this is better in some specific way? (I don't think your agent looked for existing PRs.) |
Yeah, read the issue but not the open PR's… I'll make sure there aren't any edge cases missed with the other since I ran into several in the loop. |
|
Closing this — @jakebailey is right that #1966 is the established effort (Datadog, reviewed by @arcanis, in production), and it's a strict superset of what I had here. My mistake for not checking for an existing PR before opening. Rather than a duplicate, I've put the useful part into a small PR against #1966's branch: its |
|
The test-coverage + fixes PR is up against #1966's branch: GGomez99#8 It removes the |
|
@microsoft-github-policy-service agree |
2 participants
Re #460.
What
Native Yarn Plug'n'Play resolution. When a
.pnp.data.jsonor.pnp.cjsmanifest is found (walking up from the tsconfig / cwd), tsgo resolves modules from the manifest instead of anode_modulestree — so a PnP-installed project type-checks without thepnpify/ editor-SDK shim TypeScript needs under PnP today.No manifest found → resolution is byte-for-byte the classic walk. The whole feature is inert off-PnP; the existing
compiler/execute/apisuites pass unchanged.How
Three pieces.
1.
internal/pnp— manifest loader + resolver (a port of Yarn'sRESOLVE_TO_UNQUALIFIED+FIND_LOCATOR).Resolve(specifier, importerDir): bare-identifier parse → owning-locator lookup (longest-prefix walk up the importer path) → dependency target (direct / aliased / null-peer) → top-level fallback → the package directory + subpath.__virtual__space, not dereferenced — the locator table is keyed by those raw paths, so a file resolved under a virtual location must keep it forFIND_LOCATORto identify the owning package when resolving that package's own imports.Loadreads the.pnp.data.jsonsidecar, or extracts the inlinedRAW_RUNTIME_STATEliteral from a default.pnp.cjs(no JS engine).ignorePatternDatacompiles to a Goregexp(dropping the negative-lookahead fragments RE2 rejects).2.
internal/vfs/zipoverlay— avfs.FSdecorator for reading package files..zip(…/cache/react-npm-18.3.1-<hash>.zip/node_modules/react/index.d.ts). The OS reports the.zipas a regular file, so every interior path failsFileExists/DirectoryExistsand the resolver's parent checks abort. The overlay serves interior paths from the archive and reports interior directories as directories.__virtual__/$$virtualpaths at the point of the read — to a cache.zip(npm package) or a plain directory (workspace / unplugged package).Realpathreturns virtual/zip paths as identity, so program paths stay in the locator space.3.
internal/module/resolver.go— wiring.loadModuleFromNearestNodeModulesDirectoryconsults the manifest before the ancestornode_moduleswalk:node_modules— while still allowingtypeRoots;The three production FS construction sites (
cmd/tsgo/sys.go,cmd/tsgo/lsp.go,internal/api/server.go) wrap the OS FS with the overlay.Tests
Hermetic Go unit tests, no on-disk fixtures beyond
t.TempDirzips.internal/pnp— resolution: workspace dep · npm dep inside a.zip· subpath carry-through · aliased target · unfulfilled (null) peer · undeclared→NotFoundvs ignored-importer→Skippedvs relative-importer→Skipped· top-level fallback + exclusion list · transitive import from inside a virtualized package (the case that must not fall through to root) · virtual location kept +findLocatorround-trip.internal/pnp— loader +DerefVirtualPath: inlined.pnp.cjs(default + tight spacing, marker requires=) · malformed / missing / marker-less rejected · the dereference cases checked against Yarn berry's ownVirtualFS.resolveVirtualsuite ($$virtual, non-numeric count left unchanged, count 0/1 with/without file, depth > parents, dots, empty string).internal/vfs/zipoverlay— read / stat / list / walk inside a.zip· non-.zipvirtual → plain-dir delegation · zip-backed virtual → served from archive · missing-.zipand.zip-substring edge cases.Open questions
.pnp.*). Keep it implicit, or gate behind an explicit option / env?.pnp.cjs: extracting the literal without a JS engine is pragmatic but isn't a JS parser — acceptable, or restrict to the.pnp.data.jsonsidecar?exports/importsconditions — this resolves to the package directory and reuses the existing exports resolution; worth a closer look?Happy to split into loader / overlay / resolver-wiring commits if that reviews more easily.
Disclosure: authored with Claude Code (agent-assisted); I've read and understood the result and will shepherd it through review myself, per CONTRIBUTING's AI-assistance policy.