[Cherry-Pick][compat] Resolve a scoped spec once instead of re-entering it

[LiYuRio]

PR Category

Performance Optimization

PR Types

Performance

Description

devPR:#79798 (merged 2026-09-22, merge commit 7c722e5509)

Cherry-pick of the above onto release/3.4.

TorchProxyMetaFinder._find_spec_for_specific_module looks up the real spec with importlib.util.find_spec while the proxy is disabled, which is what is meant to keep the finder out of its own lookup. That only holds while the finder is registered once: enable_torch_proxy inserts it into sys.meta_path unconditionally, so nested guards leave duplicates behind, while disable_torch_proxy drops a single occurrence. With a duplicate left in place the lookup re-enters the finder for the very same name and restarts the whole resolution, and a name that has no spec at all never converges, so the lookups compound.

Measured with a probe for a missing submodule of a package in the enabled scope:

finder registrations in sys.meta_path	1	2	3	4
lookups per probe, before	1	4	15	64
lookups per probe, after	1	2	3	4

Every lookup toggles the proxy, and every toggle walks all of sys.modules to move torch* in and out of the shadow table, so the cost is O(lookups * len(sys.modules)) -- it also grows as the program imports more. Importing one package inside a scoped guard with 7 registrations already in place expanded the probes for a missing module into 13699 lookups and ~90M module-name comparisons: 37.9s, of which 0.5s was the actual import work.

Fix: suppress the finder for the name it is currently resolving, so the outer lookup finishes instead of starting over. Each registered copy is now asked exactly once. The proxy is still disabled around the lookup and exec_module still runs under its own guard, so the resulting specs, loaders and __enable_torch_proxy__ marks are unchanged.

Adaptations for release/3.4

• The guard is named use_torch_proxy_guard on this branch (not use_compat_guard), so the one conflicting line keeps this branch's name; the new test uses paddle.compat.use_torch_proxy_guard accordingly.
• test/compat/fake_modules/torch_proxy_local_enabled_package/ does not exist on this branch, so it is brought in together with the test that needs it (two small files).

The fix itself is identical to the develop commit: import threading, _resolving / _resolving_names on the finder, the early return None in find_spec, and add/discard around importlib.util.find_spec.

Verification

The proxy.py on this branch matches the released 3.x wheel (both use use_torch_proxy_guard), so the same patch could be applied to an installed paddle and the new test run against it for real:

new test, patch on :  2 tests, OK
new test, patch off:  2 tests, FAILED   (AssertionError: 3 != 1 / 4 != 2)

Full test/compat/test_torch_proxy.py, A/B on the same interpreter:

patch off: run=13  failures=5   <- includes the 2 new ones
patch on : run=13  failures=3   <- the 2 new ones now pass

The 3 remaining failures are identical with the patch on and off, i.e. unrelated to this change (they come from running this branch's test file against a released wheel on a machine that also has real PyTorch installed).

Equivalence, measured on a package that loads 51 sonicmoe submodules under a scoped guard: identical submodule set, identical set of modules marked with __enable_torch_proxy__ (220), same quack module object before and after, quack.gemm_interface.gemm_symmetric still importable.

End to end: 8-node x 8-GPU pretraining ran 216 steps with this patch applied to the venv's paddle, loss 12.2834 -> 6.3311, memory flat, no import-related errors. import paddlefleet_ops in the same interpreter: 4.89s -> 3.01s (A/B by disabling only the reentrancy bookkeeping, so the 1.88s is attributable to it alone).

是否引起精度变化

否

Only import-time module resolution changes. No kernel, no computation and no reduction order is touched.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (release/3.4@ea1e438). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff               @@
##             release/3.4    #79812   +/-   ##
===============================================
  Coverage               ?   100.00%           
===============================================
  Files                  ?         1           
  Lines                  ?        15           
  Branches               ?         0           
===============================================
  Hits                   ?        15           
  Misses                 ?         0           
  Partials               ?         0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.