v4.11 adds phantom file node

[timj]

Describe the bug

With 4.11 one of our scons builds fails with:

scons: *** [bin/buildOpts.py] Source `bin.src/buildOpts.py' not found, needed by target `bin/buildOpts.py'.

This was triggered by one of our SCons scripts running in a subdirectory assuming that Glob would only report files that exist. We can workaround the bug by doing an additional file existence check but we are reporting this in case the change to Variables() was an unexpected side effect in v4.11.

Our workaround is in lsst/sconsUtils#150

I asked Claude to write a demonstration reproducer. I understand that you may close my ticket with prejudice because I have used Claude and failed to post on the mailing list. My apologies if I offend anyone.

Summary

In SCons 4.11.0 a Variables object whose saved-variables file does not exist still causes a File node to be created for that file. The node is created in the directory of the SConscript that was being read when the variables were applied to an environment. Glob() reports the nodes SCons knows about in a directory as well as the files present on disk, so the node for the absent file is returned by a glob of that directory. Passing it to a builder then fails the build with Source ... not found.

This is a regression from 4.10.1, where a missing saved-variables file had no observable effect.

Versions

• Broken: SCons: v4.11.0.190853a12d8c2128f29abe849a6a162f0ed9f049
• Last known good: SCons: v4.10.1.055b01f429d58b686701a56df863a817c36bb103
• Python 3.13.14, macOS 26.6.1 on arm64 (nothing here looks platform specific)

Reproducer

No saved-variables file is created anywhere in this tree.

mkdir -p globvars/sub && cd globvars

cat > SConstruct <<'EOF'
SConscript("sub/SConscript")
EOF

cat > sub/SConscript <<'EOF'
env = Environment(variables=Variables(["custom.py"]))
print("Glob('#sub/*') ->", sorted(str(n) for n in Glob("#sub/*")))
EOF

touch sub/real.py

scons -Q -n

Expected, and what 4.10.1 prints

Glob('#sub/*') -> ['SConscript', 'real.py']
scons: `.' is up to date.

Actual, on 4.11.0

Glob('#sub/*') -> ['SConscript', 'custom.py', 'real.py']
scons: `.' is up to date.

custom.py does not exist, in sub/ or anywhere else.

The consequence for a real build

Anything that feeds the glob to a builder now fails.
Replace sub/SConscript with:

env = Environment(variables=Variables(["custom.py"]))
for node in Glob("#sub/*"):
    if str(node) != "SConscript":
        Default(env.Command("#out/" + str(node), node, Copy("$TARGET", "$SOURCE")))

On 4.10.1 the one real file is copied:

Copy("out/real.py", "sub/real.py")

On 4.11.0 the build dies before copying anything:

scons: *** [out/custom.py] Source `sub/custom.py' not found, needed by target `out/custom.py'.

It is Update() that creates the node, not the constructor

opts = Variables(["custom.py"])
print("after Variables()      ->", sorted(str(n) for n in Glob("#sub/*")))
env = Environment(variables=opts)
print("after Environment()    ->", sorted(str(n) for n in Glob("#sub/*")))

after Variables()      -> ['SConscript', 'real.py']
after Environment()    -> ['SConscript', 'custom.py', 'real.py']

So which directory acquires the node depends on where the Variables object is applied to an environment, not on where it was created. A project that constructs its environment from a subsidiary SConscript gets the node in that subdirectory.

Analysis

Variables.Update() in SCons/Variables/__init__.py reads each saved-variables file through a node:

for filename in self.files:
    node = env.File(filename)
    contents = node.get_text_contents()
    if contents:
        ...

env.File() creates the node whether or not a file backs it, and it resolves a relative name against the current SConscript directory. get_text_contents() returns an empty string for a node with no file, so the read is correctly skipped, but the node remains in the directory's entries.

Dir._glob1() in SCons/Node/FS.py collects those entries before it consults the disk:

node_names = [ v.name for k, v in dir.entries.items()
               if k not in ('.', '..') ]
names.extend(node_names)

so the node is matched by the pattern and returned.

Before 4.11 the same loop tested the file with os.path.exists(filename) and opened it directly, which created nothing. The change came in with the fix for #816, so that a saved-variables file could be found in a source directory or a repository rather than only in the build directory:

• 8c9149e6151c9be207df2a80872b772a0d68b8b6 — "Read Variables saved-variables file from the source directory (Variables('custom.py') should read from source directory #816)"
• cb74b0809ec7f857bf21a2eb0901b0e5d26ae9e2 — "Simplify logic to just use default File() behavoir and use that object to get the text contents of the file"

The CHANGES.txt entry is "Fix Variables() not reading a saved-variables file (e.g. custom.py)".

The underlying goal seems right; the side effect of materializing a node for a file that exists nowhere does not.

Impact

This reached us through the LSST build system. sconsUtils always offers an optional buildOpts.py to Variables, and it installs scripts by globbing a bin.src/ directory. In one of the packages the environment is first constructed from bin.src/SConscript, so on 4.11.0 every build of that package fails with:

scons: *** [bin/buildOpts.py] Source `bin.src/buildOpts.py' not found, needed by target `bin/buildOpts.py'.

See lsst/sconsUtils#150.

Reporting it because the failure is remote from its cause: an optional file that is absent, and has always been absent, silently becomes a build target, and only for whichever directory happened to be current.

Suggested resolutions

Any of these would fix it for us; the first seems closest to the intent of #816.

1. Do not leave a node behind when no file is found. Resolve the candidate paths (local directory, source directory, repositories) and only create the node once a readable file has been located.
2. Resolve the file relative to the top-level directory rather than the current SConscript directory. The documented use is a top-level custom.py, and depending on which SConscript was being read is surprising in its own right.
3. Failing either, consider whether Glob() should offer a way to exclude nodes that neither exist nor have a builder. That is a broader change, and it would not address the surprise of the node existing at all.

[mwichmann]

Haven't had a chance to look at this yet, so this is just a response to:

This was triggered by one of our SCons scripts running in a subdirectory assuming that Glob would only report files that exist.

Glob exists specifically to pick up files that are known to the build graph, whether or not a physicial file exists at that moment. So that's "expected behavior". If you dont need to find files that don't exist, the Python glob module should be fine (Glob was developed from quite an old glob.glob - the latter has moved on so they don't diff well, but the different intent is the interesting part).

[timj]

@mwichmann thanks for that important clarification. I've modified my fix to use glob.glob since in this particular code path it's scanning for existing files and not scanning for generated files. I think it's still interesting that 4.11 added a phantom file to the graph.

[mwichmann]

I'm inclined to think there's something to your suggestion #1 - while we want a node to provide a way to reference a file without first making sure it's been built (a good thing), it doesn't feel like that reference should remain around if there's no way for SCons to cause it to exist.

[bdbaddog]

or you can add ondisk=True to your SCons Glob call and it will only pick up files currently there..

[timj]

I tried ondisk=True and that doesn't fix the problem (it's the default value anyhow) since it looks like it only adds to the list and doesn't filter out existing entries (so you just get the phantom entries with ondisk=False).

[mwichmann]

You've probably made your changes, but for grins, could you let me know if this tiny change would have helped?

diff --git a/SCons/Variables/__init__.py b/SCons/Variables/__init__.py
index c8ce37c84..49dbe8b61 100644
--- a/SCons/Variables/__init__.py
+++ b/SCons/Variables/__init__.py
@@ -275,6 +275,8 @@ class Variables:
             # is found in the source directory (or a repository) as well as
             # the build directory, e.g. when using a variant dir (issue #816).
             node = env.File(filename)
+            if not node.exists():
+                continue  # bugfix #4893
             contents = node.get_text_contents()
             if contents:
                 # issue #4645: don't exec directly into values,

[timj]

Intriguingly it does not fix my problem:

scons: Reading SConscript files ...
Does not exist:  buildOpts.py
Setting OS X binary compatibility level: 11.0
EUPS integration: enabled
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [bin/buildOpts.py] Source `bin.src/buildOpts.py' not found, needed by target `bin/buildOpts.py'.
scons: building terminated because of errors.

with a print added saying when the node is skipped.

[mwichmann]

Okay. Thanks for the info! The theory was that it's not fully in the plumbing untiil you've called something "real" on it - thus the bailout before get_text_contents hopefully leaving a better state, but I guess I was wrong about that.

[mwichmann]

Here's another approach...

diff --git a/SCons/Variables/__init__.py b/SCons/Variables/__init__.py
index c8ce37c84..014cfcb5c 100644
--- a/SCons/Variables/__init__.py
+++ b/SCons/Variables/__init__.py
@@ -275,7 +275,15 @@ class Variables:
             # is found in the source directory (or a repository) as well as
             # the build directory, e.g. when using a variant dir (issue #816).
             node = env.File(filename)
-            contents = node.get_text_contents()
+            if node.rexists():
+                contents = node.get_text_contents()
+            else:
+                # If the node doesn't exist, we don't want it to be returned
+                # by Glob() and similar. So we remove it from the directory's
+                # entries. Issue #4893.
+                node.dir.entries.pop(node.name, None)
+                continue
+
             if contents:
                 # issue #4645: don't exec directly into values,
                 #   so we can iterate through for unknown variables.

[timj]

That does fix it for me!

[mwichmann]

Here's a testcase.

issue_4893.py