Prevent jscpd and secretlint report-folder race

[trevor-vaughan]

Summary

When COPYPASTE_JSCPD and REPOSITORY_SECRETLINT ran in parallel, jscpd's per-linter cleanup deleted megalinter-reports/copy-paste while secretlint was still scanning the workspace, producing sporadic ENOENT crashes whose stack traces were then parsed as SARIF findings. This fixes the race by keeping jscpd's report out of the shared workspace until clones are found and by giving secretlint a generated ignore file that excludes only MegaLinter's own report folder. It also hardens SARIF result counting and the secretlint ignore-file handling so a crashed or misconfigured secrets scanner can no longer be reported as clean.

Assisted-By: Claude Opus 5 (1M context) noreply@anthropic.com
Closes: #3979

Changes

• jscpd writes its report to a temp folder outside the workspace and copies it into copy-paste/ only when clones are found, so no file another linter is scanning is ever deleted mid-run; a stale report from a previous run is cleared at construction time.
• The published copy-paste/ folder is chmod'd to 0755 so a later non-root artifact-upload step can read it, instead of inheriting mkdtemp's 0700.
• secretlint now merges the user's ignore patterns into a single generated .secretlintignore-megalinter file (in the report folder, or a temp folder when reports are disabled), honoring a file named in LINTER_RULES_PATH and parsing the --secretlintignore=<value> form.
• A missing --secretlintignore file no longer silently falls back to .secretlintignore or .gitignore, which would exclude the exact files a secrets scanner must inspect; it degrades with a warning.
• A linter that exits without parsable SARIF is reported as a warning (results uncountable) rather than a clean success, and one that crashes before emitting SARIF surfaces its raw output instead of a phantom finding. Applies to can_output_sarif linters including REPOSITORY_SECRETLINT, REPOSITORY_SEMGREP, and REPOSITORY_TRIVY.
• REPORT_OUTPUT_FOLDER is always excluded from linted directories, even when EXCLUDED_DIRECTORIES is overridden.
• Added unit tests for jscpd report materialization and secretlint ignore handling, with ambient-env isolation in the shared linter tests; documented the secretlint ignore model and the report-folder exclusion in the descriptor and README.

[nvuillam]

@trevor-vaughan I am currently building a generic way of dynamically inject cli args / ignore files to all linters called in project cli_lint_mode

#8646

I think it will solve your use case (and many others), please tell me if you have any remarks :)

[trevor-vaughan]

Thanks @nvuillam

I rebased and took a look at the changes afterwards with the help of our 🤖 friends. Still working on getting a fully clean run with the new linters.

main no longer globs the jscpd report in project mode so I removed my secretlint-specific ignore file in favor of
upstream.

It looks like we still have issues around the root cause of this issue. If I'm reading things correctly:

• JsCpdLinter.complete_text_reporter_report() still rmtrees megalinter-reports/copy-paste on a clean run
• Linter.py:1106-1108 produces every non-CONSOLE reporter inside the parallel worker as soon as that linter finishes

This means that the delete happens while other linters are still walking the workspace.

Any linter running in project mode without excluded-directories forwarding still scans the report folder (35 on main).
These are the ones that get there by default rather than through a user override, so they appear to still be at risk
without this fix:

• GO_GOLANGCI_LINT
• KUBERNETES_HELM
• KUBERNETES_KUBESCAPE
• REPOSITORY_DUSTILOCK
• REPOSITORY_GIT_DIFF
• REPOSITORY_TRUFFLEHOG
• RUST_CLIPPY
• SALESFORCE_CODE_ANALYZER_*
• TERRAFORM_TFLINT

Also, the report folder exclusion disappears entirely as soon as a user sets EXCLUDED_DIRECTORIES, since that replaces the default list rather than extending it.

This PR has jscpd write to a temp folder outside the workspace and never delete from the report folder while other
linters are running, plus a guard so REPORT_OUTPUT_FOLDER stays excluded through an override. A report left behind by a previous run is still cleared, but at construction time, before any linter has started.

🤖 discovered -> in #8646, the secretlint path writes .megalinter-secretlintignore at the workspace root and removes it in cleanup_workspace_generated_files() at the end of that linter's run. This is the same create-then-delete-in-the-workspace-during-a-parallel-run shape that produced #3979 with a different file.

The PR also carries fixes unrelated to exclusions that were found along the way while fixing tests:

• a .secretlintignore in LINTER_RULES_PATH is currently not applied (only its base name is passed)
• an ignore file passed as --secretlintignore=<path> is silently replaced by .gitignore, since the equals form does
  not match the check that suppresses the fallback and secretlint keeps only the last value it is given
  • a typo in the flag name lands in the same place, so either way we end up excluding exactly what a secrets scanner
    should be reading
• on the SARIF side, a linter that exits without parsable output is charged a phantom finding instead of being reported
  as unmeasured
  • a crash counts as one error, and exiting 0 with unparsable SARIF counts as one warning
  • that warning is enough to turn the run yellow on its own, so for REPOSITORY_SECRETLINT/SEMGREP/TRIVY a scan
    that never ran shows up as a yellow run with one finding nobody can locate
  • after this PR the counts stay at 0 and the linter is reported as a warning with a log line saying its results could
    not be counted

I can split the fixes out into another PR if you like, but they ended up being related to fully evaluating this PR for
correctness.

[nvuillam]

Thanks for your efforts 😊
I know forward all excluded directories to the linters, including jscpd and secret lint... but i forward them only if they exist, and they probably not when the linters are run in parallel
If i force megalinter-reports even if it doesn't exist, wouldn't it make your pr simpler ?

[trevor-vaughan]

Yeah, if megalinter_reports was always excluded then things would get quite a bit simpler. I don't know that I'd remove the temp dir use though. If I have other scripts/agents looking around in megalinter_reports, they're going to expect the state to remain steady.

[nvuillam]

@trevor-vaughan i made this attempt #8720 that seems simpler, please can you have a look and tell me what you think ?