fix(archive): implement recursive nested archive scrubbing and preven… - #2
Open
ChrisBeaker wants to merge 2 commits into
Open
ChrisBeaker wants to merge 2 commits into
ChrisBeaker wants to merge 2 commits into
Conversation
…t plaintext corruption
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / Overview
This Pull Request resolves a critical bug where embedded sub-archives (e.g., nested
.txzor.tgztarballs inside a parent supportconfig, common in SMLM 5.1 / containerized environments) were corrupted during the scrubbing process, and adds native, recursive nested archive scrubbing with shared mappings.The Problem
Plaintext Corruption: In
processor.py, any file that is not matched as ansabinary or.objis processed by default as a UTF-8 text file. If the archive contains nested.txz,.tgz, or.zipfiles,FileProcessor.process_filereads their raw compressed binary bytes, executes string replacements on them, and writes them back. This completely corrupts the binary structure, rendering the nested archives permanently unextractable.Incomplete Scrubbing: Because nested archives were not extracted, the sensitive information (IPs, hostnames, passwords) inside sub-tarballs (e.g., podman host
OS or SMLM container logs) remained completely unanonymized.
The Solution / Changes
src/supportutils_scrub/processor.py):FileProcessor.process_fileto inspect the filename usingis_archive_path.src/supportutils_scrub/modes/archive.py):scrub_nested_archives_recursively.supportutils-scrubprocesses an archive, it now recursively scans the extracted directory tree for any embedded tarballs/archives._scrub_treeusing the active shared mapping dictionary, re-compressed back to its original filename/format, and cleaned up.Benefits & Impact
_scrub_treefunction, nested archive scrubbing automatically leverages the newly implemented multi-core parallel processing (--jobs) andtrie-regexperformance improvements, making deep-nesting scrubbing extremely fast!