Skip to content

refactor(vulnfeeds): migrate debian-copyright-mirror to Go - #5885

Open
jess-lowe wants to merge 8 commits into
google:masterfrom
jess-lowe:refactor/debian-copyright-mirror
Open

refactor(vulnfeeds): migrate debian-copyright-mirror to Go#5885
jess-lowe wants to merge 8 commits into
google:masterfrom
jess-lowe:refactor/debian-copyright-mirror

Conversation

@jess-lowe

Copy link
Copy Markdown
Contributor

Migrates the debian-copyright-mirror service from Python to Go and extracts reusable networking, archive, and decompression utilities for other vulnfeeds tools.

Key Changes

  • Go Migration: Replaced debian-copyright-mirror.py and debian-copyright-mirror.sh with a native Go service (cmd/mirrors/debian-copyright-mirror/main.go).
  • Streaming Pipeline: Streams filelist.yaml.xz over HTTP through an xz -dc pipe directly into a custom streaming line parser, extracting ~44k package paths without disk writes.
  • Worker Pool: Added a concurrent worker pool with HTTP/2 keep-alive connection pooling, per-request timeouts, retry logic, and atomic failure rate tracking (-max-failure-rate).
  • Native Archiving & GCS Upload: Built-in POSIX .tar packaging and direct streaming to Google Cloud Storage (cloud.google.com/go/storage).
  • Shared Utilities:
    • utility/decompress.go: Generic XZ streaming decompression helpers.
    • utility/download.go: Resilient atomic file downloader with exponential backoff.
    • utility/archive.go: Tar archive creation and extraction with traversal/bomb protections.
    • gcs-tools/gcs.go: Added ParseGCSPath URI helper.
  • Dockerfile & Deployment: Switched to a multi-stage Go build on a minimal alpine:3.21 image with the Go binary as direct ENTRYPOINT, removing the need for google-cloud-cli and wrapper shell scripts.

Testing Done

  • go test ./... in vulnfeeds (all tests passing).
  • golangci-lint run (0 issues).
  • Verified full download and local .tar generation with live Debian changelog metadata.

Comment thread vulnfeeds/utility/archive.go Fixed
Comment thread vulnfeeds/utility/archive.go Fixed

@another-rex another-rex left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a lot of unnecessary code imho. We don't need to reimplement the whole archiving stack in go. Just shell out to tar to extract the files like it did in the original script.

This should be about 200 lines of go code, not 1500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants