Fix/copilot concurrency limit

[ApfelPresse]

Why are the changes needed?

Before this change multipart upload/download could start one goroutine per file.
With many small files, this created too much parallel work, drove CPU usage up, and in some cases caused jobs to fail.

This PR adds a configurable concurrency limit, so the number of parallel transfers is capped (GOMAXPROCS * concurrency-per-cpu, with a maximum limit).
That makes behavior more stable in Kubernetes/container environments while still allowing tuning for throughput.

What changes were proposed in this pull request?

1. Added bounded concurrency control for upload and download flows:
   • New CLI flag: --concurrency-per-cpu (default: data.DefaultConcurrencyPerCPU)
   • Effective worker cap: GOMAXPROCS * concurrency-per-cpu, bounded by MaxConcurrency
   • Applied to both download and upload scheduling paths via semaphore-based limiting
2. Improved HTTP download behavior:
   • DownloadFileFromHTTP now returns an error for non-2xx responses
   • Added logging for HTTP download start/status

How was this patch tested?

1. Added unit tests for:
   • sanitizeConcurrencyPerCPU
   • getConcurrency
   • acquireSemaphore (including context cancellation)
2. Updated existing uploader tests for the new constructor parameter.
3. Ran this fix branch in our environment for several days with real workloads (including many small files). CPU load stayed stable and we did not see regressions.

Labels

fixed
changed

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Docs link

Flytecopilot readme.md file

[codecov]

Codecov Report

❌ Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.23%. Comparing base (feb49f5) to head (5b1770d).
⚠️ Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
flytecopilot/data/download.go	64.10%	10 Missing and 4 partials ⚠️
flytecopilot/data/upload.go	75.00%	2 Missing and 1 partial ⚠️
flytecopilot/data/utils.go	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7730      +/-   ##
==========================================
+ Coverage   57.18%   57.23%   +0.04%     
==========================================
  Files         930      931       +1     
  Lines       58245    58314      +69     
==========================================
+ Hits        33308    33376      +68     
+ Misses      21874    21869       -5     
- Partials     3063     3069       +6     

Flag	Coverage Δ
unittests-datacatalog	53.51% <ø> (ø)
unittests-flyteadmin	53.23% <ø> (+0.07%)	⬆️
unittests-flytecopilot	49.26% <63.63%> (+1.20%)	⬆️
unittests-flytectl	64.09% <ø> (ø)
unittests-flyteidl	75.71% <ø> (ø)
unittests-flyteplugins	60.39% <ø> (+0.03%)	⬆️
unittests-flytepropeller	53.78% <ø> (+0.02%)	⬆️
unittests-flytestdlib	64.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.