Add Claude AI test failure analysis to Slack notifications

[robbycochran]

Summary

Adds AI-powered test failure analysis that runs within the same workflow and includes intelligent insights in Slack notifications.

New Architecture

Instead of a separate workflow_run, this adds an analyze-failures job to the existing test workflows:

Test Workflow
 ├── amd64-integration-tests → (may fail)
 ├── arm64-integration-tests → (may fail)
 ├── s390x-integration-tests → (may fail)
 ├── ppc64le-integration-tests → (may fail)
 │
 ├── analyze-failures (runs if any fail)
 │    ├── Downloads all artifacts
 │    ├── Parses JUnit XML
 │    ├── Calls Claude AI
 │    └── Uploads analysis-report.md
 │
 └── notify (waits for analysis)
      ├── Downloads analysis-report.md
      └── Posts to Slack with insights

What Changed

Modified Workflows

• .github/workflows/integration-tests.yml

  • Added analyze-failures job before notify
  • Modified notify to download and include analysis report
  • Graceful fallback if analysis unavailable

• .github/workflows/unit-tests.yml

  • Same pattern for unit test failures

New Files

• .github/scripts/analyze_test_failures.py

  • Parses JUnit XML test reports
  • Extracts failure patterns
  • Calls Claude via Vertex AI
  • Generates markdown report

• .github/scripts/requirements.txt

  • anthropic - Claude API client
  • google-auth - GCP authentication
  • google-cloud-aiplatform - Vertex AI

• .github/scripts/README.md

  • Setup instructions
  • Usage examples
  • Troubleshooting guide

Example Output

Before (current):

@acs-collector-oncall
Integration tests failed.

After (with AI):

@acs-collector-oncall

🤖 AI Analysis

**Root Cause**: eBPF LSM hook attachment failures on RHCOS nodes. 
Tests failed with "permission denied" when attaching to lsm/file_open.

**Pattern**: All 3 failures occurred on RHCOS amd64 test VMs.

**Recommendations**:
• Check kernel version on RHCOS VMs - LSM BPF requires kernel 5.7+
• Verify CONFIG_BPF_LSM is enabled in kernel config
• Review recent changes to eBPF program loading logic
• Check for SELinux policy changes blocking BPF operations

---
**Statistics**
• Total Failures: 3
• Total Errors: 0
• Failed Jobs: amd64-integration-tests

Key Benefits

✅ Same workflow - No workflow_run complexity, better job tracking
✅ Job dependencies - notify waits for analyze-failures to complete
✅ Markdown report - Clean format, easy to include in Slack
✅ Graceful fallback - Still notifies even if analysis fails
✅ Artifact-based - Analysis report uploaded as workflow artifact
✅ Context available - All test artifacts already in the workflow

Required Secrets

Add these to repository secrets (see .github/scripts/README.md for setup):

# Required
GCP_CLAUDE_SERVICE_ACCOUNT_KEY  # GCP service account JSON with Vertex AI access

# Optional (have defaults)
GCP_CLAUDE_PROJECT_ID           # Defaults to "rhacs-eng"
GCP_CLAUDE_REGION               # Defaults to "us-central1"

Existing secret used:

• SLACK_COLLECTOR_ONCALL_WEBHOOK ✅ Already configured

Setup Instructions

1. Create GCP Service Account

gcloud iam service-accounts create claude-test-analyzer \
  --display-name="Claude Test Failure Analyzer"

2. Grant Vertex AI Access

gcloud projects add-iam-policy-binding rhacs-eng \
  --member="serviceAccount:claude-test-analyzer@rhacs-eng.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

3. Create Key

gcloud iam service-accounts keys create key.json \
  --iam-account=claude-test-analyzer@rhacs-eng.iam.gserviceaccount.com

4. Add to GitHub Secrets

Repository Settings → Secrets and variables → Actions → New secret:

• Name: GCP_CLAUDE_SERVICE_ACCOUNT_KEY
• Value: Paste entire contents of key.json

Testing

The workflow will activate on the next test failure. To test manually:

# Install dependencies
pip install -r .github/scripts/requirements.txt

# Run analysis locally
python .github/scripts/analyze_test_failures.py \
  --artifacts-dir test-artifacts \
  --output-file analysis-report.md \
  --workflow-name "Integration Tests" \
  --failed-jobs "amd64-integration-tests"

# View report
cat analysis-report.md

Advantages Over workflow_run Approach

Feature	This PR	workflow_run
Workflow complexity	✅ Single workflow	❌ Two workflows
Job dependencies	✅ Native needs:	❌ Separate trigger
Artifact access	✅ Direct download	❌ Cross-workflow
Debugging	✅ Same workflow run	❌ Different run
Failure handling	✅ Built-in fallback	❌ Must handle separately

Future Enhancements

• Correlate failures with recent commits
• Link to similar past failures
• Track failure patterns over time
• Auto-create issues for recurring failures

Testing Plan

1. ✅ Workflow syntax validated
2. ✅ Python script tested locally
3. ⏳ Will trigger on next test failure
4. ⏳ Monitor for false positives
5. ⏳ Iterate based on team feedback

cc @stackrox/acs-collector

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.34%. Comparing base (01135a9) to head (f3b267c).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3381   +/-   ##
=======================================
  Coverage   27.34%   27.34%           
=======================================
  Files          95       95           
  Lines        5420     5420           
  Branches     2545     2545           
=======================================
  Hits         1482     1482           
  Misses       3211     3211           
  Partials      727      727           

Flag	Coverage Δ
collector-unit-tests	27.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.