feat: Add database-rds-resilience and database-rds-security skills

skills/database-rds-resilience/CHANGELOG.md

@@ -0,0 +1,20 @@
+```markdown
+# Changelog
+
+## 1.0.0
+
+- Initial release: 66-blocker catalog across 7 categories (Failover Timing, Snapshot
+  Restore, Encryption, KMS API Throttling, Cross-Region DR, Application-Layer
+  Resilience Gaps, Account-Level Service Quotas)
+- Topology-aware RTO/RPO calculation with 4-dimension scoring (Regional HA, Data
+  Protection, Cross-Region DR, Application Resilience) summing to 100
+- Read-only, AWS CLI + Service Quotas API only — no MCP, no database connection
+- QT-07 (concurrent cross-region snapshot copies) corrected to the documented AWS
+  default of 5 (was previously stated as 20 and inconsistent with the detection rule)
+- Deduplicated detection rules that were previously defined twice in two separate
+  YAML blocks (Quota Detection Rules and Detection Rules) into a single rule set
+- Moved the 66-blocker catalog and remediation playbooks to `references/` to keep
+  `SKILL.md` under the repository's ~500-line guideline
+- Relabeled the Multi-AZ conversion remediation from "Zero Downtime" to "Deferred —
+  Applies at Next Maintenance Window" with an explicit warning about the performance
+  impact of initial standby synchronization and the failover risk of `--apply-immediately`

skills/database-rds-resilience/README.md

@@ -0,0 +1,116 @@
+```markdown
+# Database RDS Resilience Skill
+
+A skill for AWS DevOps Agent that performs **read-only** topology-aware resilience
+assessments for Amazon RDS and Aurora, uncovering hidden blockers that silently
+prevent meeting stated RTO/RPO targets.
+
+## Purpose
+
+Customers often assume their database tier is resilient because Multi-AZ is enabled
+or backups are configured — but architectural and account-level constraints (encryption
+dependencies, KMS quota throttling, snapshot restore mechanics, cross-region service
+limits) frequently block a real recovery from meeting the stated target. This skill
+surfaces those blockers proactively, before a DR event exposes them, and calculates
+realistic RTO/RPO values based on actual configuration rather than documentation
+assumptions.
+
+## Key Capabilities
+
+- **66-blocker catalog** across 7 categories: Failover Timing, Snapshot Restore,
+  Encryption, KMS API Throttling, Cross-Region DR, Application-Layer Resilience Gaps,
+  and Account-Level Service Quotas
+- **Quota-aware RTO adjustment** — accounts for the default 5-concurrent-copy limit
+  (QT-07) when calculating recovery time for fleets needing cross-region DR
+- **4-dimension scoring (0-100)**: Regional HA, Data Protection, Cross-Region DR,
+  Application Resilience
+- **CLI remediation playbooks** with explicit downtime/performance-impact call-outs —
+  see `references/remediation-playbooks.md`
+
+## Prerequisites
+
+### IAM Permissions
+
+The DevOps Agent role needs the following read-only permissions:
+
+rds:DescribeDBClusters rds:DescribeDBInstances rds:DescribeGlobalClusters rds:DescribeAccountAttributes service-quotas:GetServiceQuota service-quotas:ListServiceQuotas
+
+`service-quotas:*` permissions above are **not** included in `AIDevOpsAgentAccessPolicy`
+and must be added explicitly to the DevOps Agent execution role, or quota-pressure
+findings (Category 7 / QT-01 through QT-20) will be skipped.
+
+### AWS Resources
+
+- An Amazon RDS or Aurora instance/cluster
+- No VPC access, no database credentials, and no Data API required — this skill is
+  control-plane only
+
+## Limitations
+
+- **Advisory only.** This skill produces findings and CLI remediation *suggestions*;
+  it never applies changes. All commands in `references/remediation-playbooks.md`
+  are for manual execution by an operator.
+- **Quota checks reflect account defaults unless increased.** If you have requested
+  quota increases via AWS Support, actual limits may exceed the defaults in
+  `references/blocker-catalog.md` — the skill reads live values via
+  `describe-account-attributes` and `service-quotas`, but the catalog's documented
+  defaults are shown for context.
+- **Some blockers cannot be detected from the control plane alone** (e.g., whether a
+  DR runbook is documented, or whether a DR test was actually performed in the last 90
+  days) — these are flagged as scoring inputs the operator must confirm manually.
+
+## Agent Types
+
+This skill is used by the following agent types (selected in the Operator Web App at
+upload time):
+
+- **Chat tasks** — interactive resilience assessments and targeted category checks
+- **Incident RCA** — root cause analysis where a failed or slow recovery is a
+  contributing factor
+
+Select **Generic** instead if you want the skill available to all agent types.
+
+## Uploading to AWS DevOps Agent
+
+**Option A: Import from GitHub (recommended)**
+
+In the DevOps Agent web app, go to Settings → Add Skill → Import from repository,
+then point to the `skills/database-rds-resilience` directory.
+
+**Option B: Upload as a zip file**
+
+```bash
+cd skills
+zip -r database-rds-resilience.zip database-rds-resilience/ -i '*.md' '*.txt' '*.json' -x '*/evals/*'
+
+Upload via Skills → Add skill → Upload skill in the DevOps Agent web app. Select agent types Chat tasks and Incident RCA.
+
+Option C: Upload via the Asset API — see the AWS DevOps Agent User Guide.
+How to Use This Skill
+"Assess resilience for my Aurora cluster prod-orders-1"
+"What's my actual RTO if I lose this region?"
+"Am I blocked from setting up Aurora Global Database on this cluster?"
+"Check my account for cross-region DR quota pressure"
+"Why would my snapshot restore take longer than expected?"
+Skill Structure
+database-rds-resilience/
+├── SKILL.md                              # Main skill instructions
+├── README.md                             # This file
+├── CHANGELOG.md                          # Version history
+├── references/
+│   ├── blocker-catalog.md                # 66 blockers across 7 categories
+│   └── remediation-playbooks.md          # CLI remediation + report format
+└── evals/
+    ├── evals.json                        # Functional test scenarios
+    ├── eval_queries.json                 # Trigger tests
+    └── report.json                       # Evaluation results
+Safety
+
+This skill operates in read-only mode:
+
+    No DDL, DML, or DCL — no infrastructure changes of any kind
+    All CLI commands in references/remediation-playbooks.md are for manual execution by an operator, with explicit prerequisite and impact call-outs where a remediation carries downtime or performance-impact risk (e.g., Multi-AZ conversion)
+
+Non-production disclaimer
+
+    ⚠️ This skill is sample code, not intended for production use without additional review and testing. Validate in a non-production environment first. Quota defaults and RTO/RPO estimates are general guidance and should be confirmed against your account's actual limits and tested recovery times.

skills/database-rds-resilience/SKILL.md

@@ -0,0 +1,246 @@
+---
+name: database-rds-resilience
+description: Topology-aware resilience assessment for RDS and Aurora — detects 66 hidden blockers across failover timing, snapshot restore, encryption, KMS throttling, cross-region DR, application layer, and account quotas that silently prevent meeting stated RTO/RPO targets
+metadata:
+  version: "1.0.0"
+  author: kiranmam
+  tags: [database, rds, aurora, resilience, dr, rto, rpo]
+---
+
+# DevOps Agent — RDS/Aurora Resilience Blockers Skill
+
+## Agent Identity
+
+You are read-only **RBUI (Resilience Blockers Underneath Iceberg) DevOps Agent** — a topology-aware resilience assessment specialist for AWS RDS and Aurora databases. Your mission is to uncover hidden blockers between documented DR capabilities and actual recovery performance.
+
+**Core Question You Answer:**
+> "Given this specific AWS infrastructure topology and snapshot strategy, what are the actual, achievable RTO and RPO values — and what hidden service limitations prevent meeting stated targets?"
+
+---
+
+## Assessment Workflow
+
+1.    COLLECT → Gather topology (describe-db-instances, describe-db-clusters, describe-account-attributes)
+2.    CLASSIFY → Map each resource against the Blocker Catalog (references/blocker-catalog.md)
+3.    CALCULATE → Compute realistic RTO/RPO per resource (quota-adjusted)
+4.    REPORT → Produce gap analysis with prioritized remediation (references/remediation-playbooks.md)
+
+
+## References
+
+- `references/blocker-catalog.md` — Full catalog of 66 blockers across 7 categories (Failover Timing, Snapshot Restore, Encryption, KMS Throttling, Cross-Region DR, Application Layer, Account Quotas)
+- `references/remediation-playbooks.md` — CLI remediation templates and the report output format
+
+---
+
+## RTO/RPO CALCULATION FORMULAS
+
+### Snapshot-Based Recovery (Backup and Restore Pattern)
+
+RTO = snapshot_locate_time + restore_initiation_time + instance_boot_time (size-dependent, lazy-loading) + parameter_group_reapply_time + security_group_reapply_time + dns_propagation_time + application_reconnection_time + data_warmup_time (if performance-critical)
+
+Typical RTO by DB size: < 100 GB: 15-30 minutes 100-500 GB: 30-60 minutes 500 GB-1 TB: 60-90 minutes
+
+    1 TB: 90-180+ minutes
+
+RPO = backup_frequency (automated: up to 24h) + transaction_log_upload_interval (5 minutes for PITR)
+
+Typical RPO: With PITR: 5 minutes maximum Without PITR (snapshot only): up to 24 hours
+
+### Multi-AZ Failover (In-Region)
+
+RTO = failover_detection_time + dns_update_time + client_dns_cache_expiry (JVM/OS/network) + connection_pool_drain_time
+
+Typical RTO: RDS Single Standby: 60-120 seconds RDS Two Standbys: <35 seconds Aurora with readers: 15-30 seconds (with proper config) Aurora without readers: 10-15 minutes (must provision new instance)
+
+RPO = 0 (synchronous replication within AZ pair)
+
+### Aurora Global Database (Cross-Region)
+
+RTO = failure_detection_time + switchover/failover_execution (typically <1 minute) + dns_propagation (5s TTL * 2-3 cycles) + application_reconnection
+
+Typical RTO: Planned switchover: <1 minute Unplanned failover: 1-2 minutes Manual failover (version mismatch): 5-15 minutes
+
+RPO = replication_lag (typically <1 second, but varies under load)
+
+### Quota-Aware RTO Adjustment
+When concurrent cross-region snapshot copy limit (QT-07, default 5) affects mass DR:
+
+adjusted_rto_per_db = base_rto + (batch_position / 5) * avg_copy_time
+Example: 12 databases, avg copy time 20 min
+Batch 1 (DBs 1-5): RTO = base_rto + 0 = 30 min
+Batch 2 (DBs 6-10): RTO = base_rto + 20 min = 50 min
+Batch 3 (DBs 11-12): RTO = base_rto + 40 min = 70 min
+When instance quota blocks restore:
+RTO = infinity until quota increase approved (hours to days via AWS Support)
+
+---
+
+## ASSESSMENT SCORING MATRIX
+
+| Score Range | Rating | Meaning |
+|-------------|--------|---------|
+| 80-100 | EXCELLENT | Multi-region, encrypted, auto-failover, tested DR |
+| 60-79 | GOOD | Regional HA present, some DR gaps, mostly encrypted |
+| 40-59 | FAIR | Basic HA (Multi-AZ) but no cross-region, some gaps |
+| 20-39 | POOR | Single-AZ, minimal backup, major gaps |
+| 0-19 | CRITICAL | No HA, no DR, unencrypted, at risk of total loss |
+
+### Scoring Dimensions (25 points each — sums to 100):
+
+**Regional HA (25 pts):**
+- Multi-AZ enabled: +10
+- Aurora with 2+ readers: +8 (or RDS 2-standby: +8)
+- Deletion protection ON: +4
+- Backup retention >= 14 days: +3
+
+**Data Protection (25 pts):**
+- Encrypted at rest: +10
+- Customer-managed KMS key: +5
+- Cross-region backup replication: +7
+- PITR enabled (retention >0): +3
+
+**Cross-Region DR (25 pts):**
+- Global Database or cross-region replica: +15
+- Same engine version across regions: +5
+- DR tested within last 90 days: +5
+
+**Application Resilience (25 pts):**
+- RDS Proxy or AWS JDBC Driver: +10
+- TCP keepalive configured: +5
+- DNS TTL <= 5s (or proxy bypass): +5
+- Failover runbook documented: +5
+
+---
+
+## DETECTION RULES
+
+Apply these rules to flag blockers when assessing a resource. Each rule fires once — there are no duplicates.
+
+```yaml
+rules:
+  - id: DETECT_SINGLE_AZ
+    condition: multiAZ == false AND dBClusterIdentifier == null
+    blockers: [FT-08]
+    severity: CRITICAL
+    message: "Single-AZ RDS instance — AZ failure = full outage"
+
+  - id: DETECT_AURORA_NO_READER
+    condition: engine starts_with "aurora" AND clusterMembers.count == 1
+    blockers: [FT-06]
+    severity: CRITICAL
+    message: "Aurora cluster with single writer — no failover target"
+
+  - id: DETECT_UNENCRYPTED
+    condition: storageEncrypted == false
+    blockers: [EN-01, EN-04]
+    severity: HIGH
+    message: "Unencrypted — blocks all cross-region DR paths"
+
+  - id: DETECT_NO_CROSS_REGION
+    condition: no global database AND no cross-region replica AND no cross-region backup
+    blockers: [CR-01]
+    severity: HIGH
+    message: "No cross-region DR — regional failure = total outage"
+
+  - id: DETECT_LOW_BACKUP_RETENTION
+    condition: backupRetentionPeriod <= 7
+    blockers: [SR-08]
+    severity: MEDIUM
+    message: "Minimum backup retention — limited PITR window"
+
+  - id: DETECT_DELETION_PROTECTION_OFF
+    condition: deletionProtection == false
+    blockers: []
+    severity: HIGH
+    message: "Deletion protection OFF — accidental deletion possible"
+
+  - id: DETECT_VERSION_MISMATCH
+    condition: global_database AND primary.version != secondary.version
+    blockers: [CR-05, CR-06]
+    severity: CRITICAL
+    message: "Version mismatch blocks global failover/switchover"
+
+  - id: DETECT_EMPTY_CLUSTER
+    condition: engine starts_with "aurora" AND clusterMembers.count == 0
+    blockers: []
+    severity: MEDIUM
+    message: "Empty cluster — no instances, no operational value"
+
+  - id: DETECT_DEFAULT_PARAM_GROUP
+    condition: parameterGroup starts_with "default."
+    blockers: [SR-05]
+    severity: LOW
+    message: "Using default parameter group — performance may not be optimized"
+
+  - id: DETECT_AWS_MANAGED_KEY
+    condition: kmsKeyId contains "alias/aws/rds"
+    blockers: [EN-04]
+    severity: MEDIUM
+    message: "AWS-managed key blocks cross-account DR"
+
+  - id: DETECT_SNAPSHOT_QUOTA_PRESSURE
+    condition: manual_snapshots_count >= (snapshot_limit * 0.8)
+    blockers: [QT-01, QT-02]
+    severity: HIGH
+    message: "Snapshot quota >80% used — DR snapshot creation may fail"
+
+  - id: DETECT_INSTANCE_QUOTA_PRESSURE
+    condition: db_instances_count >= (instance_limit * 0.8)
+    blockers: [QT-03]
+    severity: CRITICAL
+    message: "Instance quota >80% — cannot restore/create instances during DR"
+
+  - id: DETECT_CLUSTER_QUOTA_PRESSURE
+    condition: db_clusters_count >= (cluster_limit * 0.8)
+    blockers: [QT-04]
+    severity: CRITICAL
+    message: "Cluster quota >80% — cannot create clusters during DR"
+
+  - id: DETECT_CROSS_REGION_COPY_BOTTLENECK
+    condition: databases_needing_cross_region_dr > 5
+    blockers: [QT-07]
+    severity: HIGH
+    message: "More than 5 DBs need cross-region DR — concurrent copy limit (default 5) serializes recovery"
+
+  - id: DETECT_GLOBAL_DB_LIMIT
+    condition: global_clusters_count >= 4
+    blockers: [QT-10]
+    severity: MEDIUM
+    message: "Approaching Global Database limit (5 max) — not all clusters can get cross-region DR"
+
+  - id: DETECT_CROSS_REGION_BACKUP_LIMIT
+    condition: cross_region_replications >= 20
+    blockers: [QT-06]
+    severity: MEDIUM
+    message: "Approaching cross-region automated backup replication limit (20 max)"
+
+  - id: DETECT_DR_REGION_HEADROOM
+    condition: target_region_instances >= (instance_limit * 0.6)
+    blockers: [QT-03, QT-04]
+    severity: HIGH
+    message: "DR target region has limited headroom — may not accommodate full failover"
+
+QUOTA ASSESSMENT COMMANDS
+# Primary command — shows all RDS quota usage vs limits in one call
+aws rds describe-account-attributes --region {{REGION}}
+
+# Detailed quota limits (if custom limits were requested)
+aws service-quotas list-service-quotas --service-code rds --region {{REGION}}
+
+# Check DR target region headroom
+aws rds describe-account-attributes --region {{DR_REGION}}
+
+# Check KMS quota usage
+aws service-quotas get-service-quota \
+  --service-code kms \
+  --quota-code L-6E388A8A \
+  --region {{REGION}}
+Safety
+
+This skill operates read-only:
+
+    No DDL, DML, or DCL
+    Produces findings and CLI remediation suggestions only — never executes remediation
+    All commands in references/remediation-playbooks.md are for manual execution by an operator, with explicit call-outs for actions that carry downtime or performance-impact risk
+

skills/database-rds-resilience/evals/eval_queries.json

@@ -0,0 +1,12 @@
+[
+  {"query": "Assess resilience for my Aurora cluster prod-orders-1", "should_trigger": true},
+  {"query": "What is my actual RTO if I lose this region?", "should_trigger": true},
+  {"query": "Am I blocked from setting up Aurora Global Database on this cluster?", "should_trigger": true},
+  {"query": "Check my account for cross-region DR quota pressure", "should_trigger": true},
+  {"query": "Why would my snapshot restore take longer than expected?", "should_trigger": true},
+  {"query": "Will KMS throttling affect my disaster recovery restore?", "should_trigger": true},
+  {"query": "What hidden limits could break my RDS failover?", "should_trigger": true},
+  {"query": "Write me a Python function to reverse a string", "should_trigger": false},
+  {"query": "What's the weather in Seattle today?", "should_trigger": false},
+  {"query": "Translate this paragraph to French", "should_trigger": false}
+]