Skip to content

EKS modules: rvn-eks composite, selectable add-ons, services, and observability - #114

Open
mabadir wants to merge 32 commits into
mainfrom
mabadir/eks-observability
Open

EKS modules: rvn-eks composite, selectable add-ons, services, and observability#114
mabadir wants to merge 32 commits into
mainfrom
mabadir/eks-observability

Conversation

@mabadir

@mabadir mabadir commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

The module-side half of EKS deployment support (pairs with ravionhq/flightcontrol#6973).

Cluster and add-ons

  • compute/eks composite with the rvn-eks module definition
  • Selectable add-ons stack (compute/eks/addons): Karpenter (controller + default node pool), EBS CSI, AWS Load Balancer Controller (installed automatically with any shared load balancer), shared load balancers
  • Runner authenticates to the cluster via EKS access entries: self-registered Terraform runner entry plus an assumable Ravion Runner role with cluster-admin; orphaned Helm releases adopted via upgrade_install
  • Ravion Beacon agent shipped through the add-ons chart, defaulting to the public a8z1i1r2 registry (rvn-eks-addons 0.6.1)

Services

  • EKS service modules with container image builds

Observability

  • Workload metrics: Amazon Managed Prometheus + OTel collector + Grafana role in rvn-eks-addons
  • Logs: in-cluster Loki on S3 (replacing Container Insights), covered in tests, docs, and the module definition
  • UI wiring: workload and cluster metrics to AMP, logs to Loki

Greptile Summary

The PR adds a composite EKS platform, selectable cluster add-ons, AWS-side EKS workload infrastructure, reusable Helm charts, observability integrations, module definitions, and integration tests.

  • Provisions EKS clusters, node capacity, runner access, and optional Karpenter, storage, load-balancing, secrets, metrics, logs, and Beacon components.
  • Adds web, worker, and cron workload definitions with image builds and Helm deployments.
  • Adds shared ALB target-group routing and chart-based Kubernetes workload resources.
  • Adds chart CI and AWS/Terraform test coverage.

Confidence Score: 4/5

The target-group naming collision should be fixed before merging because two valid web workload names can make infrastructure creation fail.

Load-balanced workloads whose names share the retained 24-character prefix resolve to the same account-and-region target group name, while the workflow action pinning issue is independently non-blocking.

Files Needing Attention: compute/eks_service/locals.tf and .github/workflows/helm-charts.yml

Security Review

The new Helm workflow uses mutable action tags. Its permissions are limited to repository read access, but pinning these actions to full commit SHAs would prevent upstream tag movement from changing executed CI code.

Important Files Changed

Filename Overview
compute/eks_service/locals.tf Enables optional load balancing and derives target group names, but truncation can collide for workloads sharing a long prefix.
compute/eks_service/target_group.tf Creates an IP-mode target group consumed by the chart's TargetGroupBinding.
compute/eks/addons/beacon.tf Provisions Beacon credentials and deployment configuration; no changed-code blocking defect was established.
charts/rvn-eks-web/templates/targetgroupbinding.yaml Renders one TargetGroupBinding per supplied target group ARN for pod-IP registration.
.github/workflows/helm-charts.yml Adds chart lint/template CI with minimal permissions, but its action dependencies are referenced through mutable tags.
compute/eks/eks_cluster.tf Composes the cluster module and forwards networking, authentication, encryption, add-on, and Pod Identity settings.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Definition["Ravion EKS module definitions"] --> Cluster["compute/eks composite"]
  Cluster --> EKS["EKS control plane"]
  Cluster --> Nodes["System and optional node groups"]
  Cluster --> Addons["compute/eks/addons"]
  Addons --> Controllers["Karpenter, EBS CSI, LB controller, ESO"]
  Addons --> Observability["AMP, OTel, Loki, Alloy, Grafana"]
  Definition --> Service["compute/eks_service"]
  Service --> ECR["Optional ECR repository"]
  Service --> TG["ALB target group and listener rule"]
  Definition --> Charts["Web, worker, and cron Helm charts"]
  Charts --> Workloads["Kubernetes workloads"]
  Workloads --> TG
Loading
Prompt To Fix All With AI
### Issue 1
compute/eks_service/locals.tf:24
**Truncated target-group names collide**

When two load-balanced EKS workloads in the same account and region share the first 24 characters of their names, both resolve to the same target-group name, causing AWS to reject creation of the second target group because that name already exists.

### Issue 2
.github/workflows/helm-charts.yml:26-36
**Action tags remain mutable**

The new workflow references `actions/checkout`, `azure/setup-helm`, and `mikefarah/yq` through mutable tags, allowing upstream tag movement to change the code executed on the runner with access to the checked-out source and read-only repository token. Pin each action to a full commit SHA, matching the repository's existing workflows.

**How this was verified:** Every pre-existing workflow pins action references to full commit SHAs, while these three new references use tags.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "Wire EKS workload and cluster UI metrics..." | Re-trigger Greptile

Greptile also left 2 inline comments on this PR.

Context used (4)

mabadir and others added 30 commits July 20, 2026 21:27
Co-authored-by: Cursor <cursoragent@cursor.com>
…finition

Co-authored-by: Cursor <cursoragent@cursor.com>
…e/eks

Co-authored-by: Cursor <cursoragent@cursor.com>
…e/eks root

Module-level depends_on defers submodule data sources on fresh creates,
making managed policy ARNs unknown at plan time and failing
toset(var.managed_policy_arns) in security/iam. Pass the partition from a
root-level data source into the node group, addons, karpenter, and
fargate profile submodules so policy ARNs stay known during plan.

Co-authored-by: Cursor <cursoragent@cursor.com>
…pute/eks

Installs the amazon-cloudwatch-observability add-on (Container Insights) by
default with a shared Pod Identity role for both cloudwatch-agent and
fluent-bit, and surfaces control plane and Container Insights metrics plus
application/dataplane log groups in the rvn-eks definition UI.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keeps compute/eks pure AWS API so the cluster provisions in a single
apply; the Karpenter controller, CRDs, and default NodePool move to a
new compute/eks/components stack with its own rvn-eks-components
definition referencing rvn-eks. Adds a default Ravion Runner security
group so runners can reach private API endpoints.

Co-authored-by: Cursor <cursoragent@cursor.com>
Inputs hidden by show_when resolve to nil; fall back to the exact
Terraform defaults for public_access_cidrs and the Karpenter
namespace/service account per the updated config conventions.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… addons stack, gate Karpenter

Co-authored-by: Cursor <cursoragent@cursor.com>
…ation

Co-authored-by: Cursor <cursoragent@cursor.com>
…ons authenticate via get-token --role-arn

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds public/private ALB and NLB toggles composing the networking/alb and
networking/nlb modules, mirroring the rvn-ecs-cluster shared-LB pattern.
Each enabled load balancer gets an ingress rule admitting its security
group on the EKS cluster security group so it can reach pod IPs, and
listener/security-group/DNS outputs are exposed for workload modules to
attach target groups via the ALB controller's TargetGroupBinding. The
cluster module now passes public_subnet_ids through for public LBs.
…alancer

The controller is no longer a form toggle: it installs when any of the
four load balancer toggles is on, since workload target registration
(TargetGroupBinding) depends on it. lb_controller_enabled now defaults
to false and remains as an advanced-variable opt-in for Ingress-driven
provisioning without shared load balancers. Also merges the two ALB
form sections into one with public/private toggles mirroring the NLB
section, and groups all load balancer fields together after the AWS-API
add-ons.
The controller has no toggle; its chart version field moves next to
deletion protection as 'Load balancer controller chart version'.
A release left in the cluster by a deleted module instance made fresh
installs fail with 'cannot re-use a name that is still in use'. All
addons Helm releases now use upgrade --install semantics so an apply
adopts an existing same-named release instead of erroring.
# Conflicts:
#	Makefile
#	README.md
#	compute/eks/addons/modules/eks_karpenter/README.md
#	compute/eks/modules/eks_cluster/README.md
#	compute/eks/modules/eks_fargate_profile/README.md
#	compute/eks/modules/eks_node_group/README.md
rvn-eks-web, rvn-eks-worker, and rvn-eks-cron could only deploy a
pre-built image, which is the ECS build_source: image_registry path
only. They now offer Dockerfile, Railpack, or registry image, matching
the ECS service family.

Builds need somewhere to push, so compute/eks_service gains an optional
ECR repository gated on ecr_repository_creation_enabled, mirroring
compute/ecs_service. To let worker and cron share that root module, the
load balancer is now gated on a nullable listener_arn via
local.enable_load_balancer, and the target group, listener rule, and
load balancer data sources are counted off it. All load balancer
outputs return null when it is disabled.

Worker and cron were previously deploy-only. They gain a Terraform
stack that creates the ECR repository and nothing else.

Also fixes vpc_id on the workload cluster reference, which read
ref.stack.output.vpc_id and so resolved only after the cluster stack had
applied, leaving the required field empty while adding a workload. It
now reads ref.input.vpc_id, matching ECS.

Adds Terratest coverage for compute/eks_service across the four
configurations: load balancer only, ECR only, both, and repository name
override.
…role

metrics_enabled turns on a Prometheus pipeline that lives entirely in the
customer's account: an Amazon Managed Prometheus workspace (created, or brought
with amp_workspace_id), kube-state-metrics, and a single-replica OpenTelemetry
collector running the ADOT image. The collector scrapes cAdvisor and the
kubelet's resource endpoint through the API server proxy - so it works unchanged
on private-endpoint clusters - plus kube-state-metrics in-cluster, keeps a
curated ~34-family allow-list with a `keep` action in metric_relabel_configs,
and remote-writes the survivors signed with SigV4 through the sigv4auth
extension. Write identity is EKS Pod Identity scoped to aps:RemoteWrite on the
single workspace ARN.

The allow-list is the load-bearing part: AMP bills per sample, so everything
outside locals.metrics_*_allowlist is dropped before it enters collector memory.
cAdvisor's id/name/image labels are dropped, and its empty-container aggregate
rows are dropped for the CPU/memory/OOM families but kept for the network
families, which only exist on those rows.

With metrics on, the amazon-cloudwatch-observability add-on is trimmed to logs
only ({"containerInsights":{"enabled":false},"applicationSignals":{"enabled":
false},"containerLogs":{"enabled":true}}) - the two pipelines measure the same
containers, and the coarser one is the one billed per metric. An explicit
cloudwatch_observability_addon_configuration_values always wins, and a
precondition catches add-on versions older than v6.0.0, whose schema has no such
keys. The log destinations become part of the module contract:
container_log_group, dataplane_log_group, and log_stream_template - which
records that Fluent Bit writes node-first stream names, so a stream prefix
cannot scope to a workload.

grafana_role_enabled adds a read-only role trusted by grafana.amazonaws.com with
an aws:SourceAccount condition: PromQL against the workspace, Logs Insights
against the cluster's log groups. No Grafana workspace is provisioned - that
needs organization-level identity wiring.

Definition YAML gains a "Metrics (Amazon Managed Prometheus)" section and a
0.5.0 release. Tests cover the toggle matrix, allow-list composition, endpoint
derivation, the Container Insights default and its override, and IAM scoping.
Replaces the Container-Insights-logs-only trim from 905cd2f. logs_enabled now
installs Grafana Alloy as a DaemonSet reading every container's stdout off its
own node, and Loki in single-binary mode indexing it with every chunk stored in
an S3 bucket in the customer's account. cloudwatch_observability_enabled flips
to false and stays as a legacy toggle: with Loki for logs and AMP for metrics it
duplicates both halves at CloudWatch prices.

Loki is never exposed — no ingress, no load balancer, not even the chart's nginx
gateway. Ravion reads it by asking Beacon to proxy a query over the WebSocket it
already holds, which is why loki_endpoint is an in-cluster Service URL and why
there is no certificate, no auth layer and no inbound path to open.

The label set is the contract the dashboard's LogQL layer is written against:
namespace, app and workload as labels, level as structured metadata, and
deliberately nothing per-pod — a pod-name label turns one stream per workload
into one per replica per restart. Alloy also drops the filename label
loki.source.file adds, because the path contains the pod UID and would smuggle
the same cardinality back in through the side door.

Retention is enforced twice on purpose. Loki's compactor is the authority
(retention_enabled is off in stock Loki, which is why a Loki bucket usually
grows forever, and why the IAM role carries s3:DeleteObject). The bucket's
lifecycle rule expires a week later, so it never races the compactor into
deleting an index still being read.

Two things found while rendering the charts rather than reading them: with
persistence off the loki chart mounts nothing at /var/loki and the container has
a read-only root filesystem, so Loki cannot write its WAL at all — an emptyDir
now fills that gap; and grafana/grafana on grafana.github.io is deprecated as of
January 2026, so the in-cluster Grafana pulls the maintained chart from
grafana-community instead.

grafana_enabled adds that Grafana, preprovisioned with AMP over SigV4 (its own
Pod Identity role, plus the grafana.ini flag without which SigV4 silently fails)
and Loki over in-cluster HTTP. grafana_role_enabled stays for Amazon Managed
Grafana, which can query AMP but cannot reach a ClusterIP Service.

Beacon's proxy allowlist is wired from this module but the chart has no value
key for it yet — see the PENDING BEACON CHART CONTRACT comment in loki.tf.
tests/logs.tftest.hcl adds the toggle matrix for the logs half: logs off renders
no bucket, no releases and no IAM; a bring-your-own bucket suppresses creation
while every derived value still resolves; retention reaches both the compactor
and the bucket rule; Loki really is single-binary with the caches, gateway and
MinIO off and no static credentials; and the Alloy label contract is asserted
name by name, including the absence of a pod label — that assertion is the point
of the file, because breaking it breaks the dashboard silently and expensively.

tests/metrics.tftest.hcl drops the Container Insights runs and instead asserts
the add-on is off by default.

README gains the Loki/Alloy/S3 and Grafana sections — the label contract, the
two-place retention, why persistence is opt-in, and the split between in-cluster
Grafana and Amazon Managed Grafana — plus updated inputs and outputs tables.
The definition YAML gains Logs and Grafana sections, drops Container Insights to
off, and bumps to 0.6.0.
The dashboard's Prometheus read path had no producer: no definition
declared a prometheus metric source, worker and cron declared no metrics
at all, and the cluster module's charts pointed at ContainerInsights,
which the add-ons module now defaults off.

- rvn-eks-web/worker/cron 0.3.0: golden-signal pod charts (CPU, memory,
  restarts, network, plus replicas or run outcomes) from the add-ons
  module's AMP workspace, and a Loki log source served through Beacon.
  Worker and cron gain an optional EKS Add-ons reference; web reuses its
  existing one with two new AMP mapped inputs.
- rvn-eks-addons 0.7.0: cluster-health charts (node CPU/memory,
  readiness, pod counts) from its own AMP outputs; README now states the
  real Container Insights default.
- rvn-eks 0.2.0: drop the dead ContainerInsights charts, keep the
  AWS/EKS control-plane ones.

# ELBv2 target group names are capped at 32 characters and the "-tg" suffix
# takes 3, mirroring the ECS service module's truncation.
target_group_name = "${substr(var.name, 0, min(length(var.name), 24))}-tg"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Truncated target-group names collide

When two load-balanced EKS workloads in the same account and region share the first 24 characters of their names, both resolve to the same target-group name, causing AWS to reject creation of the second target group because that name already exists.

Knowledge Base Used: Networking modules: VPC, load balancers, DNS, security groups, EIPs

Prompt To Fix With AI
This is a comment left during a code review.
Path: compute/eks_service/locals.tf
Line: 24

Comment:
**Truncated target-group names collide**

When two load-balanced EKS workloads in the same account and region share the first 24 characters of their names, both resolve to the same target-group name, causing AWS to reject creation of the second target group because that name already exists.

**Knowledge Base Used:** [Networking modules: VPC, load balancers, DNS, security groups, EIPs](https://app.greptile.com/flightcontrol/-/custom-context/knowledge-base/ravionhq/modules/-/docs/networking.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment on lines +26 to +36
uses: actions/checkout@v4

- name: Setup Helm
uses: azure/setup-helm@v4
with:
version: v3.16.3

# yq is preinstalled on ubuntu-latest runners; pinned here so the
# assertion syntax in charts/test.sh cannot drift underneath us.
- name: Setup yq
uses: mikefarah/yq@v4.44.3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 security Action tags remain mutable

The new workflow references actions/checkout, azure/setup-helm, and mikefarah/yq through mutable tags, allowing upstream tag movement to change the code executed on the runner with access to the checked-out source and read-only repository token. Pin each action to a full commit SHA, matching the repository's existing workflows.

How this was verified: Every pre-existing workflow pins action references to full commit SHAs, while these three new references use tags.

Prompt To Fix With AI
This is a comment left during a code review.
Path: .github/workflows/helm-charts.yml
Line: 26-36

Comment:
**Action tags remain mutable**

The new workflow references `actions/checkout`, `azure/setup-helm`, and `mikefarah/yq` through mutable tags, allowing upstream tag movement to change the code executed on the runner with access to the checked-out source and read-only repository token. Pin each action to a full commit SHA, matching the repository's existing workflows.

**How this was verified:** Every pre-existing workflow pins action references to full commit SHAs, while these three new references use tags.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant