Skip to content

Commit 3a21efd

Browse files
h4x3rotabclaude
h4x3rotab
and
claude
committed
add k3s-in-dstack tutorial
Single-node k3s cluster running inside a dstack CVM with: - Wildcard HTTPS via dstack-ingress (Let's Encrypt DNS-01) - Remote kubectl access via TLS-passthrough gateway - TEE attestation evidence served at /evidences/ - Optional RBAC and network policy hardening Verified on Phala Cloud with dstacktee/dstack-ingress:2.1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 18eacc9 commit 3a21efd

5 files changed

Lines changed: 464 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010
**Example applications for [dstack](https://github.com/Dstack-TEE/dstack) - Deploy containerized apps to TEEs with end-to-end security in minutes**
1111

12-
[Getting Started](#getting-started) • [Use Cases](#use-cases) • [Core Patterns](#core-patterns) • [Dev Tools](#dev-scaffolding) • [Starter Packs](#starter-packs) • [Other Use Cases](#other-use-cases)
12+
[Getting Started](#getting-started) • [Use Cases](#use-cases) • [Core Patterns](#core-patterns) • [Infrastructure](#infrastructure) • [Dev Tools](#dev-scaffolding) • [Starter Packs](#starter-packs) • [Other Use Cases](#other-use-cases)
1313

1414
</div>
1515

@@ -152,6 +152,16 @@ Development and debugging tools. **Not for production.**
152152

153153
---
154154

155+
## Infrastructure
156+
157+
Run infrastructure services inside TEEs.
158+
159+
| Example | Description |
160+
|---------|-------------|
161+
| [k3s](./k3s) | Single-node k3s cluster in a TEE with wildcard HTTPS and remote kubectl |
162+
163+
---
164+
155165
## Tech Demos
156166

157167
Interesting demonstrations.

‎k3s/README.md‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
# k3s on dstack
2+
3+
Run a single-node Kubernetes cluster inside an Intel TDX Confidential VM with wildcard HTTPS for all your services.
4+
5+
## What You Get
6+
7+
- k3s cluster running in a hardware-isolated TEE
8+
- Wildcard TLS certificate (Let's Encrypt) so every service gets HTTPS automatically
9+
- Remote `kubectl` access through the dstack gateway
10+
- Traefik ingress controller for routing HTTP traffic to pods
11+
12+
## Prerequisites
13+
14+
- [Phala Cloud](https://cloud.phala.network) account (or a self-hosted dstack deployment — see [Running on Raw dstack](#running-on-raw-dstack))
15+
- Phala CLI installed and authenticated:
16+
```bash
17+
npm install -g phala
18+
phala auth login
19+
```
20+
- `kubectl` installed ([install guide](https://kubernetes.io/docs/tasks/tools/))
21+
- A domain you control (for the wildcard certificate)
22+
- Cloudflare API token with **Zone:Read** and **DNS:Edit** permissions (see [DNS_PROVIDERS.md](../custom-domain/dstack-ingress/DNS_PROVIDERS.md) for other DNS providers)
23+
24+
## Quick Start
25+
26+
### 1. Deploy the CVM
27+
28+
```bash
29+
phala deploy \
30+
-n my-k3s \
31+
-c docker-compose.yaml \
32+
-t tdx.medium \
33+
--disk-size 50G \
34+
--dev-os \
35+
-e "CLOUDFLARE_API_TOKEN=your-cloudflare-token" \
36+
-e "CERTBOT_EMAIL=you@example.com" \
37+
-e "CLUSTER_DOMAIN=k3s.example.com" \
38+
--wait
39+
```
40+
41+
Replace `k3s.example.com` with your actual domain. All subdomains under it (e.g., `nginx.k3s.example.com`) will get TLS automatically.
42+
43+
The `--dev-os` flag enables SSH access (needed to extract the kubeconfig). The `--disk-size 50G` gives enough room for k3s images and workloads.
44+
45+
The deploy command outputs an **App ID** and gateway info. Save the App ID (a 40-character hex string):
46+
47+
```
48+
App ID: a1b2c3d4e5f6...
49+
```
50+
51+
You can also retrieve these later:
52+
53+
```bash
54+
phala cvms get my-k3s --json
55+
```
56+
57+
### 2. Get Your Kubeconfig
58+
59+
Wait 3-4 minutes for the CVM to boot, then extract the kubeconfig:
60+
61+
```bash
62+
APP_ID=<your-app-id>
63+
GATEWAY_DOMAIN=<your-gateway-domain> # e.g. dstack-pha-prod5.phala.network
64+
65+
# Find the gateway domain from the CVM info if you don't have it
66+
# phala cvms get my-k3s --json | jq -r '.gateway.base_domain'
67+
68+
# Extract kubeconfig from the CVM
69+
phala ssh "$APP_ID" -- \
70+
"docker exec dstack-k3s-1 cat /etc/rancher/k3s/k3s.yaml" \
71+
2>/dev/null > k3s.yaml
72+
73+
# Rewrite the API server URL to use the gateway TLS passthrough endpoint
74+
sed -i "s|https://127.0.0.1:6443|https://${APP_ID}-6443s.${GATEWAY_DOMAIN}|" k3s.yaml
75+
76+
export KUBECONFIG=$(pwd)/k3s.yaml
77+
```
78+
79+
> **Note:** The `-6443s` suffix tells the dstack gateway to use TLS passthrough (the `s` means passthrough). This way `kubectl` talks directly to the k3s API server's TLS — the gateway never sees the traffic contents.
80+
81+
### 3. Verify the Cluster
82+
83+
```bash
84+
kubectl get nodes
85+
```
86+
87+
Wait until the node shows `Ready` (1-2 minutes after SSH becomes available):
88+
89+
```
90+
NAME STATUS ROLES AGE VERSION
91+
k3s-node Ready control-plane,master 2m v1.31.6+k3s1
92+
```
93+
94+
### 4. Wait for the Wildcard Certificate
95+
96+
The dstack-ingress container issues a Let's Encrypt wildcard certificate via DNS-01 challenge. This takes 5-8 minutes after CVM boot (certbot installation + 120s DNS propagation + issuance). If your domain has existing CAA records, the first attempt may fail and auto-retry after setting the correct `issuewild` CAA record.
97+
98+
Check with:
99+
100+
```bash
101+
curl -sI "https://test.k3s.example.com/" 2>&1 | head -3
102+
```
103+
104+
Retry until you see an HTTP response (a 404 is fine — it means TLS works but no service is routed yet):
105+
106+
```
107+
HTTP/1.1 404 Not Found
108+
```
109+
110+
### 5. Deploy a Test Workload
111+
112+
```bash
113+
CLUSTER_DOMAIN=k3s.example.com
114+
115+
# Create an nginx pod and service
116+
kubectl run nginx --image=nginx:alpine --port=80
117+
kubectl expose pod nginx --port=80 --target-port=80 --name=nginx
118+
kubectl wait --for=condition=Ready pod/nginx --timeout=120s
119+
120+
# Create a Traefik IngressRoute to route traffic to nginx
121+
kubectl apply -f - <<EOF
122+
apiVersion: traefik.io/v1alpha1
123+
kind: IngressRoute
124+
metadata:
125+
name: nginx
126+
spec:
127+
entryPoints: [web]
128+
routes:
129+
- match: Host(\`nginx.${CLUSTER_DOMAIN}\`)
130+
kind: Rule
131+
services:
132+
- name: nginx
133+
port: 80
134+
EOF
135+
136+
# Wait for route propagation, then test
137+
sleep 10
138+
curl -s "https://nginx.${CLUSTER_DOMAIN}/"
139+
```
140+
141+
You should see the nginx welcome page served over HTTPS with a valid Let's Encrypt certificate.
142+
143+
### 6. Clean Up
144+
145+
Remove the test workload:
146+
147+
```bash
148+
kubectl delete ingressroute.traefik.io nginx
149+
kubectl delete svc nginx
150+
kubectl delete pod nginx
151+
```
152+
153+
Delete the CVM entirely:
154+
155+
```bash
156+
echo y | phala cvms delete my-k3s
157+
rm k3s.yaml
158+
```
159+
160+
## How It Works
161+
162+
### Architecture
163+
164+
```mermaid
165+
graph LR
166+
subgraph Internet
167+
User[Browser / curl]
168+
Kubectl[kubectl]
169+
end
170+
171+
subgraph "dstack CVM (Intel TDX)"
172+
Ingress[dstack-ingress<br/>TLS termination]
173+
Traefik[Traefik<br/>HTTP routing]
174+
K3sAPI[k3s API server<br/>port 6443]
175+
Pod1[Pod A]
176+
Pod2[Pod B]
177+
end
178+
179+
User -->|HTTPS| Ingress
180+
Ingress -->|HTTP| Traefik
181+
Traefik --> Pod1
182+
Traefik --> Pod2
183+
184+
Kubectl -->|TLS passthrough<br/>via gateway| K3sAPI
185+
```
186+
187+
External HTTPS traffic hits dstack-ingress, which terminates TLS using the wildcard certificate and forwards plain HTTP to Traefik. Traefik routes requests to pods based on `Host` header matching via IngressRoutes.
188+
189+
`kubectl` connects through the dstack gateway's TLS passthrough mode (port suffix `-6443s`), so the gateway forwards encrypted traffic directly to the k3s API server without inspecting it.
190+
191+
### Services
192+
193+
| Service | Purpose |
194+
|---------|---------|
195+
| `kmod-installer` | Loads kernel modules required by k3s networking (runs once at boot) |
196+
| `k3s` | Single-node k3s server running in privileged mode |
197+
| `dstack-ingress` | Wildcard TLS termination via Let's Encrypt DNS-01, proxies to Traefik |
198+
199+
### How Wildcard HTTPS Works
200+
201+
1. dstack-ingress requests a wildcard certificate for `*.k3s.example.com` from Let's Encrypt using DNS-01 validation
202+
2. It creates a DNS TXT record via the Cloudflare API to prove domain ownership
203+
3. After certificate issuance, all HTTPS traffic to `*.k3s.example.com` is terminated by dstack-ingress
204+
4. The decrypted HTTP traffic is forwarded to Traefik on port 80
205+
5. Traefik matches the `Host` header against IngressRoute rules and routes to the right pod
206+
207+
## Configuration
208+
209+
### Environment Variables
210+
211+
| Variable | Required | Description |
212+
|----------|----------|-------------|
213+
| `CLOUDFLARE_API_TOKEN` | Yes | Cloudflare API token for DNS-01 certificate challenges |
214+
| `CERTBOT_EMAIL` | Yes | Email for Let's Encrypt registration |
215+
| `CLUSTER_DOMAIN` | Yes | Your domain for the wildcard cert (e.g., `k3s.example.com`) |
216+
| `K3S_NODE_NAME` | No | Kubernetes node name (default: `k3s-node`) |
217+
218+
The following are auto-injected by Phala Cloud and used in the compose file:
219+
220+
| Variable | Description |
221+
|----------|-------------|
222+
| `DSTACK_APP_ID` | CVM application ID (used for k3s TLS SAN) |
223+
| `DSTACK_GATEWAY_DOMAIN` | Gateway domain (used for k3s TLS SAN and ingress routing) |
224+
225+
### Using a Different DNS Provider
226+
227+
The compose file defaults to Cloudflare. To use a different provider, change `DNS_PROVIDER` and the corresponding credentials. See [DNS_PROVIDERS.md](../custom-domain/dstack-ingress/DNS_PROVIDERS.md) for supported providers (Linode, Namecheap, Route53).
228+
229+
### Instance Sizing
230+
231+
| Instance Type | Recommended For |
232+
|---------------|-----------------|
233+
| `tdx.medium` | Testing and tutorials |
234+
| `tdx.4xlarge` | Small production workloads (5-10 pods) |
235+
| `tdx.8xlarge` | Larger workloads |
236+
237+
k3s itself uses ~500MB RAM. Budget the rest for your workloads. A 50GB disk is recommended minimum.
238+
239+
## Hardening (Optional)
240+
241+
### Scoped RBAC for Programmatic Access
242+
243+
The kubeconfig from step 2 uses the built-in `cluster-admin` credentials. For programmatic access with limited permissions, create a scoped service account:
244+
245+
```bash
246+
kubectl apply -f manifests/rbac.yaml
247+
kubectl create token k3s-admin --duration=8760h
248+
```
249+
250+
Use the resulting token with the API server URL from your kubeconfig.
251+
252+
### Network Policies
253+
254+
Restrict pod-to-pod traffic so pods can only receive traffic from Traefik and reach the internet (but not each other):
255+
256+
```bash
257+
kubectl apply -f manifests/network-policy.yaml
258+
```
259+
260+
This applies three policies:
261+
- **default-deny**: blocks all ingress and egress by default
262+
- **allow-internet-egress**: allows outbound internet (but blocks pod-to-pod via CIDR exclusions) and DNS
263+
- **allow-traefik-ingress**: allows inbound traffic only from Traefik in kube-system
264+
265+
## Running on Raw dstack
266+
267+
This tutorial targets Phala Cloud, but the same compose file works on a self-hosted dstack deployment. Key differences:
268+
269+
- `DSTACK_APP_ID` and `DSTACK_GATEWAY_DOMAIN` are not auto-injected. Add a `K3S_TLS_SAN` environment variable to the k3s service manually with your gateway endpoint, and update the `--tls-san` argument.
270+
- Deploy via the VMM web UI (port 9080) or `vmm-cli.py` instead of the Phala CLI.
271+
- See the [dstack deployment guide](https://github.com/Dstack-TEE/dstack/blob/main/docs/deployment.md) for self-hosted setup.
272+
273+
## Timing Reference
274+
275+
| Phase | Duration |
276+
|-------|----------|
277+
| CVM provision | ~1s |
278+
| SSH available | ~2-3 min |
279+
| k3s node Ready | ~1 min after SSH |
280+
| Wildcard cert issued | ~5-8 min (certbot install + DNS propagation) |
281+
| **Total to first HTTPS 200** | **~8-10 min** |
282+
283+
Measured on `tdx.medium` with 50GB disk. The wildcard cert is the bottleneck — certbot installs its dependencies on first boot, then waits 120s for DNS propagation. If your domain has existing CAA records, add another ~3 min for the auto-retry.
284+
285+
## Troubleshooting
286+
287+
**kubectl connection refused**
288+
The k3s API server may not be ready yet. Wait 3-5 minutes after deploy, then retry. Check that the `--tls-san` value matches your gateway endpoint.
289+
290+
**Wildcard cert not issuing**
291+
Check dstack-ingress logs:
292+
```bash
293+
phala ssh <app-id> -- "docker logs dstack-dstack-ingress-1 2>&1 | tail -30"
294+
```
295+
Common causes: wrong Cloudflare token, domain not on your Cloudflare account, DNS propagation delay, existing CAA records (auto-retried).
296+
297+
**IngressRoute returns 404**
298+
Traefik may not have picked up the route yet. Wait 10-15 seconds and retry. Verify the IngressRoute exists:
299+
```bash
300+
kubectl get ingressroute.traefik.io
301+
```
302+
303+
**Node stuck in NotReady**
304+
The kmod-installer may have failed. Check k3s logs:
305+
```bash
306+
phala ssh <app-id> -- "docker logs dstack-k3s-1 2>&1 | tail -30"
307+
```
308+
309+
## Files
310+
311+
```
312+
k3s/
313+
├── docker-compose.yaml # k3s + kmod-installer + dstack-ingress
314+
├── README.md
315+
└── manifests/
316+
├── rbac.yaml # Optional: scoped service account
317+
└── network-policy.yaml # Optional: restrict pod traffic
318+
```

0 commit comments

Comments
 (0)