[WIP] Set up and test the streaming camera

.github/workflows/build-deployment-zip.yaml

@@ -0,0 +1,68 @@
+name: Build Lambda Deployment Package
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write  # Required to upload assets to releases
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Create dependencies directory
+        run: mkdir -p dependencies
+
+      - name: Install dependencies
+        # Pin the target platform to AWS Lambda's runtime (x86_64/manylinux,
+        # CPython 3.12) so binary wheels such as cryptography's _rust.abi3.so
+        # match Lambda regardless of the build host's architecture. Keep
+        # --python-version in sync with the function's configured runtime.
+        run: |
+          pip install \
+            --platform manylinux2014_x86_64 \
+            --implementation cp \
+            --python-version 3.12 \
+            --only-binary=:all: \
+            --target ./dependencies \
+            boto3 \
+            google-api-python-client \
+            google-auth \
+            google-auth-oauthlib \
+            google-auth-httplib2
+
+      - name: Copy lambda function and chalicelib
+        run: |
+          cp lambda_function.py dependencies/
+          cp -r chalicelib dependencies/
+
+      - name: Create deployment package
+        run: |
+          cd dependencies
+          zip -r ../deployment.zip .
+          cd ..
+
+      - name: Upload deployment package as artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: lambda-deployment-package
+          path: deployment.zip
+          retention-days: 90
+
+      - name: Upload deployment package to release
+        if: github.event_name == 'release'
+        uses: softprops/action-gh-release@v1
+        with:
+          files: deployment.zip
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -19,3 +19,7 @@ __pycache__/
 *.bak
 
 policy.json
+
+# Deployment artifacts
+dependencies/
+deployment.zip

CLAUDE.md

@@ -0,0 +1,38 @@
+# CLAUDE.md
+
+## Tailscale → Raspberry Pi connection
+
+**You are already on the tailnet.** The workflow joins the runner via the official
+[Tailscale GitHub Action](https://tailscale.com/kb/1276/tailscale-github-action) (OAuth
+client + device tag) before you start. Run `tailscale status` to confirm — do **not**
+install Tailscale, mint auth keys via the API, or run `tailscale up` unless status
+genuinely shows you disconnected. Access to the Pi is
+[Tailscale SSH](https://tailscale.com/kb/1193/tailscale-ssh), authorized by
+[tailnet ACLs](https://tailscale.com/kb/1018/acls) rather than SSH keys — there is no key
+to find or generate. The Pi's login username, hostname, and sudo password are injected as
+environment variables (check `env` for the `RPI_*`-style names rather than assuming them);
+always reference them as `"$VAR"` and never print the hostname or any credential in
+comments, commits, or logs. If SSH is refused (`tailnet policy does not permit you to SSH
+to this node`), the fix is an ACL/tag change only the tailnet admin can make — report it
+and stop rather than working around it.
+
+**sudo on the Pi is password-gated** — no passwordless sudo, and polkit rejects
+non-interactive `systemctl`. Feed the password over stdin so it never appears in a process
+list or shell history: `ssh … "sudo -S -p '' <cmd>" <<< "$RPI_PASSWORD_VAR"`.
+
+**You have two machines — use the right one.** Your runner terminal and the Pi are
+separate environments: cloud/API credentials (AWS, etc.) live on the **runner**, so run
+that tooling there rather than routing it through the Pi. Use the Pi only for what
+genuinely requires it — its attached hardware, or its residential IP (some services block
+datacenter IPs). The Pi is typically on constrained residential Wi‑Fi and may be carrying
+live workloads, so rate-cap any large transfer (`--limit-rate` or equivalent) and never
+run full-bandwidth speed tests on it.
+
+**Treat the Pi as a live production device.** Inspect read-only first (`systemctl status`,
+`journalctl`, `crontab -l` as root) before changing state: scheduled reboots, watchdog
+timers, and `Restart=` policies may already exist, so an unreachable or restarting device
+may be behaving as designed — check the clock and the existing automation before declaring
+an outage or adding new monitoring. Restart services only when necessary and verify the
+device's workload is healthy end-to-end afterwards, reporting failures as failures.
+Changes made on the Pi (systemd units, cron, scripts, config) do not live in this repo —
+record them in the repo's docs so they can be reproduced or upstreamed.

build-deployment-zip.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+# Build script for creating AWS Lambda deployment package
+# This script creates a deployment.zip file that can be uploaded directly to AWS Lambda
+
+set -e
+
+echo "Building AWS Lambda deployment package..."
+
+# Clean up any existing artifacts
+echo "Cleaning up existing artifacts..."
+rm -rf dependencies deployment.zip
+
+# Create dependencies directory
+echo "Creating dependencies directory..."
+mkdir -p dependencies
+
+# Install Python dependencies
+# NOTE: AWS Lambda runs on x86_64/manylinux with CPython. Pin the target
+# platform so binary wheels (e.g. cryptography's compiled _rust.abi3.so, a
+# transitive dependency of google-auth) match the Lambda runtime no matter
+# where this script runs. Building unpinned on an ARM host (such as the
+# Raspberry Pi) produces ARM binaries that fail to load on Lambda with
+# Runtime.ImportModuleError (502). Keep --python-version in sync with the
+# function's configured runtime (Python 3.12).
+echo "Installing Python dependencies..."
+pip install \
+  --platform manylinux2014_x86_64 \
+  --implementation cp \
+  --python-version 3.12 \
+  --only-binary=:all: \
+  --target ./dependencies \
+  boto3 \
+  google-api-python-client \
+  google-auth \
+  google-auth-oauthlib \
+  google-auth-httplib2
+
+# Copy lambda function and chalicelib
+echo "Copying lambda function and chalicelib..."
+cp lambda_function.py dependencies/
+cp -r chalicelib dependencies/
+
+# Create deployment package
+echo "Creating deployment.zip..."
+cd dependencies
+zip -q -r ../deployment.zip .
+cd ..
+
+# Get the size of the deployment package
+SIZE=$(du -h deployment.zip | cut -f1)
+echo "✓ deployment.zip created successfully (${SIZE})"
+echo ""
+echo "You can now upload deployment.zip to AWS Lambda!"
+echo "See README.md for detailed deployment instructions."

chalicelib/ytb_api_utils.py

@@ -1,3 +1,4 @@
+import os
 import boto3
 import pickle
 import time
@@ -9,8 +10,8 @@
 CHANNEL_ID = "UCHBzCfYpGwoqygH9YNh9A6g"
 
 YOUTUBE = None
-S3_BUCKET = "ac-token-youtube-api"
-S3_KEY = "token/token.pickle"
+S3_BUCKET = os.environ.get("YOUTUBE_TOKEN_S3_BUCKET", "ac-token-youtube-api")
+S3_KEY = os.environ.get("YOUTUBE_TOKEN_S3_KEY", "token/token.pickle")
 
 def init_youtube_service():
     global YOUTUBE
@@ -99,9 +100,9 @@ def create_broadcast_and_bind_stream(cam_name: str,workflow_name: str, privacy_s
     formatted_time = datetime.utcnow().strftime("%Y-%m-%d UTC %H:%M")
     broadcast_title = f"{workflow_name} stream {cam_name}, {formatted_time}"
     broadcast_description = (
-        f"Live camera feed from {workflow_name} stationed in Toronto, ON "
-        "at the Acceleration Consortium (AC).\n\n"
-        "https://acceleration.utoronto.ca/"
+        f"Live camera feed from {workflow_name} stationed at "
+        "the Vertical Cloud Lab @ BYU\n\n"
+        "https://github.com/vertical-cloud-lab"
     )
 
     broadcast_response = YOUTUBE.liveBroadcasts().insert(

docs/ac-dev-lab-archive/README.md

@@ -0,0 +1,70 @@
+# ac-dev-lab download/process/re-upload discussion archive
+
+Complete comment archives (fetched via the GitHub API on 2026-07-08, including
+comments hidden in the web UI by GitHub's pagination) from the
+AccelerationConsortium/ac-dev-lab issues and PRs that trace the design of the
+YouTube livestream **download → post-process → re-upload** pipeline. Archived
+to give agents working in this repo the full context without re-fetching.
+
+| File | Item | State | What it covers |
+|---|---|---|---|
+| `issue-212-*.md` | [#212](https://github.com/AccelerationConsortium/ac-dev-lab/issues/212) | closed | First exploration of downloading self-owned videos; settled on yt-dlp over youtube-dl |
+| `pr-234-*.md` | [#234](https://github.com/AccelerationConsortium/ac-dev-lab/pull/234) | merged | `yt_utils.py`: yt-dlp download of the latest video (public videos, no auth) |
+| `issue-223-*.md` | [#223](https://github.com/AccelerationConsortium/ac-dev-lab/issues/223) | open | The processing pipeline: auto-editor stale-section detection, ffmpeg speed-up overlay, 16x speedup, title convention, yt-dlp cookie problems on HF Spaces |
+| `issue-231-*.md` | [#231](https://github.com/AccelerationConsortium/ac-dev-lab/issues/231) | open | 8-hr restart mechanism history (origin of this streamingLambda repo): Lambda holds token.pickle, crontab reboot `0 5,13,21`, one-time stream keys |
+| `issue-341-*.md` | [#341](https://github.com/AccelerationConsortium/ac-dev-lab/issues/341) | closed | Kickoff for Playwright-based downloads of private videos |
+| `pr-343-*.md` | [#343](https://github.com/AccelerationConsortium/ac-dev-lab/pull/343) | merged | **The adopted downloader**: YouTube Data API discovery + Playwright Studio download with TOTP login |
+
+## Key findings
+
+### Download (the hard part — YouTube has no API to download video content)
+
+- **yt-dlp** (PR #234) works fine for public/unlisted videos with no auth, and
+  is the preferred tool on headless/ephemeral machines. For **private** videos
+  it needs browser cookies, which expire within hours
+  ([yt-dlp#8227](https://github.com/yt-dlp/yt-dlp/issues/8227)) — a dead end
+  for unattended automation on HF Spaces (issue #223).
+- **Playwright + YouTube Studio** (PR #343, the merged approach in
+  `src/ac_training_lab/video_editing/download.py`): a dedicated Google account
+  (`achardwarestreams.downloader@...`) is a **channel editor** (viewer role has
+  the download button disabled). The script:
+  1. lists playlists/videos via the YouTube Data API and filters out
+     already-downloaded/processed ones,
+  2. logs into Google with email + password + **pyotp TOTP** (2FA re-enabled
+     deliberately: Google blocks password-only logins from ephemeral machines
+     with "couldn't verify this account belongs to you", but accepts
+     password+TOTP),
+  3. navigates to `studio.youtube.com/video/{video_id}/edit` and clicks the
+     ⋮ menu → Download.
+- Playwright **headless fails** (Google bot detection) — a virtual framebuffer
+  (xvfb) is required. On the BALAM cluster it runs inside an **apptainer**
+  container (system deps not installable), and only login nodes have internet.
+- HF Spaces cannot run Playwright at all (no GUI browser env; Docker attempt
+  stalled) — reproducer: `huggingface.co/spaces/AccelerationConsortium/playwright-reproducer`.
+- Alternative acknowledged in the threads: make videos public/unlisted so
+  yt-dlp works without auth (sgbaird encourages public/unlisted "in part for
+  this reason").
+
+### Processing (issue #223)
+
+- **auto-editor** pipeline by @Jonathan-Woo:
+  1. `auto-editor --edit motion:threshold=…` produces v1 timestamps of stale
+     (no-motion) sections,
+  2. ffmpeg burns a speed-up indicator overlay (e.g. `16x`) on stale sections,
+  3. auto-editor speeds up stale sections (`--silent-speed`), keeping motion at 1x.
+- Tuning knobs: `threshold` (fraction of changed pixels to count as motion) and
+  `margin` (padding around kept sections). False positives are a concern, so
+  stale sections are sped up rather than deleted (a huge `--silent-speed`
+  effectively deletes).
+- Title convention so originals and processed videos can be matched via the
+  Data API: keep the original title/ID and append **`[processed, 16x]`**.
+- Later development moved to
+  [AccelerationConsortium/youtube-livestream-processor](https://github.com/AccelerationConsortium/youtube-livestream-processor).
+
+### Upload / orchestration
+
+- Uploads and playlist management use the YouTube Data API with the channel
+  owner's **token.pickle** (same credential this repo's Lambda uses from S3).
+- Issue #231 documents why broadcasts are chunked by reboot (`0 5,13,21 * * *`
+  crontab) with the Lambda doing `end` → `create` — YouTube autostart is
+  unreliable if a broadcast is created while data is already flowing.

docs/ac-dev-lab-archive/issue-212-download-self-owned-youtube-videos.md

@@ -0,0 +1,36 @@
+# Issue #212: Download self-owned YouTube videos
+
+- **URL:** https://github.com/AccelerationConsortium/ac-dev-lab/issues/212
+- **Author:** @sgbaird
+- **State:** closed
+- **Created:** 2025-03-27T16:47:25Z  **Closed:** 2025-05-01T23:21:05Z
+- **Comments archived:** 2 issue comments
+
+---
+
+## Original description
+
+In creator dashboard, there's a way to download it via GUI. Some possibilities: 
+
+- use browser cookies: https://stackoverflow.com/a/55272225
+- use youtube-dl: https://github.com/ytdl-org/youtube-dl
+
+Terms of service: https://www.youtube.com/t/terms
+
+Cc @Jonathan-Woo 
+
+---
+
+## Comments (complete, in chronological order)
+
+### Comment 1 — @sgbaird at 2025-04-08T14:18:33Z
+
+@Neil-YL if you end up getting stuck on the new picam device.py script, could you give this one a try? Ideally, it would be great to have the full restart + download workflow tested to show that we can store and retrieve 24/7 streams.
+
+---
+
+### Comment 2 — @Jonathan-Woo at 2025-05-01T15:49:48Z
+
+Switched to using yt-dlp instead of youtube-dl because it's more actively maintained.
+
+---