fix(sidekiq): repair the worker boot, make the deploy check real, date usage from the request

[ndemianc]

Root cause of the frozen usage ledger, found in the EB log bundle. Sidekiq had not run since 2026-07-22 23:25 UTC (~27h), with 141 jobs queued and 8 dead, while the site served normally.

1. The boot failure

start-sidekiq.sh hardcoded vendor/bundle/ruby/3.4.0/bin. The platform moved to Ruby 4.0, that directory stopped existing, and Sidekiq resolved through system gems where the pinned bundler is absent:

Could not find 'bundler' (~> 2.5) - did find: [bundler-4.0.10]

sidekiq.log contains nothing else, on repeat — Restart=always retried forever. Puma was unaffected because EB launches it through the platform's own Ruby setup rather than this wrapper, which is precisely why the outage was invisible from the outside.

Now resolves the vendored bundle for whatever ABI is present, installs the Gemfile.lock-pinned bundler on demand (--user-install, no privileges needed), and uses bundle exec instead of the binstub that depends on that pin.

2. Why it hid for a day

99_verify_sidekiq.sh only printed diagnostics. Every check ended in || echo, and the file ended on a succeeding ps, so it returned 0 regardless. It reported SUCCESS on every deploy while the worker was dead — including 01:18 on 24 Jul.

Verified against the old file, with Sidekiq absent:

OLD hook -> exit=0     NEW hook -> exit=1

It now polls up to 45s, requires the unit active and a live worker process, re-checks after 5s so a Restart=always crash-loop can't pass mid-bounce, and fails the deploy otherwise.

A check that cannot fail is not a check.

3. Queue lag rewrote history

usage_events has only created_at, the job never set it, and the enqueue passed no timestamp — so a late job is dated from when it was processed. Draining the 141-job backlog would have stamped all of them with the recovery moment: a false spike on one day, a permanent hole on the days the work happened.

Both enqueue sites now pass an occurred-at epoch and the job dates the row from it. The new argument is last and optional so the 141 jobs already serialized with the old arity keep working when this deploys — pinned by a spec.

Verification

973 examples, 0 failures. The dating fix is mutation-checked (reverting to insert-time dating fails). The ai_spec assertion checks the timestamp is a plausible now rather than kind_of(Integer) — a wrong integer would satisfy a type check while misdating every row.

Not verified on the instance — I have no access. The launcher change is reasoned from the log bundle (GEM_PATH=…/ruby/4.0.0 vs the hardcoded 3.4.0). Fix 2 is what makes that safe: if the launcher is still wrong, the next deploy fails loudly instead of pretending.

After deploying

• The 141 queued jobs will still be mis-dated — their payloads predate this change and carry no timestamp. Unavoidable.
• The 8 dead jobs are lost usage events (unbilled spend); worth inspecting before clearing, as they may name the original trigger.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR addresses a production incident where Sidekiq silently stopped running after an Elastic Beanstalk Ruby ABI upgrade, deploys incorrectly reported success, and queued usage jobs would have been mis-dated when eventually processed.

Changes:

• Make Sidekiq startup resilient to Ruby ABI upgrades and bundler-version pinning.
• Make the EB postdeploy Sidekiq verifier actually fail the deploy when the worker isn’t truly running.
• Thread an “occurred-at” timestamp from request time into RecordUsageJob so ledger rows are dated from the request, not processing time (with specs pinning backward compatibility).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
spec/requests/api/levelcode/v1/ai_spec.rb	Tightens the enqueue assertion to ensure an occurred-at epoch close to request time is passed.
spec/jobs/record_usage_job_spec.rb	Adds coverage for backdated created_at behavior and legacy-arity fallback.
app/jobs/record_usage_job.rb	Accepts optional occurred-at epoch and uses it to set UsageEvent.created_at on insert.
app/controllers/api/levelcode/v1/ai_controller.rb	Passes request-time epoch into RecordUsageJob.perform_async at both enqueue sites.
.platform/hooks/postdeploy/99_verify_sidekiq.sh	Replaces “print-only” diagnostics with a deploy-failing Sidekiq readiness check.
.platform/files/start-sidekiq.sh	Removes hardcoded ABI path and attempts to install the lockfile-pinned bundler version before launching Sidekiq.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.