fix(ops): groq-key restart loop, disk cleanup, context sync - #235
Draft
JoTalbot wants to merge 33 commits into
Draft
fix(ops): groq-key restart loop, disk cleanup, context sync#235JoTalbot wants to merge 33 commits into
JoTalbot wants to merge 33 commits into
Conversation
Deployed CatBoost direction model was degenerate (prob_up=0.433 for 30/35 assets, AUC 0.504, never reaches the 0.65 entry gate). Replace with a model trained on scale-free features with a strict per-symbol walk-forward split: AUC 0.533, hit-rate 81-83% at prob>=0.65 on two independent OOS windows, positive net PnL with Directional v2 paper exit rules. Old model kept as fallback; ml_predictor prefers catboost_price_dir_v2.cbm.
RLSignalBridge vs training env (kg_v8) mismatches: 1. onehot always marked BTC - now per-asset index in ASSET_ORDER (32 assets, alphabetical, as MultiAssetEnv); unknown asset -> None (no signal). 2. static feature 3 was vol_chg, training used vol_ratio - fixed. 3. action was not clamped to [-1,1] like in training rollout - pos could be -0.5 (outside [0,1]) - fixed. Dead tickers MATIC/RNDR (renamed to POL/RENDER) removed from ML signals, RL default universe and notebook generator template. rl_signals.json refreshed (9 assets, honest FLAT verdicts). 61 quant/ml tests pass.
…ction Data: - scripts/quant_backfill_history.py: paginated Binance klines (+Bybit fallback for KAS), refreshes stale tails first; 17 assets extended to ~5500 bars. Signal product (generate_quant_signal_product.py): - _latest_rows now picks the most complete FRESH series (staleness >2h skipped, fixes delisted TON on Binance shadowing live Bitstamp data); - regime computed on the last CLOSED bar (in-progress bar had partial volume -> false illiquid); NO_DATA dropped 16 -> 0. ML (quant_ml_eval_train.py): - engine-style simulation now executes at trigger levels (conservative), not at piercing bar closes; retrained on full dataset: AUC 0.513 -> 0.536, hit@0.65 = 82.4%, +31.6% net on OOS sim at thr 0.65. RL (quant_train_ppo.py + rl_signal_bridge.py): - train LSTM-PPO v9 1:1 with kg_v8 methodology (300 episodes, GAE, clip 0.2) on local data, universe POL instead of delisted MATIC; - validation: sum_rl +96.0% vs Buy&Hold -114.0% (v8: +51.4%); - bridge reads asset names from checkpoint (supports v8 MATIC and v9 POL), MODEL_FILE -> ppo_v9.pt; signals refreshed (10 assets, honest FLAT veto). - 61 quant/ml tests pass.
- collect_orderbook_snapshots.py: fix kucoin depth (20/100 only), add okx/bitstamp/coinbase to the collector (binance,kucoin,mexc,okx,bitstamp, coinbase); unit interval 30s -> 15s. Collection rate ~3x faster. - scripts/analyze_orderbook_data.py (new): per-exchange spread/depth stats + cross-exchange mid disparity windows (read-only); report -> data/reports/ orderbook_analysis.json. - Market-making simulator first run (>=200 snapshots): fill_rate 63-96% but negative PnL - naive passive MM suffers adverse selection; full run needs >=1000 snapshots/pair (binance/mexc ~40% there, ~1.5h to go). - docs/QUANT_SIGNAL_PRODUCT.md: 33 assets, backfill, fresh-series selection, closed-bar regime, ML v2 / PPO v9 summary. - docs/PROJECT_INVENTORY.md regenerated.
…ng brief - scripts/run_market_making_simulator_v2.py (new): inventory-aware MM research (fills only on favorable moves, single-position rule, no stacking). First run confirms naive MM economics: maker fee 0.1% (10bps) exceeds median spreads (BTC 0.002bps, ETH 0.05, SOL 1.3) -> spot MM on top pairs needs rebate programs or wider-spread universe; documented in report. - run_morning_brief.py: read-only Quant WATCH section (WATCH_UP/WATCH_DOWN from quant_signal_product.json, max 5) after the crypto PnL line.
Old paths (ppo_trader.pt, ppo_multi_24.pt, catboost_price_dir.cbm) are superseded by ppo_v9.pt and catboost_price_dir_v2.cbm; removed stale entries.
…train timers
- scripts/quant_watch_backtest.py (new): replays signal-product WATCH rules
on OOS tail; result: WATCH_DOWN precision 59.4% (85/143) - moderate edge
vs 50% baseline; WATCH_UP produced 0 signals (rule too strict in current
down-market) - documented, no rule changes.
- scripts/quant_ml_monitor.py (new): prob_up distribution stats, signal/CSV
freshness, drift vs previous snapshot; history in
data/reports/quant_ml_monitor_history.json; status OK/WARN.
- scripts/quant_ml_feature_experiment.py (new): 13 base vs 21 extended
features on same OOS; extended is NOT better (AUC 0.5326 vs 0.5355,
hit@0.65 70.8% vs 82.2%) - base feature set stays.
- deploy/systemd/aios-quant-ml-retrain.{service,timer}: weekly retrain
(Mon 04:00, deploy-only-if-better guard already in the script).
- deploy/systemd/aios-quant-ml-monitor.{service,timer}: hourly drift monitor.
- Both timers installed and enabled on the host.
Critical methodology fix: v8/v9 validation lacked the action clamp used by
the deployed bridge; act < -1.5 silently became SHORT positions (-0.5) that
are impossible in the discrete {0, 0.5, 1} policy. Those historical
profits (v9 +96%) were artifacts of hidden shorts.
- quant_train_ppo.py / quant_train_ppo_v10.py: val_on_asset now clamps
act to [-1,1] exactly like rl_signal_bridge.py.
- quant_train_ppo_v10.py (new): honest walk-forward split - env trains on
first 70% of each asset, validates on the unseen last 30% (gap 48).
- Honest result: deployed ppo_v9 is a pure FLAT agent on OOS (sum_rl 0.0
vs Buy&Hold -233%) - value is loss avoidance, not earning. v10 not
deployed (identical FLAT, no OOS edge under deployed action space).
- quant_ml_horizon_experiment.py (new): h1 label optimal (AUC 0.5355,
hit@0.65 82.2%); h4/h8/h24 worse - next-bar model stays.
- Reports: ppo_oos_honest.json, ppo_v10_oos_eval.json,
quant_ml_horizon_experiment.json.
Replays the production algorithm (ML gate, owner risk profile, TP/SL/trail exits, fees) as if trading started exactly one month ago, synchronous per-bar processing of all 33 symbols with a single 000 portfolio. Result (2026-07-14 -> 2026-08-14): - current algorithm: -0.30% (1 trade; ML>=0.65 gate blocked 1948 entries) - control (no ML gate): -0.54% (5 trades, 2 wins) - market Buy&Hold: mean -9.30%, best ADA +9.00%, worst BONK -37.73%, BTC -2.71% The algorithm beat the average currency by +9.0pp and BTC by +2.4pp, i.e. the gate acted as capital protection (mostly cash), not an earner.
Re-run for 3/6/12 calendar months. The deployed algorithm holds cash in all windows (ML>=0.65 gate almost never fires; after the first -1.5% loss the 0.25% DD kill blocks further entries), so portfolio PnL is -0.30% in every window while the market fell: mean currency -30.75% (3m), -24.11% (6m), -68.32% (12m). BTC -46.7%, ETH -58.1%, ADA -80.0% over 12m. The gate acts as capital protection (beats average currency by +9..+68pp) but never earns: cash (0%) beats the algorithm by 0.3pp in every window.
Systematic research over 16 strategies on honest OOS (no lookahead, 0.25% per-side costs, ML retrained on train window only, equal-weight symbols): - Winner: daily SMA50/200 long/short cross-following: 70/30 split +7.8..+11.2% (params 40/160, 50/200, 60/240 all positive); 50/50 split +34.7% (50/200) / +19.7% (60/240), both OOS halves positive. - Long-only variant of the same rules: -4.5% -> profit comes from shorts. - XS mean-reversion unstable (bot3_p7: +9.6% then -12.9%); RSI daily MR weak (+1.8..+5.6%); ML long/short and inverted-ML negative on OOS. - Report: data/reports/strategy_research_summary.md + JSON artifacts. Caveats: funding/borrow costs not modeled; profit concentrated in the bear regime; constitutional gate requires owner decision for real shorts.
…earns Backtest engine fix (v2): equity now compounds bar by bar (earlier version summed arithmetic returns, which overstates PnL on volatile assets - e.g. sum -97.96% vs actual -71.3% for BONK) and the position adopts the signal at the same bar close (true next-bar semantics). Corrected results for daily SMA50/200 long/short with 0.25%/side costs: OOS30: +12.62% net (funding base), +1.68% (funding stress) OOS50: +41.56% net (base), +23.92% (stress); half1 +32.4%, half2 +6.5%; 24/33 symbols positive. Both halves and all split x funding combos > 0. MA_LS_60_240: +14.0% (OOS30), +29.9% (OOS50). All long-only variants and Donchian/XS stay negative -> shorts are the edge. Report: data/reports/earn_research_summary.md + JSON.
Findings: aios-groq-key restart loop (missing script, 7816 restarts), disk 83%, context/runtime entry_mode drift, quant exchange_not_allowed=96, 31 unpushed commits. No code or runtime changes.
aios-groq-key.service pointed at groq_key_retry.py which exists neither in the filesystem nor in git history; the unit had accumulated 7832 restarts (~2833/day). Its function is already covered by the live successor aios-groq-autopilot.timer (hourly, status ok, 8 keys). Unit stopped, disabled and masked; base unit preserved in deploy/systemd and a runtime backup. Masks snapshot and its inventory test updated accordingly. Disk 83% -> 81% by removing only disposable data (apt cache, crash dumps, disabled snap revisions, pip/pip-tools caches, stale /tmp, __pycache__). AIOS backups were left intact - all within the projects own retention policy. cleanup_disk.sh now also prunes regenerable caches so this does not recur. Main consumers (ollama 14G, docker 17G) are in active use and deliberately untouched - owner decision. PROJECT_CONTEXT corrected: runtime entry_mode is enabled under the owner-approved constrained paper profile, not freeze; documented the verified quant slice (18h, trades=0, exchange_not_allowed=96/scan). Verified: systemctl --failed empty, audit_deployment_sources drift 0, tests/test_systemd_inventory.py 3 passed, ruff clean, health 21/22.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Восстановленный orphan-PR: работа без PR с 2026-08-14. +33/−64 vs main.
Этот PR создан ИИ-агентом (OpenHands) от имени оператора при наведении порядка в ветках.