diff --git a/README.md b/README.md index 1e51d44a..b0a5f8a5 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,9 @@ GAS runs two parallel competition tracks on Bittensor Subnet 34: **Key facts:** - **Three modalities**: Image, video, and audio detection are all scored independently -- **Cloud-evaluated**: Discriminator models are benchmarked on cloud infrastructure -- no GPU hosting required - **Model format**: Safetensors only (ONNX submissions are not accepted) - **Datasets refresh weekly** with fresh GAS-Station data alongside static benchmarks -- **One model per modality per hotkey** for discriminative miners +- **One model per hotkey** for discriminative miners See [Incentive Mechanism](docs/Incentive.md) for full scoring details. @@ -71,17 +70,10 @@ gascli validator start # Miners: Start or restart generative miner gascli generator start -# Miners: Push discriminator models (all three modalities at once) -gascli d push \ - --image-model image_detector.zip \ - --video-model video_detector.zip \ - --audio-model audio_detector.zip \ +# Miners: Push one discriminator model per hotkey +gascli d push --image-model image_detector.zip \ --wallet-name default --wallet-hotkey default - -# Or push one model at a time -gascli d push --image-model image_detector.zip -gascli d push --video-model video_detector.zip -gascli d push --audio-model audio_detector.zip +# Video or audio: use --video-model or --audio-model on a different hotkey # Miners: Check your benchmark performance (epistula-authenticated) gascli d perf --wallet-name default --wallet-hotkey default @@ -104,12 +96,10 @@ pm2 start validator.config.js # Miners: Start or restart generative miner pm2 start gen_miner.config.js -# Miners: Push discriminator models +# Miners: Push one discriminator model per hotkey source .venv/bin/activate python neurons/discriminator/push_model.py \ --image-model image_detector.zip \ - --video-model video_detector.zip \ - --audio-model audio_detector.zip \ --wallet-name default --wallet-hotkey default ``` For detailed installation and usage instructions, see [Installation Guide](docs/Installation.md). @@ -120,7 +110,7 @@ For detailed installation and usage instructions, see [Installation Guide](docs/ > This documentation assumes basic familiarity with [Bittensor concepts](https://docs.bittensor.com/learn/bittensor-building-blocks). #### Discriminative Miners [[docs](docs/Discriminative-Mining.md)] -Discriminative miners submit detection models for evaluation against a wide variety of real and synthetic media across **image, video, and audio** modalities. Models are evaluated on cloud infrastructure and rewarded based on their accuracy and calibration. This significantly reduces the capital required to mine compared to previous versions that required GPU hosting, and allows the subnet to more reliably identify unique models and reward novel contributions proportionally to their accuracy. +Discriminative miners submit detection models for evaluation against a wide variety of real and synthetic media across **image, video, and audio** modalities. Models are rewarded based on their accuracy and calibration. This significantly reduces the capital required to mine compared to previous versions that required GPU hosting, and allows the subnet to more reliably identify unique models and reward novel contributions proportionally to their accuracy. #### Generative Miners [[docs](docs/Generative-Mining.md)] diff --git a/VERSION b/VERSION index 43110fd2..28cbf7c0 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -4.9.11 +5.0.0 \ No newline at end of file diff --git a/docs/Discriminative-Mining.md b/docs/Discriminative-Mining.md index d7984e58..82909118 100644 --- a/docs/Discriminative-Mining.md +++ b/docs/Discriminative-Mining.md @@ -9,7 +9,7 @@ Follow the [Installation Guide](Installation.md) to set up your environment befo - Miners submit media-provenance classifiers across three modalities: **image**, **video**, and **audio**. - Image models classify `[real, synthetic, semisynthetic]`; video models classify `[real, synthetic, semisynthetic, rendered]`; audio remains `[real, synthetic]`. - The visual taxonomy is experimental. Semisynthetic media retains materially captured visual content alongside spatially localized generated or replaced content. Fully synthesized output remains synthetic even when captured media conditions generation. -- Models are evaluated on cloud infrastructure -- miners do not need to host hardware for inference. +- Miners do not need to host hardware for inference. Class order is part of the submission contract: @@ -31,11 +31,13 @@ Discriminative miners must submit models in **safetensors format**: **πŸ“– [Safetensors Model Specification](https://github.com/bitmind-ai/gasbench/blob/main/docs/Safetensors.md)** - Requirements for model submission -You can submit models for any combination of modalities: +Each hotkey can submit **one** model, in one of these modalities: - `image_detector.zip` - Image classification model -- `video_detector.zip` - Video classification model +- `video_detector.zip` - Video classification model - `audio_detector.zip` - Audio classification model +A second modality needs a second registered hotkey. + ## Pushing Your Model First, activate the virtual environment: @@ -43,22 +45,17 @@ First, activate the virtual environment: source .venv/bin/activate ``` -Push your models to the network using the `push` command: +Push one model per hotkey using the `push` command: ```bash -# Upload all three models gascli d push \ --image-model image_detector.zip \ - --video-model video_detector.zip \ - --audio-model audio_detector.zip \ --wallet-name your_wallet_name \ --wallet-hotkey your_hotkey_name -# Or upload individual models -gascli d push \ - --image-model image_detector.zip \ - --wallet-name your_wallet_name \ - --wallet-hotkey your_hotkey_name +# Video or audio on a different hotkey: +# gascli d push --video-model video_detector.zip --wallet-hotkey video_key +# gascli d push --audio-model audio_detector.zip --wallet-hotkey audio_key ``` ### Command Options @@ -68,8 +65,6 @@ The `push` command accepts several parameters: ```bash gascli d push \ --image-model image_detector.zip \ - --video-model video_detector.zip \ - --audio-model audio_detector.zip \ --wallet-name your_wallet_name \ --wallet-hotkey your_hotkey_name \ --netuid 34 \ @@ -87,7 +82,16 @@ gascli d push \ - `--chain-endpoint`: Subtensor network endpoint (default: "wss://entrypoint-finney.opentensor.ai:443/") - `--retry-delay`: Retry delay in seconds (default: 60) -At least one model (image, video, or audio) must be provided. +Provide exactly one of `--image-model`, `--video-model`, or `--audio-model`. A second model needs a different hotkey. + +## Submission Limits + +Each registered hotkey gets **one counted submission** (image, video, or audio β€” not one of each). + +- Exam failures and incomplete uploads do not consume the slot. You can retry on the same key until a model is successfully uploaded and not later marked exam-failed. +- A confirmed or superseded model **does** consume the slot for the life of that hotkey, for every modality. +- A new benchmark version does **not** refill the slot. +- To submit another model, register a new miner hotkey. --- @@ -104,7 +108,7 @@ The normalized terms apply exponents $1.2$ to MCC performance and $1.8$ to Brier ### Model Requirements - **Format**: Safetensors only (ONNX is no longer accepted) -- **Three model per modality per hotkey**: You can submit up to three image, three video, and three audio model per registered hotkey +- **Submission cap**: one counted model per hotkey (see [Submission Limits](#submission-limits)) ### Sandbox and Import Restrictions @@ -114,7 +118,6 @@ For the complete list of allowed and blocked imports, see the [Safetensors Model ### Evaluation -- Models are benchmarked on cloud infrastructure (not miner hardware) - Evaluation runs against a diverse dataset of image samples, video samples, and audio samples per benchmark cycle - Datasets are refreshed weekly with new GAS-Station data alongside static benchmark datasets @@ -149,7 +152,7 @@ gascli d push --image-model my_detector.zip ### What Happens During Push 1. **Model Validation**: The system checks that the zip files are present and valid -2. **Model Upload**: Your model zip files are uploaded to the cloud inference system +2. **Model Upload**: Your model zip files are uploaded for evaluation 3. **Blockchain Registration**: Model metadata is registered on the Bittensor blockchain 4. **Verification**: The system verifies the registration was successful @@ -166,7 +169,7 @@ Before your model is ever scored on the network, it must pass an **entrance exam - Internally this runs `gasbench run --small`, which downloads one archive per dataset and evaluates roughly 100 samples per dataset - Your model must achieve **β‰₯ 80% accuracy** averaged across all submitted modalities to pass - The exam has a **maximum wall-clock timeout of 1 hour 25 minutes** (5,100 seconds); models that exceed this are treated as failed -- The exam runs in an **isolated cloud sandbox** β€” your code has no network access and cannot interact with the host environment +- The exam runs in an **isolated sandbox** β€” your code has no network access and cannot interact with the host environment - Submissions are statically analyzed and executed in an isolated sandbox; prohibited code or imports result in rejection **Model status during the exam:** @@ -194,7 +197,7 @@ Models that pass the entrance exam are benchmarked against the **complete datase - **Private holdout datasets** β€” curated datasets not visible to miners, used to prevent overfitting to the public benchmark set - Refreshed weekly with new data from the GAS-Station pipeline -The full benchmark has a **maximum wall-clock timeout of 5 hours** (18,000 seconds) per modality. The benchmark score from this stage determines your **TAO emissions** on Subnet 34. The active round configuration selects provenance weighting, multiclass scoring, and augmentation robustness parameters; see [Incentive Mechanism](Incentive.md). +The full benchmark has a **maximum wall-clock timeout of 5 hours** (18,000 seconds) per modality. This `sn34_score` is what the King of the Hill competition uses: a high enough score can take or keep a lane, and emissions then follow the 85/10/5 split on the current king plus the previous two. The active round configuration selects provenance weighting, multiclass scoring, and augmentation robustness parameters; see [Incentive Mechanism](Incentive.md). You can simulate a full benchmark run locally (without holdouts) to get a sense of your model's performance: diff --git a/docs/Incentive.md b/docs/Incentive.md index f061b380..558f0ec2 100644 --- a/docs/Incentive.md +++ b/docs/Incentive.md @@ -1,7 +1,7 @@ # Incentive Mechanism ## Benchmark Runs -Submitted discriminator miners are evaluated against a subset of the data sources listed below. Models are evaluated on cloud infrastructure -- miners do not need to host hardware for inference. A portion of the evaluation data comes from generative miners, who are rewarded based on their ability to submit data that both pass validator sanity checks (prompt alignment, etc.) and fool discriminators in benchmark runs. +Submitted discriminator miners are evaluated against a subset of the data sources listed below. Miners do not need to host hardware for inference. A portion of the evaluation data comes from generative miners, who are rewarded based on their ability to submit data that both pass validator sanity checks (prompt alignment, etc.) and fool discriminators in benchmark runs. Each modality (image, video, audio) is scored independently using the `sn34_score` metric, which combines classification performance (MCC) with probability calibration (Brier score). The active round selects binary or multiclass scoring per modality. @@ -138,19 +138,21 @@ The benchmark records `base_sn34_score`, `aug_sn34_score`, and robustness diagno The normative implementation details and complete metric field glossary live in GASBench's [Classification Taxonomy and Scoring](https://github.com/BitMind-AI/gasbench/blob/main/docs/Classification-and-Scoring.md). -### Competition Rounds +### King of the Hill -The discriminator competition is organized into **rounds**. Each round introduces new benchmark datasets and evaluates all submitted models. Winners are determined **per modality** (image, video, audio) independently. +Discriminator emission is King of the Hill. Each modality has one reigning model. Validators set that lane's weight on registered hotkeys every tempo β€” not on an escrow wallet. Each hotkey may land **one counted submission** for the life of that registration (any modality; exam failures do not count; a new model needs a new key). -#### How Rounds Work +Current split: -1. **New round begins**: Benchmark datasets are updated (new GAS-Station data, potentially new static datasets). All modalities share the same benchmark version number. -2. **Models are benchmarked**: All submitted discriminator models are evaluated against the current round's datasets and scored using `sn34_score`. -3. **Winner determined per modality**: The highest-scoring model for each modality wins that round. -4. **Alpha reward**: The round winner for each modality receives an alpha reward. +- Image lane: 40% +- Video lane: 40% +- Audio lane: 4% +- Generators: 16% -#### Winner-Take-All Per Round +Each discriminator lane is split **85 / 10 / 5** across the current king and the previous two **distinct** crowned hotkeys. If a lane has no previous king, that residual rolls up to the current king (a first king receives the full lane). An unresolvable current king burns its share; an unresolvable previous king rolls to the current king when that UID is registered. -Each round is winner-take-all -- only the top-scoring discriminator for each modality receives the alpha reward for that round. This incentivizes miners to continuously improve their models and push the state of the art in AI-generated content detection. +A challenger takes the crown when it posts an `sn34_score` at least **0.01** higher than the sitting king on the **same** `CURRENT_BENCHMARK_VERSION`. Empty-lane seeding and failed-defense replacement do not use the margin. The same `file_hash` can refresh its stored score without resetting the reign. -Rounds progress as benchmark versions are incremented, ensuring that models are always evaluated against fresh, evolving data. +When a new benchmark version is released the current king keeps receiving weights. The throne is marked `defending` until that exact model completes a full re-eval on the new version. Dethroning is frozen during defense. After a successful defense, deferred challengers still need the 0.01 margin. If the re-eval fails or times out (48 hours), the crown goes to the best successful new-version model, or that lane's share burns until one exists. + +Alpha accrues on the chain hotkeys while they hold those residual shares. There is no end-of-round escrow transfer and no pot. diff --git a/docs/Mining.md b/docs/Mining.md index d8eacc89..d2cab13f 100644 --- a/docs/Mining.md +++ b/docs/Mining.md @@ -3,7 +3,7 @@ GAS supports two types of miners that work together in an adversarial loop: ## [Discriminative Mining](Discriminative-Mining.md) πŸ“– -Miners submit classifiers that detect AI-generated content across **image, video, and audio** modalities. Models are evaluated on cloud infrastructure against diverse benchmark datasets and scored using the `sn34_score` metric (accuracy + calibration). +Miners submit classifiers that detect AI-generated content across **image, video, and audio** modalities. Models are evaluated against diverse benchmark datasets and scored using the `sn34_score` metric (accuracy + calibration). Each hotkey gets one counted submission (any modality); see [Discriminative Mining](Discriminative-Mining.md#submission-limits). ## [Generative Mining](Generative-Mining.md) 🎨 Miners create synthetic media (images and videos) that challenges the discriminators. They generate increasingly realistic content to test and improve detection capabilities, and are rewarded based on validation pass rate and adversarial performance. diff --git a/docs/Validating-New.md b/docs/Validating-New.md deleted file mode 100644 index ad8e4c6f..00000000 --- a/docs/Validating-New.md +++ /dev/null @@ -1,50 +0,0 @@ -# Validator Guide - -Run the SN34 validator (validator + generator + data services). Two ways: **PM2** (native) or **Docker**. - -Prerequisites: [Installation Guide](Installation.md), Bittensor wallet, GPU for generation. - ---- - -## Quick start: PM2 - -```bash -cp .env.validator.template .env.validator -# Edit .env.validator: WALLET_NAME, WALLET_HOTKEY, API keys, CHAIN_ENDPOINT -source .venv/bin/activate -gascli validator start -``` - -That starts three PM2 processes: `sn34-validator`, `sn34-generator`, `sn34-data`. Use `gascli v stop|status|logs|--help` as needed. - ---- - -## Quick start: Docker - -Three containers (validator, generator, data) share one image and volumes; one process per container. - -```bash -cp .env.validator.template .env.validator -# Edit .env.validator (same as above; WALLET_PATH is for host wallet bind-mount) -docker compose --env-file .env.validator up -d --build -docker compose logs -f validator # or generator, data -``` - -Always use `--env-file .env.validator`. First run downloads 100+ GB of models into the `hf-cache` volume. - -**Update:** Manual: `git pull && docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d`. Automatic: cron with `docker/autoupdate.sh` (see [.env.validator.template](../.env.validator.template) comments for crontab example). - ---- - -## Config - -One file for both PM2 and Docker: `.env.validator`. Copy from [.env.validator.template](../.env.validator.template) and fill in wallet, API keys, `CHAIN_ENDPOINT`. The template lists every option and Docker-specific notes (e.g. `WALLET_PATH`, cache paths, autoupdate cron). - ---- - -## Reference - -| Path | Start | Stop / logs | -|--------|-------|-------------| -| PM2 | `gascli validator start` | `gascli v stop` / `gascli v logs` | -| Docker | `docker compose --env-file .env.validator up -d` | `docker compose down` / `docker compose logs -f validator` (or `generator`, `data`) | diff --git a/docs/Validating.md b/docs/Validating.md index 23a18aa9..ad8e4c6f 100644 --- a/docs/Validating.md +++ b/docs/Validating.md @@ -1,169 +1,50 @@ # Validator Guide -## Required .env.validator variables (PM2 and Docker) +Run the SN34 validator (validator + generator + data services). Two ways: **PM2** (native) or **Docker**. -You **must** set these in `.env.validator` before starting; the rest have defaults in the template. - -| Variable | Required? | Notes | -|----------|-----------|--------| -| `WALLET_NAME` | **Yes** | Your Bittensor wallet name (e.g. `default` or a name you created). | -| `WALLET_HOTKEY` | **Yes** | Hotkey for this validator (e.g. `default`). Must be registered on the subnet. | -| `HUGGINGFACE_HUB_TOKEN` | **Yes** | Needed to download models and upload data. Create at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). Reach out to a member of the BitMind team to have your account added to the `gasstation` org | -| `CHAIN_ENDPOINT` | No | Default is mainnet. Set to testnet URL if needed. | -| `WANDB_API_KEY` | No | Optional; for Weights & Biases logging. | -| Cache / Docker paths | No | `SN34_CACHE_DIR`, `HF_HOME`, `WALLET_PATH`, `BT_LOGGING_LOGGING_DIR` have defaults; override if you use different paths. | - -See [.env.validator.template](../.env.validator.template) for every option. +Prerequisites: [Installation Guide](Installation.md), Bittensor wallet, GPU for generation. --- -## PM2 setup - -Follow the [Installation Guide](Installation.md) to set up your environment before proceeding with validator setup. - -> Create a `.env.validator` file and **set the required variables above** (wallet, API keys). Then: - -```bash -$ cp .env.validator.template .env.validator -# Edit .env.validator: fill in WALLET_NAME, WALLET_HOTKEY, HUGGINGFACE_HUB_TOKEN at minimum -``` - -See [.env.validator.template](../.env.validator.template) for all options. Then activate the virtual environment and start your validator processes: - -```bash -$ source .venv/bin/activate -$ gascli validator start -``` -The above command will create 3 pm2 processes: -```bash -β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ id β”‚ name β”‚ namespace β”‚ version β”‚ mode β”‚ pid β”‚ uptime β”‚ β†Ί β”‚ status β”‚ cpu β”‚ mem β”‚ user β”‚ watching β”‚ -β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ 2 β”‚ sn34-data β”‚ default β”‚ N/A β”‚ fork β”‚ 4032914 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 529.0mb β”‚ user β”‚ disabled β”‚ -β”‚ 1 β”‚ sn34-generator β”‚ default β”‚ N/A β”‚ fork β”‚ 4032936 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 448.5mb β”‚ user β”‚ disabled β”‚ -β”‚ 0 β”‚ sn34-validator β”‚ default β”‚ N/A β”‚ fork β”‚ 4032918 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 504.0mb β”‚ user β”‚ disabled β”‚ -β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` -- **sn34-data**: Handles data downloads -- **sn34-generator**: Responsible for generating prompts, synthetic media, and validating miner-generated data -- **sn34-validator**: Core validator logic. Challenges, scoring, weight setting. - +## Quick start: PM2 -### Validator Operations - -First, activate the virtual environment: ```bash +cp .env.validator.template .env.validator +# Edit .env.validator: WALLET_NAME, WALLET_HOTKEY, API keys, CHAIN_ENDPOINT source .venv/bin/activate -``` - -Then run validator commands: -```bash -# Start validator services gascli validator start -gascli v start # Using alias -gascli v stop -gascli v status -gascli v logs -gascli v --help ``` +That starts three PM2 processes: `sn34-validator`, `sn34-generator`, `sn34-data`. Use `gascli v stop|status|logs|--help` as needed. -## Docker Deployment - -As an alternative to the PM2-based setup above, you can run the validator stack in Docker. Three containers (validator, generator, data) run one process each and share bind-mounted cache and wallet; the validator container uses `network_mode: host` so its FastAPI callback is on the host port miners are told to hit. - -### Prerequisites - -- **Docker** (with Docker Compose v2) -- **NVIDIA Container Toolkit** for GPU passthrough. Install guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html -- **Bittensor wallet** files already created on the host (typically at `~/.bittensor/wallets/`) - -### Quick Start - -1. **Clone the repository** (if not already done): - ```bash - git clone - cd bitmind-subnet - ``` - -2. **Create your `.env.validator`** and set the **required variables** (same as PM2): - ```bash - cp .env.validator.template .env.validator - ``` - You **must** set: `WALLET_NAME`, `WALLET_HOTKEY`, `HUGGINGFACE_HUB_TOKEN`, `OPEN_ROUTER_API_KEY`. See [Required .env.validator variables](#required-envvalidator-variables-pm2-and-docker) and [.env.validator.template](../.env.validator.template) for the full list. - -3. **Build and start** (use `--env-file .env.validator` so the same file drives both container env and Compose options like `WALLET_PATH` and `CALLBACK_PORT`): - ```bash - docker compose --env-file .env.validator up -d --build - ``` - -4. **View logs** (per service): - ```bash - docker compose logs -f validator # or generator, data - ``` - - -### Updating +--- -**Automatic (recommended):** add a crontab entry so `docker/autoupdate.sh` runs periodically (it checks VERSION, then pulls, runs `down --remove-orphans`, rebuilds, and brings the stack up): +## Quick start: Docker -```bash -*/5 * * * * /path/to/bitmind-subnet/docker/autoupdate.sh >> /var/log/bitmind-docker-update.log 2>&1 -``` - -**Manual:** rebuild and recreate all services: +Three containers (validator, generator, data) share one image and volumes; one process per container. ```bash -git pull -docker compose --env-file .env.validator down --remove-orphans -docker compose --env-file .env.validator build -docker compose --env-file .env.validator up -d +cp .env.validator.template .env.validator +# Edit .env.validator (same as above; WALLET_PATH is for host wallet bind-mount) +docker compose --env-file .env.validator up -d --build +docker compose logs -f validator # or generator, data ``` -### Configuration +Always use `--env-file .env.validator`. First run downloads 100+ GB of models into the `hf-cache` volume. -| Variable | Docker behavior | -|---|---| -| `SN34_CACHE_DIR` | Bind-mounted into the container; same path as PM2 so cache is shared when you switch. Set in `.env.validator`; use `--env-file .env.validator`. | -| `HF_HOME` | Bind-mounted into the container; same path as PM2 so model cache is shared. Set in `.env.validator`. | -| `AUTO_UPDATE` | In-container autoupdate is disabled. Use manual rebuild or the cron-based `docker/autoupdate.sh` (see Updating). | -| `WALLET_PATH` | Host base path for wallets. Only the directory `WALLET_PATH/WALLET_NAME` is mounted (not the whole wallets folder). Set in `.env.validator`. | -| `WALLET_NAME` | Which wallet dir to mount; with `WALLET_PATH` only that wallet is visible to the container. | -| `BT_LOGGING_LOGGING_DIR` | Base path for validator state (scores, challenge tasks) and bittensor logs. State lives under `BT_LOGGING_LOGGING_DIR`/`WALLET_NAME`/`WALLET_HOTKEY`/…; bind-mounted so it persists. Default `~/.bittensor`. | -| `NETUID` | Auto-derived from `CHAIN_ENDPOINT`. Set explicitly if using a custom endpoint. | +**Update:** Manual: `git pull && docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d`. Automatic: cron with `docker/autoupdate.sh` (see [.env.validator.template](../.env.validator.template) comments for crontab example). -### Wallet, cache, and validator state - -- **Wallet:** Only the configured wallet directory (`WALLET_PATH`/`WALLET_NAME`) is bind-mounted, not the entire wallets folder. -- **Cache:** `SN34_CACHE_DIR` and `HF_HOME` are bind-mounted so PM2 and Docker share the same data (no re-download when switching). -- **Validator state:** Scores and challenge tasks are saved under bittensor’s logging path (`BT_LOGGING_LOGGING_DIR`/`WALLET_NAME`/`WALLET_HOTKEY`/…). That path is bind-mounted so state persists across container restarts. - -> **Note**: The first startup will download 100+ GB of ML models into the HF cache directory. Subsequent restarts reuse the cached models. - -### Common Operations - -Use `--env-file .env.validator` so Compose reads `WALLET_PATH`, `WALLET_NAME`, `CALLBACK_PORT`, and cache paths from your config: - -```bash -# Start all three services (validator, generator, data) -docker compose --env-file .env.validator up -d - -# Stop all -docker compose down +--- -# View logs (per service) -docker compose logs -f validator # or generator, data +## Config -# Rebuild after code changes -docker compose --env-file .env.validator down --remove-orphans -docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d +One file for both PM2 and Docker: `.env.validator`. Copy from [.env.validator.template](../.env.validator.template) and fill in wallet, API keys, `CHAIN_ENDPOINT`. The template lists every option and Docker-specific notes (e.g. `WALLET_PATH`, cache paths, autoupdate cron). -# Restart one service -docker compose restart validator # or generator, data +--- -# Container status -docker compose ps +## Reference -# Shell into a container -docker compose exec validator bash # or generator, data -``` +| Path | Start | Stop / logs | +|--------|-------|-------------| +| PM2 | `gascli validator start` | `gascli v stop` / `gascli v logs` | +| Docker | `docker compose --env-file .env.validator up -d` | `docker compose down` / `docker compose logs -f validator` (or `generator`, `data`) | diff --git a/gas/__init__.py b/gas/__init__.py index f3d88ce6..1d041248 100644 --- a/gas/__init__.py +++ b/gas/__init__.py @@ -1,4 +1,4 @@ -__version__ = "4.9.11" +__version__ = "5.0.0" version_split = __version__.split(".") __spec_version__ = ( diff --git a/gas/cli.py b/gas/cli.py index e71fd759..33af9fa7 100644 --- a/gas/cli.py +++ b/gas/cli.py @@ -454,8 +454,11 @@ def discriminator(): @click.option("--chain-endpoint", help="Subtensor network endpoint") @click.option("--retry-delay", default=60, help="Retry delay in seconds") @click.option("--vertical", type=click.Choice(["general", "human"]), default="general", help="Competition vertical (default: general)") +@click.option("--upload-endpoint", default=None, help="Model upload URL (default: production https://upload.bitmind.ai/upload)") +@click.option("--skip-chain", is_flag=True, help="Exit after a successful upload without writing on-chain metadata") +@click.option("--max-retries", default=3, type=int, help="Max blockchain registration attempts per modality (0 = retry forever)") def push_discriminator( - image_model, video_model, audio_model, wallet_name, wallet_hotkey, netuid, chain_endpoint, retry_delay, vertical + image_model, video_model, audio_model, wallet_name, wallet_hotkey, netuid, chain_endpoint, retry_delay, vertical, upload_endpoint, skip_chain, max_retries ): """Push discriminator model(s) and register on blockchain. At least one model zip file (image, video, or audio) must be provided.""" if not image_model and not video_model and not audio_model: @@ -488,14 +491,19 @@ def push_discriminator( cmd.extend(["--retry-delay", str(retry_delay)]) cmd.extend(["--vertical", vertical]) - - # Execute the push_model script + cmd.extend(["--max-retries", str(max_retries)]) + if upload_endpoint: + cmd.extend(["--upload-endpoint", upload_endpoint]) + if skip_chain: + cmd.append("--skip-chain") + + # Execute the push_model script. Always propagate its exit code so a + # successful upload does not leave this click process hanging. try: - result = subprocess.run(cmd, check=True) - except subprocess.CalledProcessError as e: - sys.exit(e.returncode) - except Exception as e: + result = subprocess.run(cmd) + except Exception: sys.exit(1) + sys.exit(result.returncode) discriminator.add_command(push_discriminator, name="push") diff --git a/gas/koth_weights.py b/gas/koth_weights.py new file mode 100644 index 00000000..3b1dec07 --- /dev/null +++ b/gas/koth_weights.py @@ -0,0 +1,200 @@ +"""Build on-chain weights for King-of-the-Hill discriminator lanes.""" + +from datetime import datetime, timezone +from typing import Callable, Dict, Iterable, List, Optional + +import numpy as np + +KOTH_SPLIT = { + "image": 0.40, + "video": 0.40, + "audio": 0.04, + "generator": 0.16, +} +# Per-lane residual after the current king: previous, then two-back. +# Unused slots (no prior distinct king) roll up to the current king. +KOTH_LANE_RESIDUAL = (0.85, 0.10, 0.05) +KOTH_CHAIN_ROLES = ("current", "previous", "two_back") + + +def discriminator_emissions_enabled( + payload: Optional[dict], now: Optional[datetime] = None +) -> bool: + """Require explicit GAS activation and an elapsed, timezone-aware boundary. + + A cached warm-up response stays disabled after the boundary. GAS must first + publish an enabled response; the validator never starts its own timer. + """ + if not isinstance(payload, dict) or payload.get("emissions_enabled") is not True: + return False + raw_start = payload.get("emissions_start_at") + if not isinstance(raw_start, str): + return False + try: + start = datetime.fromisoformat(raw_start.replace("Z", "+00:00")) + except ValueError: + return False + if start.tzinfo is None: + return False + return start <= (now or datetime.now(timezone.utc)) + + +def kings_by_modality(payload: Optional[dict]) -> Dict[str, str]: + """Map modality -> hotkey from a /kings response.""" + out: Dict[str, str] = {} + if not payload: + return out + for king in payload.get("kings") or []: + modality = king.get("modality") + hotkey = king.get("ss58_address") + if modality in ("image", "video", "audio") and hotkey: + out[modality] = hotkey + return out + + +def assign_residual_shares(hotkeys: Iterable[str]) -> List[Dict[str, object]]: + """Map last-N distinct hotkeys to 85/10/5, rolling unused slots to current.""" + keys = [] + seen = set() + for key in hotkeys: + if not key or key in seen: + continue + keys.append(key) + seen.add(key) + if len(keys) >= len(KOTH_LANE_RESIDUAL): + break + if not keys: + return [] + shares = list(KOTH_LANE_RESIDUAL[: len(keys)]) + shares[0] += sum(KOTH_LANE_RESIDUAL[len(keys) :]) + return [ + { + "ss58_address": key, + "share": shares[index], + "role": KOTH_CHAIN_ROLES[index], + } + for index, key in enumerate(keys) + ] + + +def chains_by_modality(payload: Optional[dict]) -> Dict[str, List[Dict[str, object]]]: + """Last-3 distinct kings per lane, with residual shares recomputed locally.""" + if not payload: + return {} + raw = payload.get("chain") or {} + kings = kings_by_modality(payload) + out: Dict[str, List[Dict[str, object]]] = {} + for modality in ("image", "video", "audio"): + members = raw.get(modality) or [] + hotkeys = [] + for member in members: + if isinstance(member, str): + hotkeys.append(member) + elif isinstance(member, dict): + hotkeys.append(member.get("ss58_address")) + assigned = assign_residual_shares(hotkeys) + if not assigned and kings.get(modality): + assigned = assign_residual_shares([kings[modality]]) + if assigned: + out[modality] = assigned + return out + + +def build_koth_weights( + n: int, + scores: np.ndarray, + generator_uids: Iterable[int], + kings: Dict[str, str], + uid_for_hotkey: Callable[[str], Optional[int]], + burn_uid: Optional[int] = None, + split: Optional[Dict[str, float]] = None, + chains: Optional[Dict[str, List[Dict[str, object]]]] = None, + emissions_enabled: bool = True, +) -> np.ndarray: + """Return a length-n weight vector. Missing kings go to burn_uid. + + `uid_for_hotkey` must resolve at current chain head. Escrow addresses are + never used. Each discriminator lane is 85/10/5 across the current king and + the previous two distinct kings. Unused residual slots roll to the current + king. An unresolvable current king burns its share; unresolvable previous + kings roll to the current king when that UID resolved. When emissions are + disabled, all discriminator lanes burn; generator rewards are unchanged. + """ + split = dict(KOTH_SPLIT if split is None else split) + try: + split = {key: float(split[key]) for key in KOTH_SPLIT} + except (KeyError, TypeError, ValueError) as exc: + raise ValueError("Invalid KOTH split") from exc + if any( + not np.isfinite(value) or value < 0 for value in split.values() + ) or not np.isclose(sum(split.values()), 1.0): + raise ValueError("Invalid KOTH split") + weights = np.zeros(n, dtype=np.float64) + if len(scores) < n: + scores = np.append(scores, np.zeros(n - len(scores))) + elif len(scores) > n: + scores = scores[:n] + + king_uids = set() + burned = 0.0 + for modality in ("image", "video", "audio"): + pct = split[modality] + if not emissions_enabled: + burned += pct + continue + members = list((chains or {}).get(modality) or []) + if not members: + hotkey = kings.get(modality) + members = assign_residual_shares([hotkey] if hotkey else []) + if not members: + burned += pct + continue + + current_uid = None + leftover = 0.0 + current_resolved = False + for index, member in enumerate(members): + hotkey = member.get("ss58_address") + share = float(member.get("share") or 0.0) + if share <= 0: + continue + uid = uid_for_hotkey(hotkey) if hotkey else None + if uid is None or uid < 0 or uid >= n: + if index == 0: + burned += pct * share + else: + leftover += share + continue + weights[uid] += pct * share + king_uids.add(uid) + if index == 0: + current_uid = uid + current_resolved = True + + if leftover > 0: + if current_resolved and current_uid is not None: + weights[current_uid] += pct * leftover + else: + burned += pct * leftover + + generator_pct = split["generator"] + active = [uid for uid in generator_uids if 0 <= uid < n and uid not in king_uids] + if active and generator_pct > 0: + gen_scores = np.array([max(float(scores[uid]), 0.0) for uid in active]) + total = float(np.sum(gen_scores)) + if total > 0: + for uid, score in zip(active, gen_scores): + weights[uid] += generator_pct * (score / total) + else: + burned += generator_pct + else: + burned += generator_pct + + if burned > 0: + if burn_uid is None or not 0 <= burn_uid < n: + raise ValueError( + f"Cannot allocate {burned:.4f} burn weight: burn UID is unavailable" + ) + weights[burn_uid] += burned + + return weights diff --git a/gas/protocol/miner_requests.py b/gas/protocol/miner_requests.py index dd9d3ab8..3a00e474 100644 --- a/gas/protocol/miner_requests.py +++ b/gas/protocol/miner_requests.py @@ -279,6 +279,7 @@ def extract_error(result: dict) -> str: "error": extract_error(presigned_result), "response": presigned_result['response'], "already_uploaded": True, + "file_hash": file_hash, } return { "success": False, diff --git a/gas/protocol/validator_requests.py b/gas/protocol/validator_requests.py index 6a6eae0a..180ebc8c 100644 --- a/gas/protocol/validator_requests.py +++ b/gas/protocol/validator_requests.py @@ -179,6 +179,35 @@ async def query_generative_miner( return response +async def get_current_kings( + hotkey, + base_url: str = "https://gas.bitmind.ai", +) -> Optional[Dict[str, Any]]: + """Fetch current KOTH kings from gas-api /validator/kings.""" + try: + bt.logging.info(f"Fetching current kings from {base_url}/api/v1/validator/kings") + timeout = aiohttp.ClientTimeout(total=30) + async with aiohttp.ClientSession(timeout=timeout) as session: + url = f"{base_url}/api/v1/validator/kings" + epistula_headers = generate_header(hotkey, b"", None) + async with session.get(url, headers=epistula_headers) as response: + if response.status == 200: + payload = await response.json() + bt.logging.info( + f"Fetched {len(payload.get('kings') or [])} current king(s)" + ) + return payload + error_text = await response.text() + bt.logging.warning( + f"Failed to fetch current kings: HTTP {response.status}, response: {error_text}" + ) + return None + except Exception as e: + bt.logging.error(f"Error fetching current kings from API: {e}") + bt.logging.error(traceback.format_exc()) + return None + + async def get_escrow_addresses( hotkey, base_url: str = "https://gas.bitmind.ai", diff --git a/neurons/discriminator/push_model.py b/neurons/discriminator/push_model.py index 481dfbba..9154cc69 100644 --- a/neurons/discriminator/push_model.py +++ b/neurons/discriminator/push_model.py @@ -54,6 +54,17 @@ class Style: MODEL_UPLOAD_ENDPOINT = "https://upload.bitmind.ai/upload" +DEFAULT_MAX_RETRIES = 3 + + +def should_retry_register(attempt: int, max_retries: int) -> bool: + """True if another chain-register try should run after this failed attempt. + + attempt is 1-based. max_retries <= 0 keeps the old unlimited loop. + """ + if max_retries <= 0: + return True + return attempt < max_retries def print_success(message: str): @@ -90,6 +101,23 @@ def _print_submission_count(result: dict, modality: str, hotkey: str): print_info(f"{count_after}/{submissions_max} {modality} models submitted for {hotkey}") +def _accept_already_uploaded(modality: str, result: dict) -> bool: + """Handle a 409 / already-accepted hash. True means this modality is done.""" + if result.get("already_uploaded"): + print_warning( + f"{modality.capitalize()} model already uploaded β€” server already has this hash." + ) + print_success( + f"{modality.capitalize()} upload already accepted; exam will still run." + ) + return True + print_error(f"{modality.capitalize()} model upload failed at step: {result.get('step', 'unknown')}") + print_error(f"Error: {result.get('error', 'Unknown error')}") + if result.get("response"): + print_error(f"Server response: {result['response']}") + return False + + async def push_separate_models( image_model_path: Optional[str] = None, video_model_path: Optional[str] = None, @@ -99,6 +127,9 @@ async def push_separate_models( netuid: int = 34, chain_endpoint: Optional[str] = None, vertical: str = "general", + upload_endpoint: Optional[str] = None, + skip_chain: bool = False, + max_retries: int = DEFAULT_MAX_RETRIES, ): """Pushes separate image, video, and/or audio detector models and registers on the Bittensor blockchain. @@ -114,6 +145,7 @@ async def push_separate_models( if audio_model_path and not os.path.exists(audio_model_path): raise FileNotFoundError(f"Audio model file not found: {audio_model_path}") + endpoint = upload_endpoint or MODEL_UPLOAD_ENDPOINT results = {} upload_count = 0 total_uploads = (1 if image_model_path else 0) + \ @@ -129,26 +161,17 @@ async def push_separate_models( wallet, image_model_path, 'image', - MODEL_UPLOAD_ENDPOINT, + endpoint, vertical=vertical, ) results['image'] = image_result if not image_result['success']: - if image_result.get('already_uploaded'): - print_warning("Image model already uploaded β€” skipping (model is already accepted by the server)") - # Still need r2_key for blockchain registration; server returned 409 so we - # don't have it here. The miner must use the r2_key from the original upload. - print_error("Cannot retrieve r2_key for already-uploaded image model. Re-run with the original r2_key or wait for the current upload to be processed.") + if not _accept_already_uploaded('image', image_result): return False - print_error(f"Image model upload failed at step: {image_result.get('step', 'unknown')}") - print_error(f"Error: {image_result.get('error', 'Unknown error')}") - if image_result.get('response'): - print_error(f"Server response: {image_result['response']}") - return False - - print_success("Image model uploaded successfully!") - _print_submission_count(image_result, 'image', wallet.hotkey.ss58_address) + else: + print_success("Image model uploaded successfully!") + _print_submission_count(image_result, 'image', wallet.hotkey.ss58_address) except Exception as e: print_error(f"Image model upload failed with exception: {e}") return False @@ -162,24 +185,17 @@ async def push_separate_models( wallet, video_model_path, 'video', - MODEL_UPLOAD_ENDPOINT, + endpoint, vertical=vertical, ) results['video'] = video_result if not video_result['success']: - if video_result.get('already_uploaded'): - print_warning("Video model already uploaded β€” skipping (model is already accepted by the server)") - print_error("Cannot retrieve r2_key for already-uploaded video model. Re-run with the original r2_key or wait for the current upload to be processed.") + if not _accept_already_uploaded('video', video_result): return False - print_error(f"Video model upload failed at step: {video_result.get('step', 'unknown')}") - print_error(f"Error: {video_result.get('error', 'Unknown error')}") - if video_result.get('response'): - print_error(f"Server response: {video_result['response']}") - return False - - print_success("Video model uploaded successfully!") - _print_submission_count(video_result, 'video', wallet.hotkey.ss58_address) + else: + print_success("Video model uploaded successfully!") + _print_submission_count(video_result, 'video', wallet.hotkey.ss58_address) except Exception as e: print_error(f"Video model upload failed with exception: {e}") return False @@ -193,28 +209,26 @@ async def push_separate_models( wallet, audio_model_path, 'audio', - MODEL_UPLOAD_ENDPOINT, + endpoint, vertical=vertical, ) results['audio'] = audio_result if not audio_result['success']: - if audio_result.get('already_uploaded'): - print_warning("Audio model already uploaded β€” skipping (model is already accepted by the server)") - print_error("Cannot retrieve r2_key for already-uploaded audio model. Re-run with the original r2_key or wait for the current upload to be processed.") + if not _accept_already_uploaded('audio', audio_result): return False - print_error(f"Audio model upload failed at step: {audio_result.get('step', 'unknown')}") - print_error(f"Error: {audio_result.get('error', 'Unknown error')}") - if audio_result.get('response'): - print_error(f"Server response: {audio_result['response']}") - return False - - print_success("Audio model uploaded successfully!") - _print_submission_count(audio_result, 'audio', wallet.hotkey.ss58_address) + else: + print_success("Audio model uploaded successfully!") + _print_submission_count(audio_result, 'audio', wallet.hotkey.ss58_address) except Exception as e: print_error(f"Audio model upload failed with exception: {e}") return False + if skip_chain: + print() + print_success("Upload complete. Skipping blockchain registration (--skip-chain).") + return True + print() print_step(upload_count + 1, total_uploads + 1, "Registering model metadata on blockchain...") @@ -226,11 +240,16 @@ async def push_separate_models( metadata_store = ChainModelMetadataStore(subtensor, netuid) # Register each model separately on the blockchain + chain_ok = True for modality, result in results.items(): model_key = result.get("r2_key", "") if not model_key: - print_error(f"{modality.capitalize()} model key not provided in upload response") - return False + print_warning( + f"Skipping {modality} chain registration β€” no r2_key in the upload " + "response. The upload already succeeded and the model will still be examined." + ) + chain_ok = False + continue # Create hash for this specific model model_hash = hashlib.sha256( @@ -243,7 +262,10 @@ async def push_separate_models( model_id = ModelId(key=model_key, hash=hash_value) print_info(f"Registering {modality} model (ID: {model_id.key})...") + attempt = 0 + registered = False while True: + attempt += 1 try: await metadata_store.store_model_metadata(wallet, model_id) @@ -265,14 +287,33 @@ async def push_separate_models( raise ValueError(f"{modality.capitalize()} metadata verification failed") print_success(f"{modality.capitalize()} model registered on blockchain!") - break # Success, move to next model + registered = True + break except Exception as e: print_error(f"Failed to register {modality} model on blockchain: {e}") - print_warning(f"Retrying in {retry_delay_secs} seconds...") - time.sleep(retry_delay_secs) + if should_retry_register(attempt, max_retries): + print_warning( + f"Retrying in {retry_delay_secs} seconds " + f"(attempt {attempt}/{max_retries if max_retries > 0 else 'unlimited'})..." + ) + time.sleep(retry_delay_secs) + continue + print_warning( + f"Giving up chain registration after {attempt} attempt(s). " + "The upload already succeeded and the model will still be examined. " + f"r2_key={model_key}" + ) + chain_ok = False + break - print_success("All models registered successfully!") + if not registered: + chain_ok = False + + if chain_ok: + print_success("All models registered successfully!") + else: + print_warning("Upload succeeded; one or more chain registrations did not complete.") return True @@ -327,6 +368,25 @@ def main(): choices=["general", "human"], help="Competition vertical (default: general)" ) + parser.add_argument( + "--upload-endpoint", + default=MODEL_UPLOAD_ENDPOINT, + help=f"Model upload URL (default: {MODEL_UPLOAD_ENDPOINT})", + ) + parser.add_argument( + "--skip-chain", + action="store_true", + help="Exit after a successful upload without writing on-chain metadata", + ) + parser.add_argument( + "--max-retries", + type=int, + default=DEFAULT_MAX_RETRIES, + help=( + "Max blockchain registration attempts per modality after a failed try " + f"(default: {DEFAULT_MAX_RETRIES}; 0 = retry forever)" + ), + ) args = parser.parse_args() @@ -351,6 +411,11 @@ def main(): print(f"Subnet UID: {args.netuid}") print(f"Chain Endpoint: {args.chain_endpoint}") print(f"Vertical: {args.vertical}") + print(f"Upload endpoint: {args.upload_endpoint}") + if args.skip_chain: + print("Chain registration: skipped") + else: + print(f"Chain register retries: {args.max_retries}") print() print(f"{Fore.CYAN}{Style.BRIGHT}=== Starting Model Push ==={Style.RESET_ALL}") @@ -363,14 +428,17 @@ def main(): print_error(f"Failed to initialize wallet: {e}") sys.exit(1) - # Check if hotkey is registered + # Opening a Metagraph websocket keeps this process alive after success. + # Only do it when we are about to write on-chain metadata. netuid = args.netuid - mg = bt.Metagraph(netuid=netuid, network="finney" if netuid == 34 else "test") - hotkey = wallet.hotkey.ss58_address - if hotkey not in mg.hotkeys: - print_warning(f"Hotkey {hotkey} not registered on netuid {netuid}") - print_warning("This may cause issues with model registration") - + if not args.skip_chain: + mg = bt.Metagraph(netuid=netuid, network="finney" if netuid == 34 else "test") + hotkey = wallet.hotkey.ss58_address + if hotkey not in mg.hotkeys: + print_warning(f"Hotkey {hotkey} not registered on netuid {netuid}") + print_warning("This may cause issues with model registration") + + exit_code = 0 try: success = asyncio.run( push_separate_models( @@ -382,6 +450,9 @@ def main(): netuid=netuid, chain_endpoint=args.chain_endpoint, vertical=args.vertical, + upload_endpoint=args.upload_endpoint, + skip_chain=args.skip_chain, + max_retries=args.max_retries, ) ) @@ -390,14 +461,17 @@ def main(): print_success("πŸŽ‰ Model push completed successfully!") else: print_error("πŸ’₯ Model push failed!") - sys.exit(1) + exit_code = 1 except Exception as e: print() print_error(f"πŸ’₯ Model push failed with error: {e}") print_info("Full traceback:") print(traceback.format_exc()) - sys.exit(1) + exit_code = 1 + + # Bittensor leaves non-daemon websocket threads; SystemExit will not end the process. + os._exit(exit_code) if __name__ == "__main__": diff --git a/neurons/validator/validator.py b/neurons/validator/validator.py index 87ec747e..359e2cba 100644 --- a/neurons/validator/validator.py +++ b/neurons/validator/validator.py @@ -10,8 +10,13 @@ import bittensor as bt from gas import __spec_version__ as spec_version -from gas.protocol.validator_requests import get_benchmark_results -from gas.protocol.validator_requests import get_escrow_addresses # noqa: F401 HOTFIX: temporarily unused +from gas.protocol.validator_requests import get_benchmark_results, get_current_kings +from gas.koth_weights import ( + build_koth_weights, + chains_by_modality, + discriminator_emissions_enabled, + kings_by_modality, +) from gas.utils.autoupdater import autoupdate from gas.cache import ContentManager from gas.utils.metagraph import create_set_weights @@ -40,12 +45,33 @@ MAINNET_UID = 34 -SS58_ADDRESSES = { - "burn": "5HjBSeeoz52CLfvDWDkzupqrYLHz1oToDPHjdmJjc4TF68LQ", - "video_escrow": "5G6BJ1Z6LeDptRn5GTw74QSDmG1FP3eqVque5JhUb5zeEyQa", - "image_escrow": "5EUJFyH4ZSSiD3C8sM698nsVE26Tq98LoBwkmopmWZqaZqCA", - "audio_escrow": "5F9Qo4jqurfx3qHsC2kQtvge7Si5aW1BfYKwpxnnpVxouPyF", -} +BURN_SS58 = "5HjBSeeoz52CLfvDWDkzupqrYLHz1oToDPHjdmJjc4TF68LQ" + + +class _KingsState: + """Persist last-known KOTH kings across validator restarts.""" + + def __init__(self): + self.payload = None + + def save_state(self, save_dir: str, filename: str) -> None: + import json + import os + + path = os.path.join(save_dir, filename) + with open(path, "w") as f: + json.dump(self.payload, f) + + def load_state(self, save_dir: str, filename: str) -> bool: + import json + import os + + path = os.path.join(save_dir, filename) + if not os.path.exists(path): + return False + with open(path) as f: + self.payload = json.load(f) + return True class Validator(BaseNeuron): @@ -77,6 +103,7 @@ def init(self): ## Typesafety self.set_weights_fn = create_set_weights(spec_version, self.config.netuid) self.scores = np.zeros(self.metagraph.n, dtype=np.float32) + self.kings_state = _KingsState() bt.logging.info(f"Initialized scores vector for {len(self.scores)} miners") if not self.config.wandb_off: @@ -175,10 +202,25 @@ async def set_weights(self, block): generator_uids = [] bt.logging.warning("No generator rewards available; using empty generator_uids") - # HOTFIX: API unstable; always use hardcoded escrow addresses. - bt.logging.info("HOTFIX: using hardcoded default escrow addresses") - active_ss58_addresses = SS58_ADDRESSES - + kings_payload = await get_current_kings( + self.wallet.hotkey, base_url=self.config.benchmark_api_url + ) + if kings_payload is not None: + self.kings_state.payload = kings_payload + elif self.kings_state.payload is not None: + bt.logging.warning("kings API unavailable; using last known kings") + kings_payload = self.kings_state.payload + else: + bt.logging.warning( + "kings API unavailable and no cached kings; " + "discriminator shares will burn" + ) + kings_payload = {"kings": []} + + kings = kings_by_modality(kings_payload) + chains = chains_by_modality(kings_payload) + split = (kings_payload or {}).get("split") + async with self._state_lock: bt.logging.debug("set_weights() acquired state lock") try: @@ -194,73 +236,35 @@ async def set_weights(self, block): "responses from miners, or a bug in your reward functions." ) - # Weight budget (must sum to 1.0) - burn_pct = 0.84 - video_pct = 0. #.4 - image_pct = 0. #.4 - audio_pct = 0. #.04 - generator_pct = 0.16 - - # Resolve escrow/burn UIDs at current chain head. `block` is an - # interval marker (0 at startup) and must not be used as a query - # block: a non-archive node has pruned old state and raises - # StateDiscardedError under async-substrate-interface 2.x. - burn_uid = self.subtensor.get_uid_for_hotkey_on_subnet( - hotkey_ss58=active_ss58_addresses["burn"], - netuid=self.config.netuid, - ) - video_escrow_uid = self.subtensor.get_uid_for_hotkey_on_subnet( - hotkey_ss58=active_ss58_addresses["video_escrow"], - netuid=self.config.netuid, - ) - image_escrow_uid = self.subtensor.get_uid_for_hotkey_on_subnet( - hotkey_ss58=active_ss58_addresses["image_escrow"], - netuid=self.config.netuid, - ) - audio_escrow_uid = self.subtensor.get_uid_for_hotkey_on_subnet( - hotkey_ss58=active_ss58_addresses["audio_escrow"], - netuid=self.config.netuid, + def uid_for_hotkey(hotkey_ss58: str): + try: + return self.subtensor.get_uid_for_hotkey_on_subnet( + hotkey_ss58=hotkey_ss58, + netuid=self.config.netuid, + ) + except Exception as e: + bt.logging.warning(f"Could not resolve UID for {hotkey_ss58[:8]}...: {e}") + return None + + burn_uid = uid_for_hotkey(BURN_SS58) + normed_weights = build_koth_weights( + n=int(self.metagraph.n), + scores=self.scores, + generator_uids=generator_uids, + kings=kings, + uid_for_hotkey=uid_for_hotkey, + burn_uid=burn_uid, + split=split, + chains=chains, + emissions_enabled=discriminator_emissions_enabled(kings_payload), ) - # Escrow hotkeys can be unregistered (uid=None). Do not index - # numpy with None β€” that broadcasts and zeros the whole vector. - special_uids = { - uid - for uid in (burn_uid, image_escrow_uid, video_escrow_uid, audio_escrow_uid) - if uid is not None - } - - # Compute norm excluding specials - norm = np.ones_like(self.scores) - active_uids = [uid for uid in generator_uids if uid not in special_uids] - if active_uids: - norm[active_uids] = np.linalg.norm(self.scores[active_uids], ord=1) - - if np.any(norm == 0) or np.isnan(norm).any(): - norm = np.ones_like(norm) - - normed_weights = self.scores / norm - - active_uids_set = set(active_uids) - for uid in range(len(normed_weights)): - if uid not in special_uids and uid not in active_uids_set: - normed_weights[uid] = 0.0 - - active_mask = np.array([uid in active_uids_set for uid in range(len(normed_weights))]) - normed_weights[active_mask] *= generator_pct - - normed_weights[burn_uid] = burn_pct - if video_escrow_uid is not None: - normed_weights[video_escrow_uid] = video_pct - if image_escrow_uid is not None: - normed_weights[image_escrow_uid] = image_pct - if audio_escrow_uid is not None: - normed_weights[audio_escrow_uid] = audio_pct - - # Verify allocations - total_weight = np.sum(normed_weights) - actual_burn_rate = normed_weights[burn_uid] / total_weight if total_weight > 0 else 0 - bt.logging.info(f"Total weight sum: {total_weight:.4f}, Actual burn rate: {actual_burn_rate:.4f} (target: {burn_pct})") + total_weight = float(np.sum(normed_weights)) + bt.logging.info( + f"KOTH weights sum={total_weight:.4f} kings={list(kings.keys())} " + f"chain={ {mod: [m.get('role') for m in members] for mod, members in chains.items()} } " + f"generators={len(generator_uids)}" + ) self.set_weights_fn( self.wallet, self.metagraph, self.subtensor, (uids, normed_weights) @@ -401,7 +405,8 @@ async def save_state(self): try: state_data = {"scores.npy": self.scores} state_objects = [ - (self.generative_challenge_manager, "challenge_tasks.pkl") + (self.generative_challenge_manager, "challenge_tasks.pkl"), + (self.kings_state, "kings.json"), ] success = save_validator_state( @@ -427,7 +432,8 @@ async def load_state(self): try: state_data_keys = ["scores.npy"] state_objects = [ - (self.generative_challenge_manager, "challenge_tasks.pkl") + (self.generative_challenge_manager, "challenge_tasks.pkl"), + (self.kings_state, "kings.json"), ] loaded_state = load_validator_state( diff --git a/tests/test_koth_warmup.py b/tests/test_koth_warmup.py new file mode 100644 index 00000000..2858b545 --- /dev/null +++ b/tests/test_koth_warmup.py @@ -0,0 +1,69 @@ +"""Activation metadata gates discriminator payouts without changing generators.""" + +from datetime import datetime, timedelta, timezone + +import numpy as np +import pytest + +from gas.koth_weights import ( + build_koth_weights, + chains_by_modality, + discriminator_emissions_enabled, + kings_by_modality, +) + + +@pytest.mark.parametrize("metadata", [ + None, + {}, + {"emissions_enabled": True}, + {"emissions_enabled": "true", "emissions_start_at": "2026-09-01T00:00:00Z"}, + {"emissions_enabled": True, "emissions_start_at": "invalid"}, + {"emissions_enabled": True, "emissions_start_at": "2026-09-01T00:00:00"}, + {"emissions_enabled": True, "emissions_start_at": 123}, +]) +def test_missing_or_malformed_activation_keeps_burning(metadata): + assert not discriminator_emissions_enabled(metadata) + + +def test_activation_requires_elapsed_boundary_and_explicit_enabled_response(): + boundary = datetime(2026, 9, 1, tzinfo=timezone.utc) + cached = {"emissions_enabled": False, "emissions_start_at": boundary.isoformat()} + assert not discriminator_emissions_enabled(cached, now=boundary + timedelta(days=1)) + enabled = {**cached, "emissions_enabled": True} + assert not discriminator_emissions_enabled(enabled, now=boundary - timedelta(seconds=1)) + assert discriminator_emissions_enabled(enabled, now=boundary) + assert discriminator_emissions_enabled(enabled, now=boundary + timedelta(seconds=1)) + + +def test_warmup_burns_kings_and_residuals_then_resumes_without_changing_generators(): + boundary = datetime(2026, 9, 1, tzinfo=timezone.utc) + payload = { + "emissions_start_at": boundary.isoformat(), + "emissions_enabled": False, + "kings": [{"modality": "image", "ss58_address": "current"}], + "chain": {"image": ["current", "previous"]}, + } + args = dict( + n=5, scores=np.array([0., 0., 0., 2., 1.]), generator_uids=[3, 4], + kings=kings_by_modality(payload), chains=chains_by_modality(payload), + uid_for_hotkey={"current": 1, "previous": 2}.get, burn_uid=0, + split={"image": .3, "video": .2, "audio": .1, "generator": .4}, + ) + warmup = build_koth_weights( + **args, emissions_enabled=discriminator_emissions_enabled(payload, now=boundary) + ) + payload["emissions_enabled"] = True + live = build_koth_weights( + **args, emissions_enabled=discriminator_emissions_enabled(payload, now=boundary) + ) + assert warmup[1] == warmup[2] == 0 + assert warmup[0] == pytest.approx(.6) + assert live[1] > 0 and live[2] > 0 + assert live[1] + live[2] == pytest.approx(.3) + assert live[0] == pytest.approx(.3) # Vacant video/audio lanes still burn. + np.testing.assert_allclose(warmup[3:], live[3:]) + assert warmup.sum() == pytest.approx(1) + assert live.sum() == pytest.approx(1) + with pytest.raises(ValueError, match="burn UID is unavailable"): + build_koth_weights(**{**args, "burn_uid": None}, emissions_enabled=False) diff --git a/tests/test_koth_weights.py b/tests/test_koth_weights.py new file mode 100644 index 00000000..1b2cb3c1 --- /dev/null +++ b/tests/test_koth_weights.py @@ -0,0 +1,171 @@ +"""Unit tests for KOTH validator weight vectors.""" + +import numpy as np +import pytest + +from gas.koth_weights import ( + assign_residual_shares, + build_koth_weights, + chains_by_modality, + kings_by_modality, +) + +ESCROW = { + "image": "5EUJFyH4ZSSiD3C8sM698nsVE26Tq98LoBwkmopmWZqaZqCA", + "video": "5G6BJ1Z6LeDptRn5GTw74QSDmG1FP3eqVque5JhUb5zeEyQa", + "audio": "5F9Qo4jqurfx3qHsC2kQtvge7Si5aW1BfYKwpxnnpVxouPyF", +} + + +def test_kings_by_modality_reads_hotkeys(): + payload = { + "kings": [ + {"modality": "image", "ss58_address": "5Img"}, + {"modality": "video", "ss58_address": "5Vid"}, + ] + } + assert kings_by_modality(payload) == {"image": "5Img", "video": "5Vid"} + assert kings_by_modality(None) == {} + + +def test_weights_go_to_king_uids_not_escrow(): + hotkeys = {"5Img": 1, "5Vid": 2, "5Aud": 3, "5Burn": 0} + kings = {"image": "5Img", "video": "5Vid", "audio": "5Aud"} + scores = np.array([0.0, 0.0, 0.0, 0.0, 2.0, 1.0]) + + weights = build_koth_weights( + n=6, + scores=scores, + generator_uids=[4, 5], + kings=kings, + uid_for_hotkey=hotkeys.get, + burn_uid=0, + ) + + assert abs(weights[1] - 0.40) < 1e-9 + assert abs(weights[2] - 0.40) < 1e-9 + assert abs(weights[3] - 0.04) < 1e-9 + assert abs(weights[4] - 0.16 * 2 / 3) < 1e-9 + assert abs(weights[5] - 0.16 * 1 / 3) < 1e-9 + assert weights[0] == 0.0 + for escrow in ESCROW.values(): + assert escrow not in kings.values() + assert hotkeys.get(escrow) is None + + +def test_missing_king_burns_that_lane(): + weights = build_koth_weights( + n=4, + scores=np.zeros(4), + generator_uids=[], + kings={"image": "5Img"}, + uid_for_hotkey=lambda hk: 1 if hk == "5Img" else None, + burn_uid=0, + ) + assert abs(weights[1] - 0.40) < 1e-9 + assert abs(weights[0] - 0.60) < 1e-9 # video + audio + generator + + +def test_api_down_empty_kings_burns_discriminator_shares(): + weights = build_koth_weights( + n=3, + scores=np.array([0.0, 4.0, 0.0]), + generator_uids=[1], + kings={}, + uid_for_hotkey=lambda hk: None, + burn_uid=0, + ) + assert abs(weights[0] - 0.84) < 1e-9 + assert abs(weights[1] - 0.16) < 1e-9 + + +def test_required_burn_fails_when_burn_uid_is_unavailable(): + with pytest.raises(ValueError, match="burn UID is unavailable"): + build_koth_weights( + n=3, + scores=np.array([0.0, 1.0, 0.0]), + generator_uids=[1], + kings={}, + uid_for_hotkey=lambda hk: None, + burn_uid=None, + ) + + +@pytest.mark.parametrize( + "split", + [ + {"image": 0.4, "video": 0.4, "audio": -0.01, "generator": 0.21}, + {"image": 0.4, "video": 0.4, "audio": 0.04}, + ], +) +def test_invalid_split_is_rejected(split): + with pytest.raises(ValueError, match="Invalid KOTH split"): + build_koth_weights( + n=1, + scores=np.zeros(1), + generator_uids=[], + kings={}, + uid_for_hotkey=lambda hk: None, + burn_uid=0, + split=split, + ) + + +def test_residual_rolls_unused_slots_to_current(): + assert assign_residual_shares(["5A"]) == [ + {"ss58_address": "5A", "share": 1.0, "role": "current"} + ] + assert assign_residual_shares(["5A", "5B"]) == [ + {"ss58_address": "5A", "share": 0.90, "role": "current"}, + {"ss58_address": "5B", "share": 0.10, "role": "previous"}, + ] + + +def test_chains_by_modality_recomputes_shares_and_falls_back(): + payload = { + "kings": [{"modality": "audio", "ss58_address": "5Aud"}], + "chain": { + "image": [ + {"ss58_address": "5Img", "share": 0.5, "role": "current"}, + {"ss58_address": "5Prev", "share": 0.5, "role": "previous"}, + ] + }, + } + chains = chains_by_modality(payload) + assert [member["share"] for member in chains["image"]] == [0.90, 0.10] + assert chains["audio"][0]["ss58_address"] == "5Aud" + assert chains["audio"][0]["share"] == 1.0 + + +def test_lane_residual_splits_across_last_three_kings(): + hotkeys = {"5Img": 1, "5Prev": 2, "5Two": 3, "5Burn": 0} + weights = build_koth_weights( + n=5, + scores=np.zeros(5), + generator_uids=[], + kings={"image": "5Img"}, + uid_for_hotkey=hotkeys.get, + burn_uid=0, + chains={ + "image": assign_residual_shares(["5Img", "5Prev", "5Two"]), + }, + ) + assert abs(weights[1] - 0.40 * 0.85) < 1e-9 + assert abs(weights[2] - 0.40 * 0.10) < 1e-9 + assert abs(weights[3] - 0.40 * 0.05) < 1e-9 + assert abs(weights[0] - 0.60) < 1e-9 # video + audio + generator + + +def test_unresolvable_previous_king_rolls_to_current(): + hotkeys = {"5Img": 1, "5Burn": 0} + weights = build_koth_weights( + n=3, + scores=np.zeros(3), + generator_uids=[], + kings={"image": "5Img"}, + uid_for_hotkey=hotkeys.get, + burn_uid=0, + chains={"image": assign_residual_shares(["5Img", "5Gone"])}, + ) + assert abs(weights[1] - 0.40) < 1e-9 + assert abs(weights[0] - 0.60) < 1e-9 diff --git a/tests/test_push_model_retries.py b/tests/test_push_model_retries.py new file mode 100644 index 00000000..50e37d63 --- /dev/null +++ b/tests/test_push_model_retries.py @@ -0,0 +1,96 @@ +import asyncio +from types import SimpleNamespace + +import neurons.discriminator.push_model as push_model +from neurons.discriminator.push_model import ( + _accept_already_uploaded, + push_separate_models, + should_retry_register, +) + + +def test_default_retries_stop_after_three(): + assert should_retry_register(1, 3) is True + assert should_retry_register(2, 3) is True + assert should_retry_register(3, 3) is False + + +def test_zero_retries_is_unlimited(): + assert should_retry_register(1, 0) is True + assert should_retry_register(99, 0) is True + + +def test_already_uploaded_is_success(): + result = {"already_uploaded": True, "success": False} + assert _accept_already_uploaded("image", result) is True + + +def test_chain_registration_failure_does_not_fail_push(monkeypatch, tmp_path): + model_path = tmp_path / "model.zip" + model_path.touch() + wallet = SimpleNamespace(hotkey=SimpleNamespace(ss58_address="5Miner")) + + monkeypatch.setattr( + push_model, + "upload_single_modality", + lambda *args, **kwargs: { + "success": True, + "r2_key": "model-key", + "file_hash": "file-hash", + }, + ) + monkeypatch.setattr(push_model.bt, "Subtensor", lambda **kwargs: object()) + + class FailingMetadataStore: + async def store_model_metadata(self, wallet, model_id): + raise RuntimeError("chain unavailable") + + monkeypatch.setattr( + push_model, + "ChainModelMetadataStore", + lambda subtensor, netuid: FailingMetadataStore(), + ) + + success = asyncio.run( + push_separate_models( + image_model_path=str(model_path), + wallet=wallet, + retry_delay_secs=0, + max_retries=1, + ) + ) + + assert success is True + + +def test_already_uploaded_without_r2_key_still_succeeds(monkeypatch, tmp_path): + model_path = tmp_path / "model.zip" + model_path.touch() + wallet = SimpleNamespace(hotkey=SimpleNamespace(ss58_address="5Miner")) + + monkeypatch.setattr( + push_model, + "upload_single_modality", + lambda *args, **kwargs: { + "success": False, + "already_uploaded": True, + "file_hash": "file-hash", + }, + ) + monkeypatch.setattr(push_model.bt, "Subtensor", lambda **kwargs: object()) + monkeypatch.setattr( + push_model, + "ChainModelMetadataStore", + lambda subtensor, netuid: object(), + ) + + success = asyncio.run( + push_separate_models( + image_model_path=str(model_path), + wallet=wallet, + retry_delay_secs=0, + max_retries=1, + ) + ) + + assert success is True diff --git a/validator.config.js b/validator.config.js index a622e668..b2436421 100644 --- a/validator.config.js +++ b/validator.config.js @@ -72,8 +72,13 @@ const config = { startData: process.env.START_DATA !== 'false', }; -// Determine netuid -const netuid = getNetworkSettings(config.chainEndpoint); +// Determine netuid. NETUID wins so testnet 169 is not forced to 379. +const netuid = process.env.NETUID + ? parseInt(process.env.NETUID, 10) + : getNetworkSettings(config.chainEndpoint); +if (!netuid) { + throw new Error('NETUID is required (set NETUID or a known CHAIN_ENDPOINT)'); +} // Build command parameters const logParam = getLogParam(config.loglevel); @@ -131,6 +136,9 @@ if (config.startValidator) { logParam, autoUpdateParam, ]; + if (process.env.EPOCH_LENGTH) { + validatorArgs.push('--epoch-length', process.env.EPOCH_LENGTH); + } // Add external callback port if provided if (config.externalCallbackPort) {