diff --git a/README.md b/README.md index 1e51d44a..f8794c7e 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ GAS runs two parallel competition tracks on Bittensor Subnet 34: - **Cloud-evaluated**: Discriminator models are benchmarked on cloud infrastructure -- no GPU hosting required - **Model format**: Safetensors only (ONNX submissions are not accepted) - **Datasets refresh weekly** with fresh GAS-Station data alongside static benchmarks -- **One model per modality per hotkey** for discriminative miners +- **One model per hotkey** for discriminative miners See [Incentive Mechanism](docs/Incentive.md) for full scoring details. diff --git a/docs/Discriminative-Mining.md b/docs/Discriminative-Mining.md index d7984e58..4860b8d9 100644 --- a/docs/Discriminative-Mining.md +++ b/docs/Discriminative-Mining.md @@ -89,6 +89,15 @@ gascli d push \ At least one model (image, video, or audio) must be provided. +## Submission Limits + +Each registered hotkey gets **one counted submission** (image, video, or audio β€” not one of each). + +- Exam failures and incomplete uploads do not consume the slot. You can retry on the same key until a model is successfully uploaded and not later marked exam-failed. +- A confirmed or superseded model **does** consume the slot for the life of that hotkey, for every modality. +- A new benchmark version does **not** refill the slot. +- To submit another model, register a new miner hotkey. + --- ## Competition Rules and Constraints @@ -104,7 +113,7 @@ The normalized terms apply exponents $1.2$ to MCC performance and $1.8$ to Brier ### Model Requirements - **Format**: Safetensors only (ONNX is no longer accepted) -- **Three model per modality per hotkey**: You can submit up to three image, three video, and three audio model per registered hotkey +- **Submission cap**: one counted model per hotkey (see [Submission Limits](#submission-limits)) ### Sandbox and Import Restrictions @@ -194,7 +203,7 @@ Models that pass the entrance exam are benchmarked against the **complete datase - **Private holdout datasets** β€” curated datasets not visible to miners, used to prevent overfitting to the public benchmark set - Refreshed weekly with new data from the GAS-Station pipeline -The full benchmark has a **maximum wall-clock timeout of 5 hours** (18,000 seconds) per modality. The benchmark score from this stage determines your **TAO emissions** on Subnet 34. The active round configuration selects provenance weighting, multiclass scoring, and augmentation robustness parameters; see [Incentive Mechanism](Incentive.md). +The full benchmark has a **maximum wall-clock timeout of 5 hours** (18,000 seconds) per modality. This `sn34_score` is what the King of the Hill competition uses: a high enough score can take or keep a lane, and emissions then follow the 85/10/5 split on the current king plus the previous two. The active round configuration selects provenance weighting, multiclass scoring, and augmentation robustness parameters; see [Incentive Mechanism](Incentive.md). You can simulate a full benchmark run locally (without holdouts) to get a sense of your model's performance: diff --git a/docs/Incentive.md b/docs/Incentive.md index b83ac87e..6aa51365 100644 --- a/docs/Incentive.md +++ b/docs/Incentive.md @@ -140,7 +140,7 @@ The normative implementation details and complete metric field glossary live in ### King of the Hill -Discriminator emission is King of the Hill. Each modality has one reigning model. Validators set that lane's weight on registered hotkeys every tempo β€” not on an escrow wallet. +Discriminator emission is King of the Hill. Each modality has one reigning model. Validators set that lane's weight on registered hotkeys every tempo β€” not on an escrow wallet. Each hotkey may land **one counted submission** for the life of that registration (any modality; exam failures do not count; a new model needs a new key). Current split: diff --git a/docs/Mining.md b/docs/Mining.md index d8eacc89..829bf5cb 100644 --- a/docs/Mining.md +++ b/docs/Mining.md @@ -3,7 +3,7 @@ GAS supports two types of miners that work together in an adversarial loop: ## [Discriminative Mining](Discriminative-Mining.md) πŸ“– -Miners submit classifiers that detect AI-generated content across **image, video, and audio** modalities. Models are evaluated on cloud infrastructure against diverse benchmark datasets and scored using the `sn34_score` metric (accuracy + calibration). +Miners submit classifiers that detect AI-generated content across **image, video, and audio** modalities. Models are evaluated on cloud infrastructure against diverse benchmark datasets and scored using the `sn34_score` metric (accuracy + calibration). Each hotkey gets one counted submission (any modality); see [Discriminative Mining](Discriminative-Mining.md#submission-limits). ## [Generative Mining](Generative-Mining.md) 🎨 Miners create synthetic media (images and videos) that challenges the discriminators. They generate increasingly realistic content to test and improve detection capabilities, and are rewarded based on validation pass rate and adversarial performance. diff --git a/docs/Validating-New.md b/docs/Validating-New.md deleted file mode 100644 index ad8e4c6f..00000000 --- a/docs/Validating-New.md +++ /dev/null @@ -1,50 +0,0 @@ -# Validator Guide - -Run the SN34 validator (validator + generator + data services). Two ways: **PM2** (native) or **Docker**. - -Prerequisites: [Installation Guide](Installation.md), Bittensor wallet, GPU for generation. - ---- - -## Quick start: PM2 - -```bash -cp .env.validator.template .env.validator -# Edit .env.validator: WALLET_NAME, WALLET_HOTKEY, API keys, CHAIN_ENDPOINT -source .venv/bin/activate -gascli validator start -``` - -That starts three PM2 processes: `sn34-validator`, `sn34-generator`, `sn34-data`. Use `gascli v stop|status|logs|--help` as needed. - ---- - -## Quick start: Docker - -Three containers (validator, generator, data) share one image and volumes; one process per container. - -```bash -cp .env.validator.template .env.validator -# Edit .env.validator (same as above; WALLET_PATH is for host wallet bind-mount) -docker compose --env-file .env.validator up -d --build -docker compose logs -f validator # or generator, data -``` - -Always use `--env-file .env.validator`. First run downloads 100+ GB of models into the `hf-cache` volume. - -**Update:** Manual: `git pull && docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d`. Automatic: cron with `docker/autoupdate.sh` (see [.env.validator.template](../.env.validator.template) comments for crontab example). - ---- - -## Config - -One file for both PM2 and Docker: `.env.validator`. Copy from [.env.validator.template](../.env.validator.template) and fill in wallet, API keys, `CHAIN_ENDPOINT`. The template lists every option and Docker-specific notes (e.g. `WALLET_PATH`, cache paths, autoupdate cron). - ---- - -## Reference - -| Path | Start | Stop / logs | -|--------|-------|-------------| -| PM2 | `gascli validator start` | `gascli v stop` / `gascli v logs` | -| Docker | `docker compose --env-file .env.validator up -d` | `docker compose down` / `docker compose logs -f validator` (or `generator`, `data`) | diff --git a/docs/Validating.md b/docs/Validating.md index 23a18aa9..ad8e4c6f 100644 --- a/docs/Validating.md +++ b/docs/Validating.md @@ -1,169 +1,50 @@ # Validator Guide -## Required .env.validator variables (PM2 and Docker) +Run the SN34 validator (validator + generator + data services). Two ways: **PM2** (native) or **Docker**. -You **must** set these in `.env.validator` before starting; the rest have defaults in the template. - -| Variable | Required? | Notes | -|----------|-----------|--------| -| `WALLET_NAME` | **Yes** | Your Bittensor wallet name (e.g. `default` or a name you created). | -| `WALLET_HOTKEY` | **Yes** | Hotkey for this validator (e.g. `default`). Must be registered on the subnet. | -| `HUGGINGFACE_HUB_TOKEN` | **Yes** | Needed to download models and upload data. Create at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). Reach out to a member of the BitMind team to have your account added to the `gasstation` org | -| `CHAIN_ENDPOINT` | No | Default is mainnet. Set to testnet URL if needed. | -| `WANDB_API_KEY` | No | Optional; for Weights & Biases logging. | -| Cache / Docker paths | No | `SN34_CACHE_DIR`, `HF_HOME`, `WALLET_PATH`, `BT_LOGGING_LOGGING_DIR` have defaults; override if you use different paths. | - -See [.env.validator.template](../.env.validator.template) for every option. +Prerequisites: [Installation Guide](Installation.md), Bittensor wallet, GPU for generation. --- -## PM2 setup - -Follow the [Installation Guide](Installation.md) to set up your environment before proceeding with validator setup. - -> Create a `.env.validator` file and **set the required variables above** (wallet, API keys). Then: - -```bash -$ cp .env.validator.template .env.validator -# Edit .env.validator: fill in WALLET_NAME, WALLET_HOTKEY, HUGGINGFACE_HUB_TOKEN at minimum -``` - -See [.env.validator.template](../.env.validator.template) for all options. Then activate the virtual environment and start your validator processes: - -```bash -$ source .venv/bin/activate -$ gascli validator start -``` -The above command will create 3 pm2 processes: -```bash -β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ id β”‚ name β”‚ namespace β”‚ version β”‚ mode β”‚ pid β”‚ uptime β”‚ β†Ί β”‚ status β”‚ cpu β”‚ mem β”‚ user β”‚ watching β”‚ -β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ 2 β”‚ sn34-data β”‚ default β”‚ N/A β”‚ fork β”‚ 4032914 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 529.0mb β”‚ user β”‚ disabled β”‚ -β”‚ 1 β”‚ sn34-generator β”‚ default β”‚ N/A β”‚ fork β”‚ 4032936 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 448.5mb β”‚ user β”‚ disabled β”‚ -β”‚ 0 β”‚ sn34-validator β”‚ default β”‚ N/A β”‚ fork β”‚ 4032918 β”‚ 2s β”‚ 72 β”‚ online β”‚ 100% β”‚ 504.0mb β”‚ user β”‚ disabled β”‚ -β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` -- **sn34-data**: Handles data downloads -- **sn34-generator**: Responsible for generating prompts, synthetic media, and validating miner-generated data -- **sn34-validator**: Core validator logic. Challenges, scoring, weight setting. - +## Quick start: PM2 -### Validator Operations - -First, activate the virtual environment: ```bash +cp .env.validator.template .env.validator +# Edit .env.validator: WALLET_NAME, WALLET_HOTKEY, API keys, CHAIN_ENDPOINT source .venv/bin/activate -``` - -Then run validator commands: -```bash -# Start validator services gascli validator start -gascli v start # Using alias -gascli v stop -gascli v status -gascli v logs -gascli v --help ``` +That starts three PM2 processes: `sn34-validator`, `sn34-generator`, `sn34-data`. Use `gascli v stop|status|logs|--help` as needed. -## Docker Deployment - -As an alternative to the PM2-based setup above, you can run the validator stack in Docker. Three containers (validator, generator, data) run one process each and share bind-mounted cache and wallet; the validator container uses `network_mode: host` so its FastAPI callback is on the host port miners are told to hit. - -### Prerequisites - -- **Docker** (with Docker Compose v2) -- **NVIDIA Container Toolkit** for GPU passthrough. Install guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html -- **Bittensor wallet** files already created on the host (typically at `~/.bittensor/wallets/`) - -### Quick Start - -1. **Clone the repository** (if not already done): - ```bash - git clone - cd bitmind-subnet - ``` - -2. **Create your `.env.validator`** and set the **required variables** (same as PM2): - ```bash - cp .env.validator.template .env.validator - ``` - You **must** set: `WALLET_NAME`, `WALLET_HOTKEY`, `HUGGINGFACE_HUB_TOKEN`, `OPEN_ROUTER_API_KEY`. See [Required .env.validator variables](#required-envvalidator-variables-pm2-and-docker) and [.env.validator.template](../.env.validator.template) for the full list. - -3. **Build and start** (use `--env-file .env.validator` so the same file drives both container env and Compose options like `WALLET_PATH` and `CALLBACK_PORT`): - ```bash - docker compose --env-file .env.validator up -d --build - ``` - -4. **View logs** (per service): - ```bash - docker compose logs -f validator # or generator, data - ``` - - -### Updating +--- -**Automatic (recommended):** add a crontab entry so `docker/autoupdate.sh` runs periodically (it checks VERSION, then pulls, runs `down --remove-orphans`, rebuilds, and brings the stack up): +## Quick start: Docker -```bash -*/5 * * * * /path/to/bitmind-subnet/docker/autoupdate.sh >> /var/log/bitmind-docker-update.log 2>&1 -``` - -**Manual:** rebuild and recreate all services: +Three containers (validator, generator, data) share one image and volumes; one process per container. ```bash -git pull -docker compose --env-file .env.validator down --remove-orphans -docker compose --env-file .env.validator build -docker compose --env-file .env.validator up -d +cp .env.validator.template .env.validator +# Edit .env.validator (same as above; WALLET_PATH is for host wallet bind-mount) +docker compose --env-file .env.validator up -d --build +docker compose logs -f validator # or generator, data ``` -### Configuration +Always use `--env-file .env.validator`. First run downloads 100+ GB of models into the `hf-cache` volume. -| Variable | Docker behavior | -|---|---| -| `SN34_CACHE_DIR` | Bind-mounted into the container; same path as PM2 so cache is shared when you switch. Set in `.env.validator`; use `--env-file .env.validator`. | -| `HF_HOME` | Bind-mounted into the container; same path as PM2 so model cache is shared. Set in `.env.validator`. | -| `AUTO_UPDATE` | In-container autoupdate is disabled. Use manual rebuild or the cron-based `docker/autoupdate.sh` (see Updating). | -| `WALLET_PATH` | Host base path for wallets. Only the directory `WALLET_PATH/WALLET_NAME` is mounted (not the whole wallets folder). Set in `.env.validator`. | -| `WALLET_NAME` | Which wallet dir to mount; with `WALLET_PATH` only that wallet is visible to the container. | -| `BT_LOGGING_LOGGING_DIR` | Base path for validator state (scores, challenge tasks) and bittensor logs. State lives under `BT_LOGGING_LOGGING_DIR`/`WALLET_NAME`/`WALLET_HOTKEY`/…; bind-mounted so it persists. Default `~/.bittensor`. | -| `NETUID` | Auto-derived from `CHAIN_ENDPOINT`. Set explicitly if using a custom endpoint. | +**Update:** Manual: `git pull && docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d`. Automatic: cron with `docker/autoupdate.sh` (see [.env.validator.template](../.env.validator.template) comments for crontab example). -### Wallet, cache, and validator state - -- **Wallet:** Only the configured wallet directory (`WALLET_PATH`/`WALLET_NAME`) is bind-mounted, not the entire wallets folder. -- **Cache:** `SN34_CACHE_DIR` and `HF_HOME` are bind-mounted so PM2 and Docker share the same data (no re-download when switching). -- **Validator state:** Scores and challenge tasks are saved under bittensor’s logging path (`BT_LOGGING_LOGGING_DIR`/`WALLET_NAME`/`WALLET_HOTKEY`/…). That path is bind-mounted so state persists across container restarts. - -> **Note**: The first startup will download 100+ GB of ML models into the HF cache directory. Subsequent restarts reuse the cached models. - -### Common Operations - -Use `--env-file .env.validator` so Compose reads `WALLET_PATH`, `WALLET_NAME`, `CALLBACK_PORT`, and cache paths from your config: - -```bash -# Start all three services (validator, generator, data) -docker compose --env-file .env.validator up -d - -# Stop all -docker compose down +--- -# View logs (per service) -docker compose logs -f validator # or generator, data +## Config -# Rebuild after code changes -docker compose --env-file .env.validator down --remove-orphans -docker compose --env-file .env.validator build && docker compose --env-file .env.validator up -d +One file for both PM2 and Docker: `.env.validator`. Copy from [.env.validator.template](../.env.validator.template) and fill in wallet, API keys, `CHAIN_ENDPOINT`. The template lists every option and Docker-specific notes (e.g. `WALLET_PATH`, cache paths, autoupdate cron). -# Restart one service -docker compose restart validator # or generator, data +--- -# Container status -docker compose ps +## Reference -# Shell into a container -docker compose exec validator bash # or generator, data -``` +| Path | Start | Stop / logs | +|--------|-------|-------------| +| PM2 | `gascli validator start` | `gascli v stop` / `gascli v logs` | +| Docker | `docker compose --env-file .env.validator up -d` | `docker compose down` / `docker compose logs -f validator` (or `generator`, `data`) | diff --git a/gas/protocol/miner_requests.py b/gas/protocol/miner_requests.py index dd9d3ab8..3a00e474 100644 --- a/gas/protocol/miner_requests.py +++ b/gas/protocol/miner_requests.py @@ -279,6 +279,7 @@ def extract_error(result: dict) -> str: "error": extract_error(presigned_result), "response": presigned_result['response'], "already_uploaded": True, + "file_hash": file_hash, } return { "success": False, diff --git a/neurons/discriminator/push_model.py b/neurons/discriminator/push_model.py index 9486bc39..a90660c0 100644 --- a/neurons/discriminator/push_model.py +++ b/neurons/discriminator/push_model.py @@ -107,16 +107,10 @@ def _accept_already_uploaded(modality: str, result: dict, skip_chain: bool) -> b print_warning( f"{modality.capitalize()} model already uploaded β€” server already has this hash." ) - if skip_chain: - print_success( - f"{modality.capitalize()} upload already accepted; nothing more to do (--skip-chain)." - ) - return True - print_error( - f"Cannot retrieve r2_key for already-uploaded {modality} model. " - "Re-run with the original r2_key, or pass --skip-chain if you only needed the upload." + print_success( + f"{modality.capitalize()} upload already accepted; exam will still run." ) - return False + return True print_error(f"{modality.capitalize()} model upload failed at step: {result.get('step', 'unknown')}") print_error(f"Error: {result.get('error', 'Unknown error')}") if result.get("response"): @@ -250,8 +244,12 @@ async def push_separate_models( for modality, result in results.items(): model_key = result.get("r2_key", "") if not model_key: - print_error(f"{modality.capitalize()} model key not provided in upload response") - return False + print_warning( + f"Skipping {modality} chain registration β€” no r2_key in the upload " + "response. The upload already succeeded and the model will still be examined." + ) + chain_ok = False + continue # Create hash for this specific model model_hash = hashlib.sha256( @@ -303,7 +301,8 @@ async def push_separate_models( continue print_warning( f"Giving up chain registration after {attempt} attempt(s). " - "The upload already succeeded and the model will still be examined." + "The upload already succeeded and the model will still be examined. " + f"r2_key={model_key}" ) chain_ok = False break @@ -315,7 +314,7 @@ async def push_separate_models( print_success("All models registered successfully!") else: print_warning("Upload succeeded; one or more chain registrations did not complete.") - return chain_ok + return True def main(): diff --git a/tests/test_push_model_retries.py b/tests/test_push_model_retries.py index 264df53b..a446475f 100644 --- a/tests/test_push_model_retries.py +++ b/tests/test_push_model_retries.py @@ -20,17 +20,13 @@ def test_zero_retries_is_unlimited(): assert should_retry_register(99, 0) is True -def test_already_uploaded_is_success_when_skipping_chain(): +def test_already_uploaded_is_success(): result = {"already_uploaded": True, "success": False} + assert _accept_already_uploaded("image", result, skip_chain=False) is True assert _accept_already_uploaded("image", result, skip_chain=True) is True -def test_already_uploaded_fails_when_chain_register_needs_r2_key(): - result = {"already_uploaded": True, "success": False} - assert _accept_already_uploaded("image", result, skip_chain=False) is False - - -def test_chain_registration_failure_is_reported(monkeypatch, tmp_path): +def test_chain_registration_failure_does_not_fail_push(monkeypatch, tmp_path): model_path = tmp_path / "model.zip" model_path.touch() wallet = SimpleNamespace(hotkey=SimpleNamespace(ss58_address="5Miner")) @@ -65,4 +61,37 @@ async def store_model_metadata(self, wallet, model_id): ) ) - assert success is False + assert success is True + + +def test_already_uploaded_without_r2_key_still_succeeds(monkeypatch, tmp_path): + model_path = tmp_path / "model.zip" + model_path.touch() + wallet = SimpleNamespace(hotkey=SimpleNamespace(ss58_address="5Miner")) + + monkeypatch.setattr( + push_model, + "upload_single_modality", + lambda *args, **kwargs: { + "success": False, + "already_uploaded": True, + "file_hash": "file-hash", + }, + ) + monkeypatch.setattr(push_model.bt, "Subtensor", lambda **kwargs: object()) + monkeypatch.setattr( + push_model, + "ChainModelMetadataStore", + lambda subtensor, netuid: object(), + ) + + success = asyncio.run( + push_separate_models( + image_model_path=str(model_path), + wallet=wallet, + retry_delay_secs=0, + max_retries=1, + ) + ) + + assert success is True