Implementation of l2 attack with different optimizers: sgd | sgdsign …

.gitignore

@@ -0,0 +1,19 @@
+data/
+graphs/
+cifar10/
+mnist/
+
+
+__pycache__/*
+cifar10-early/
+imagenet/
+mnist-early/
+nes_attack.py
+nes_results_runs/
+plots/
+run_logs/
+scripts/
+
+*.csv
+*.sh
+*.zip

README.md

@@ -1,35 +1,182 @@
-# ZOO: Zeroth Order Optimization Based Adversarial Black Box Attack (PyTorch)
-This repository contains the PyTorch implementation of Zeroth Order Optimization Based Adversarial Black Box Attack(https://arxiv.org/abs/1708.03999) using MNIST and CIFAR10 dataset. This is the exact replica as far possible of the ZOO Attack (https://github.com/IBM/ZOO-Attack) which was originally implemented in Tensorflow. The results match almost as same as the paper evaluation results for MNIST and CIFAR10 for both targeted and untargeted attack all with 100% success rate on the 7 layer CNNs model trained on MNIST with 99.5% val accuracy and on CIFAR10 with 80% val accuracy as done in the original paper work. Both ZOO_Adam and ZOO_Newton methods of Coordinate Descent Solvers are implemented.
+# ZOO Black-Box Adversarial Attack
 
-**Note: This doesn't contain implementation of importance sampling, hierarchical attack, and dimentional reduction right now (as its mainly needed for large image sized dataset like ImageNet). For larger dataset google colab viewer [link](https://colab.research.google.com/drive/1_rDu0LNAI9kg490rV9PMxb55Wwbj13aD?usp=sharing) [NOT TESTED]**
+Implementation of the ZOO (Zeroth Order Optimization) black-box adversarial attack from [Chen et al., 2017](https://arxiv.org/abs/1708.03999), supporting MNIST, CIFAR-10, and ImageNet with multiple coordinate-descent solvers. Forked from the pyTorch implementation following paper's repository.
 
-## Setup and train models
-The code is tested with Python 3.7.6 and PyTorch 1.6.0. The following packages are required:
-```
-python pip install --upgrade pip
-pip install torch==1.6.0 torchsummary==1.5.1 torchvision==0.7.0
-pip install numpy matplotlib 
-```
-To prepare model and datasets of MNIST and CIFAR10
+---
+
+## Installation
+
+```bash
+pip install torch torchvision torchsummary numpy matplotlib scikit-image pillow
 ```
+
+The pre-trained MNIST and CIFAR-10 models are already included in `models/`. The MNIST and CIFAR-10 datasets are downloaded automatically by torchvision on first run, and the ImageNette subset used for ImageNet is fetched automatically when `--dataset imagenet` is selected.
+
+To retrain the classifiers from scratch instead of using the bundled checkpoints:
+
+```bash
 python setup_mnist_model.py
 python setup_cifar10_model.py
 ```
-## Run attacks
-To run the attacks run the 
+
+---
+
+## Running attacks
+
+### Reproducing the paper results
+
+The results reported in the paper come from sweeping every solver across every model, both untargeted and targeted. A single call runs the whole sweep:
+
+```bash
+python run.py # all solvers, all models, both untargeted + targeted, plot generation
+```
+
+`run.py` calls the attack below once per (dataset, solver, mode) combination and writes each run's output to its own folder (see [Results](#results)). Restrict the sweep with `--modes untargeted` or `--modes targeted`, narrow it with `--datasets` / `--solvers` / `--samples`, or pass `--dry-run` to preview the commands first.
+
+### Basic usage
+
+```bash
+python zoo_l2_attack_black.py --dataset <mnist|cifar10|imagenet> --solver <solver> --samples <n>
+```
+
+### All solvers — MNIST
+
+```bash
+python zoo_l2_attack_black.py --dataset mnist --solver adam       --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver newton     --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver sgd        --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver sgdsign    --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver signum     --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver lion       --samples 10
+python zoo_l2_attack_black.py --dataset mnist --solver adahessian --samples 10
+```
+
+### All solvers — CIFAR-10
+
+```bash
+python zoo_l2_attack_black.py --dataset cifar10 --solver adam       --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver newton     --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver sgd        --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver sgdsign    --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver signum     --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver lion       --samples 10
+python zoo_l2_attack_black.py --dataset cifar10 --solver adahessian --samples 10
+```
+
+### Targeted attack
+
+```bash
+python zoo_l2_attack_black.py --dataset mnist   --solver adam --samples 10 --targeted
+python zoo_l2_attack_black.py --dataset cifar10 --solver adam --samples 10 --targeted
+```
+
+---
+
+## Early stopping
+
+ZOO uses an L2 objective with binary search over the trade-off constant, so by default the attack runs the full iteration budget to minimise distortion. Passing `--early-stop` stops a sample as soon as the adversarial example successfully fools the model, which measures how many queries each solver needs to break the model, a strong indicator of solver efficiency. The number of queries used is saved in `results.json`.
+
+---
+
+## All arguments
+
+| Argument | Default | Description |
+|---|---|---|
+| `--dataset` | `cifar10` | `mnist`, `cifar10`, or `imagenet` |
+| `--solver` | `adam` | `adam`, `newton`, `sgd`, `sgdsign`, `signum`, `lion`, `adahessian` |
+| `--samples` | `10` | Number of **source** images. Untargeted: N attacks. Targeted: each source is attacked toward every other class, so for 10-class MNIST/CIFAR-10 that's N × 9 (e.g. 10 → 90 attacks) |
+| `--start` | `6` | Offset into the test set |
+| `--targeted` | `False` | Targeted attack (default: untargeted) |
+| `--targeted-k` | `None` | In targeted mode, use at most K random target classes per source image |
+| `--imagenet_dir` | `None` | Path to ImageNet val directory (used when `--dataset imagenet`) |
+| `--batch-size` | `128` | Coordinates updated per optimization step |
+| `--max-iter` | `1000` | Max iterations per binary-search stage |
+| `--const` | `0.01` | Initial attack trade-off constant |
+| `--confidence` | `0.0` | CW confidence margin |
+| `--early-stop-iters` | `100` | Abort-check cadence |
+| `--binary-search-steps` | `9` | Binary search steps over `const` |
+| `--adam-beta1` | `0.9` | Optimizer beta1 |
+| `--adam-beta2` | `0.999` | Optimizer beta2 |
+| `--step-size` | auto | Override the solver's default coordinate step size |
+| `--early-stop` | `False` | Stop each sample once the attack succeeds |
+
+Default coordinate step size per solver (override with `--step-size`):
+
+| Solver | step size |
+|---|---|
+| adam | 0.01 |
+| newton | 0.01 |
+| sgd | 0.01 |
+| sgdsign | 0.015 |
+| signum | 0.015 |
+| lion | 0.015 |
+| adahessian | 0.01 |
+
+---
+
+## Results
+
+Results are saved in `<dataset>/<targeted|untargeted>/<solver>/`:
+
+```
+cifar10/
+  untargeted/adam/
+    results.json               : success rate, queries, distortion, PSNR, SSIM, time
+    original_0.png             : original image
+    adversarial_0.png          : adversarial image
+    grid_cifar10_untargeted_adam.png  : side-by-side grid with class labels
 ```
-python zoo_l2_attack_black.py
+
+`results.json` includes a `queries` block:
+
+```json
+"queries": {
+  "per_sample": [1200, 800, ...],
+  "mean_on_success": 1050.0,
+  "budget": 50000
+}
 ```
-Both untargeted and targeted attack are accessible via above code all the changes (comment/uncomment) for transition from ZOO_Adam/ZOO_Newton or CIFAR10/MNIST are from line 259-262, 270/271, 274-277 and for visualization of example generated, line 307/329. For more details go through the code zoo_l2_attack_black.py and the paper https://arxiv.org/abs/1708.03999
 
-## Sample Results
-#### ZOO_Adam 
-##### Untargeted on CIFAR10
-![](/sample_results/adam_untargeted_cifar10.png)
-##### Untargeted on CIFAR10
-![](/sample_results/adam_untargeted_mnist.png)
-#### ZOO_Newton
-##### Targeted on MNIST
-![](/sample_results/newton_targeted_mnist.png)
+along with `success_rate_pct`, `total_distortion`, `time_mins`, the per-sample `mse`, `mae`, `psnr`, and `ssim` blocks, and an `attack_params` block recording the hyperparameters used.
+
+---
+
+## Plots
+
+`plot_zoo_metrics_summary.py` reproduces the per-optimizer metric bar chart (MAE, MSE, PSNR, SSIM, L-inf and L2 distortion) for one dataset and threat model. MAE/MSE/PSNR/SSIM are read from each solver's `results.json`; L2 and L-inf are computed from the saved `original_*.png` / `adversarial_*.png` pairs. Folders whose name contains `pgd` are skipped.
+
+```bash
+python plot_zoo_metrics_summary.py --dataset cifar10 --attack-type targeted
+```
+
+The figure is written to `plots/`. `run.py` renders these charts automatically after the sweep finishes, one per dataset and threat model.
+
+
+---
+
+## Optimizer intuition
+
+We deliver `opt_visualization.py` for gaining intuition behind the optimizers' behaviour, independent of the attack setting. It minimises classic 2-D test functions — Rosenbrock, Himmelblau, Beale, and Sphere, with the same optimizer family used in the attacks (Adam, Newton, SGD, SignSGD, Signum, Lion), recording each optimizer's trajectory and plotting it over a filled 2-D contour and a 3-D surface, plus a convergence curve.
+
+```bash
+python opt_visualization.py
+```
+
+Switch the target function, starting point, and per-solver learning rates via the settings at the bottom of the file. A companion script, `grad_est_optimization.py`, runs the same optimizers on the same functions but contrasts exact analytical gradients against finite-difference estimates, so how gradient noise reshapes each optimizer's path can be observed.
+
+Example visualizations:
+<p align="center">
+  <img src="vis_assets/image1.jpeg" width="30%" />
+  <img src="vis_assets/image2.jpeg" width="30%" />
+  <img src="vis_assets/image3.jpeg" width="30%" />
+</p>
+
+---
+
+## Sample results
+
+Every run writes a labelled grid image (`grid_<dataset>_<mode>_<solver>.png`) into its output folder, showing each source image with its `original → adversarial` prediction. A few precomputed examples are committed under [`sample_results/`](sample_results), so you can see what the attack produces without running anything.
 
+CIFAR-10, Adam (untargeted):
 
+![CIFAR-10 adversarial examples — Adam, untargeted](sample_results/grid_cifar10_targeted_adam.png)