Distributed SGD Project

This project implements three distributed training algorithms using PyTorch C++ (LibTorch) and MPI:
1. Synchronous SGD (SSGD): Gradients are averaged across all workers after every batch.
  - Refer `Src_Prjt-CS22BTECH11055-sync_sgd.cpp`
2. Local SGD (FedAvg): Model parameters are averaged across all workers after a fixed number of local steps.
  - Refer `Src_Prjt-CS22BTECH11061-local_sgd.cpp`
3. Decentralized SGD (D-SGD): Model parameters are exchanged and averaged with neighbors in a ring topology.
  - Refer `Src_Prjt-CS22BTECH11061-decentralized_sgd.cpp`

Prerequisites
-------------
Ensure the following dependencies are installed:
- C++ Compiler: GCC or Clang with C++17 support.
- CMake: Version 3.10 or higher.
- MPI Library: OpenMPI or MPICH.
- Python 3: For dataset download and testing scripts.
- LibTorch: PyTorch C++ library.

Setup Instructions
------------------
1. Install Dependencies:
   Run the `setup.sh` script to install the required tools and libraries:
   $ bash setup.sh

2. Build the Project:
   Use the `build.sh` script to compile the project:
   $ bash build.sh

3. Download the MNIST Dataset:
   The `setup.sh` script also downloads and extracts the MNIST dataset into the `data/` directory.

Execution Instructions
----------------------
Running Individual Algorithms
-----------------------------
Each algorithm is compiled into a separate binary:
- SSGD: build/sync_sgd
- Local SGD: build/local_sgd
- Decentralized SGD: build/decentralized_sgd

To execute any binary, use `mpirun` with the desired number of workers:
$ mpirun -np <num_workers> ./build/<binary_name>

For example:
$ mpirun -np 4 ./build/local_sgd

Running Tests and Generating Results
------------------------------------
The `run_tests.py` script automates the execution of all binaries with multiple worker configurations and generates logs, CSV summaries, and plots.

Example:
$ python3 run_tests.py \
  --binaries build/sync_sgd build/local_sgd build/decentralized_sgd \
  --workers 1 2 4 8 \
  --repeats 1 \
  --out results

Outputs:
- Logs: Stored in `results/<binary_name>/workers_<num_workers>/stdout_<repeat>.txt`.
- CSV Summary: `results/summary.csv` contains runtime, loss, and accuracy metrics.
- Plots:
  - `results/<binary_name>/losses.png`: Loss vs. Epoch for different worker counts.
  - `results/runtime_comparison.png`: Runtime vs. Worker count.
  - `results/test_accuracy_comparison.png`: Test accuracy vs. Worker count.

Additional Plotting
-------------------
The `plot.py` script can be used to generate additional plots from the `results/summary.csv` file. It includes:
- Test accuracy comparison across worker counts with zoomed y-axis.
- Loss vs. Epoch for each worker count.

Example:
$ python3 plot.py --results results/summary.csv --out results

Sample Input and Output (Change in ./src/<file>.cpp)
-----------------------
Input:
- Dataset: MNIST (downloaded to `data/`).
- Hyperparameters:
  - Batch size: 4
  - Learning rate: 0.01
  - Momentum: 0.5
  - Epochs: 10
  - Synchronization intervals:
    - SSGD: Every batch.
    - Local SGD: Every 5 local steps.
    - D-SGD: Every batch with neighbors.

Output:
- Training Logs:
  Example log for `sync_sgd` with 4 workers:
  Starting SSGD with 4 ranks
  Rank 0 using seed 1234
  [SSGD] Epoch 0 | Loss: 0.0907372
  [SSGD] Epoch 9 | Loss: 8.06682e-05
  Training Complete
  Test Accuracy: 99.13% (9913/10000)

- Plots:
  - Loss vs. Epoch for each algorithm.
  - Runtime comparison across worker counts.
  - Test accuracy comparison across worker counts.

Code Overview
-------------
Algorithms:
- SSGD: src/sync_sgd.cpp
- Local SGD: src/local_sgd.cpp
- D-SGD: src/decentralized_sgd.cpp

Build System:
- CMake Configuration: CMakeLists.txt

Dataset Download:
- Script: download_mnist.py

Testing and Plotting:
- Script: run_tests.py
- Additional Plotting: plot.py

Notes
-----
- Ensure `LD_LIBRARY_PATH` is set to include the `libtorch/lib` directory before running binaries.
- Use `mpirun` or `mpiexec` depending on your MPI installation.