Fix full restore pipeline, add bind-mount backup, two-tier test suite

.github/workflows/main.yml

@@ -11,17 +11,31 @@ on:
   workflow_dispatch:
 
 jobs:
-  build:
-    runs-on: macos-13
+  # ── Fast: lint + stub-based unit tests (no daemon required) ──────────────
+  unit:
+    runs-on: ubuntu-latest
 
     steps:
       - uses: actions/checkout@v3
 
-      - name: Update brew
-        run: brew update
-
-      - name: Install shellcheck
-        run: brew install shellcheck
+      - name: Install dependencies
+        run: sudo apt-get install -y shellcheck bats jq
 
       - name: Run shellcheck
-        run: shellcheck *.sh backup/**.sh restore/*.sh -e SC2154 -e SC1091
+        run: >
+          shellcheck *.sh backup/**.sh restore/*.sh
+          test/integration/*.sh
+          -e SC2154 -e SC1091 -e SC2317
+
+      - name: Run unit tests (stub-based, no daemon)
+        run: bats test/*.bats
+
+  # ── Real: full backup → wipe → restore → verify inside dind ──────────────
+  integration:
+    runs-on: ubuntu-latest   # ubuntu runners support privileged containers
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Run integration tests (Docker-in-Docker)
+        run: bash test/integration/run.sh

Readme.md

@@ -2,56 +2,151 @@
 
 ## Description
 
-A bunch of Bash scripts to make a backup of all your running containers
-dynamically.
+A set of Bash scripts to back up and fully restore all your running Docker
+containers: images, named volumes, bind-mounted host paths, and container
+configuration.
 
-This will create a backup of docker images, volumes, upload to dropbox and
-remove the backup files after to save space
+## Requirements
 
-## Setup and Usage
+- Docker
+- `jq` — required for restore operations and bind-mount backup (`apt install jq` / `brew install jq`)
+- Sufficient permissions to read/write bind-mount host paths (may require `sudo`)
 
-This script use flags for configuration. The flags are:
+## Flags
 
-- **-m** to set the mode (backup or restore)
-- **-b** to set the backup path
-- **-t** to set the tar options
-- **-u** to upload to dropbox
-- **-f** to force
-- **-s** to run in non-interactive mode
+| Flag | Description |
+|------|-------------|
+| `-m` | **Required.** Mode: `backup` or `restore` |
+| `-p` | Backup directory path (default: `/home/core/backups`) |
+| `-t` | Extra options passed to `tar` when compressing the backup dir (e.g. `--exclude=/some/path`) |
+| `-B` | Enable bind-mount backup/restore (off by default — see note below) |
+| `-u` | Upload backup to Dropbox after completing |
+| `-f` | Force: overwrite non-empty backup dir on backup; replace existing volumes/containers/paths on restore |
+| `-s` | Non-interactive mode — skips all prompts, enables all steps (use in cron jobs) |
 
-Also, on dropbox you must create an App to store this backups, refer to
-https://www.dropbox.com/developers to get your **Generated access token**
-before running it and placed inside the **config/dropbox-uploader.conf** file
+## What gets backed up
 
-Give permissions to all sh files in the folder
+| Artifact | Location in backup |
+|----------|--------------------|
+| Container images | `<container>/<container>-image.tar` (via `docker save`) |
+| Named volumes | `volumes/<volume-name>.tar.gz` — one file per volume, shared volumes deduplicated |
+| Container inspect data | `<container>/<container>-data.txt` — JSON used to recreate containers on restore |
+| Bind-mount host paths *(opt-in with `-B`)* | `<container>/binds/bind-N.tar.gz` + `manifest.json` |
+
+## What gets restored (in order)
+
+1. **Images** — `docker load` from each image tar
+2. **Volumes** — extracted back into named Docker volumes
+3. **Bind mounts** *(only if `-B` was used during backup)* — host paths extracted to their original locations
+4. **Containers** — recreated from inspect data (best-effort: covers name, image, env, ports, volumes, restart policy, network mode)
+
+> **Container recreation is best-effort.** Common configuration is reconstructed
+> from the saved `docker inspect` output. Unusual settings (custom capabilities,
+> device mappings, secrets, etc.) may need to be applied manually.
+
+> **Bind-mount host paths** (`-v /host/path:/container/path`) require the `-B`
+> flag. Without it, only the container config (the `-v` flag reference) is
+> restored, but the actual host-side files are not touched. These paths often
+> require root-level read/write access.
+
+## Setup
 
 ```bash
 cd docker-backup-scripts
-chmod +x *.sh
+chmod +x *.sh backup/*.sh restore/*.sh
 ```
 
-Run the backup
+## Usage
+
+### Run a backup
 
 ```bash
 ./backup-manager.sh -p <output-path> -m backup
 ```
 
-Run the restoration
+Include bind-mounted host paths:
+
+```bash
+./backup-manager.sh -p <output-path> -m backup -B
+```
+
+### Run a restore
 
 ```bash
 ./backup-manager.sh -p <backup-path> -m restore
 ```
 
-## Extra
+Restore including bind-mounted host paths:
 
-Create a cron if you want to run it often (use the -s flag to run it in non-interactive mode)
+```bash
+./backup-manager.sh -p <backup-path> -m restore -B
+```
+
+Force-replace existing containers and volumes:
 
 ```bash
-crontab -e
+./backup-manager.sh -p <backup-path> -m restore -f
+```
+
+### Backup + upload to Dropbox
+
+```bash
+./backup-manager.sh -p <output-path> -m backup -u
+```
+
+You must create a Dropbox App and place your generated access token in
+`config/dropbox_uploader.conf` before using `-u`. See
+https://www.dropbox.com/developers for details.
+
+### Non-interactive / cron mode
+
+```bash
+./backup-manager.sh -p <output-path> -m backup -s
 ```
 
+All steps are enabled automatically in `-s` mode (bind mounts only if `-B` is
+also passed).
+
 ```bash
+crontab -e
+# Daily backup at midnight:
 0 0 * * * /path/to/backup-manager.sh -p <output-path> -m backup -s
 ```
 
-For CoreOS, I supply a timer to allow run it daily with an installation script
+## CoreOS timer
+
+A systemd timer for CoreOS is provided in `backup/coreos-timer/`. Run the
+installation script there to set up a daily backup service.
+
+## Testing
+
+Two test tiers are provided.
+
+### Unit tests — fast, no Docker daemon needed
+
+Uses a `docker` stub to verify script logic in isolation. Requires
+[bats-core](https://github.com/bats-core/bats-core) and `jq`.
+
+```bash
+bats test/*.bats
+```
+
+Covers: volume backup layout and deduplication (issue #14), dotted volume-name
+parsing regression, full restore pipeline ordering (issue #18), container
+reconstruction from inspect JSON, bind-mount backup/restore skip/force logic.
+
+### Integration tests — real backup → wipe → restore → verify
+
+Runs the full cycle inside an isolated **Docker-in-Docker** daemon so your host
+containers and volumes are never touched. Requires Docker with privileged
+container support.
+
+```bash
+bash test/integration/run.sh
+```
+
+Verifies end-to-end:
+- Named volume data survives backup and restore
+- Bind-mount host paths are backed up with `-B` and restored in place
+- Container images are re-loaded after being deleted
+- Restored container is running with correct env, restart policy, and port mapping

backup-manager.sh

@@ -21,12 +21,13 @@ cd "${BASH_SOURCE%/*}" || exit
 
 non_interactive=false
 backup_path="/home/core/backups"
-tar_opts="--exclude='/var/run/*'"
+tar_opts="--exclude=/var/run/*"
 docker_upload_enable=false
 force=false
 mode=""
+bind_mounts_enable=false
 
-while getopts "sp:t:ufm:" opt; do
+while getopts "sp:t:ufm:B" opt; do
   case $opt in
     s)
       non_interactive=true
@@ -46,6 +47,9 @@ while getopts "sp:t:ufm:" opt; do
     m)
         mode=$OPTARG
         ;;
+    B)
+      bind_mounts_enable=true
+      ;;
     \?)
       echo "Invalid option: -$OPTARG" >&2
       ;;

backup/backup-all.sh

@@ -50,12 +50,16 @@ then
   echo "Backup volumes ? (y/n)"
   read -r backup_volumes
 
+  echo "Backup bind-mounted host paths ? (y/n) [requires read access to those paths]"
+  read -r backup_bind_mounts
+
   echo "Should I compress the backup directory ? (y/n)"
   read -r compress_backup
 else
   backup_container_data="y"
   backup_container_images="y"
   backup_volumes="y"
+  backup_bind_mounts=$([ "$bind_mounts_enable" = true ] && echo "y" || echo "n")
   compress_backup="n"
 fi
 
@@ -74,10 +78,16 @@ then
   source backup/backup-volumes.sh
 fi
 
+if [ "$backup_bind_mounts" = "y" ]
+then
+  source backup/backup-bind-mounts.sh
+fi
+
 if [ "$compress_backup" = "y" ]
 then
   echo -n "Compressing backup directory - "
-  tar -czf "$backup_path.tar.gz" "$backup_path" >/dev/null 2>&1
+  # shellcheck disable=SC2086
+  tar $tar_opts -czf "$backup_path.tar.gz" "$backup_path" >/dev/null 2>&1
   echo "OK"
 
   echo -n "Removing backup directory - "

backup/backup-bind-mounts.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+# Path: backup-bind-mounts.sh
+# Backup host filesystem paths that are bind-mounted into containers.
+#
+# Each container's bind mounts are archived into:
+#   <backup_path>/<container>/binds/bind-N.tar.gz
+# with a manifest.json that maps each archive back to its original host path.
+#
+# Note: this script must run with sufficient permissions to read the bind-mount
+# source paths (often requires root).
+
+echo "Backing up bind mounts"
+echo "----------------------"
+
+found_any=false
+
+for container_name in $(docker ps -q | xargs docker inspect --format='{{.Name}}' | cut -f2 -d/)
+do
+  # Extract bind-mount source paths using a Go template (no jq needed here)
+  bind_sources=$(docker inspect \
+    --format='{{range .Mounts}}{{if eq .Type "bind"}}{{println .Source}}{{end}}{{end}}' \
+    "$container_name")
+
+  [ -z "$bind_sources" ] && continue
+
+  found_any=true
+  echo "$container_name:"
+
+  binds_dir="$backup_path/$container_name/binds"
+  mkdir -p "$binds_dir"
+
+  manifest="$binds_dir/manifest.json"
+  printf '[\n' > "$manifest"
+  first=true
+  i=0
+
+  while IFS= read -r host_path
+  do
+    [ -z "$host_path" ] && continue
+
+    archive_name="bind-$i.tar.gz"
+    archive_path="$binds_dir/$archive_name"
+
+    echo -n "  $host_path - "
+
+    if [ ! -e "$host_path" ]; then
+      echo "SKIPPED (path does not exist)"
+    else
+      # Archive with absolute paths (-P) so restore can put files back exactly
+      tar -P -czf "$archive_path" "$host_path" 2>/dev/null
+      echo "OK"
+    fi
+
+    # Append manifest entry (comma-separate after the first)
+    if [ "$first" = true ]; then
+      first=false
+    else
+      printf ',\n' >> "$manifest"
+    fi
+    printf '  {"archive":"%s","path":"%s"}' "$archive_name" "$host_path" >> "$manifest"
+
+    i=$((i + 1))
+  done <<< "$bind_sources"
+
+  printf '\n]\n' >> "$manifest"
+done
+
+if [ "$found_any" = false ]; then
+  echo "No bind mounts found on running containers"
+fi
+
+echo ""