Shadow-Dance: Robot Learning to Communicate through Projected Visual Abstractions

Danyang Yan*, Boyuan Wang*, Jiaxun Liu*, Boyuan Chen

Duke University

* Equal contribution.

🌐 Project Website | 📄 Paper | 🎥 Video | 🤗 Hugging Face

---

Shadow-Dance is a research codebase for reconstructing dexterous hand poses from projected visual abstractions. Given a binary shadow image—or masks extracted from video—the pipeline estimates the 21-DoF Shadow Hand joint configuration whose rendered shadow best matches the target.

The repository includes tools for inference, test-time optimization, evaluation, synthetic data generation, video preprocessing, and hardware trajectory generation for the Shadow Hand.

Citation

If you find our paper or codebase helpful, please consider citing:

@misc{yan2026robotlearningcommunicateprojected,
      title={Robot Learning to Communicate through Projected Visual Abstractions}, 
      author={Danyang Yan and Boyuan Wang and Jiaxun Liu and Boyuan Chen},
      year={2026},
      eprint={2607.22434},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.22434}, 
}

Overview

Shadow-Dance follows a practical shadow-to-pose workflow:

1. Prepare target shadows from sample images or a video.
2. Predict candidate hand poses with the learned forward decoder.
3. Refine poses at test time with the hillclimber and optional expressive-region masks.
4. Evaluate reconstructed poses with simulator-rendered shadow loss.
5. Optionally export a hardware-ready trajectory for the Shadow hand.

The repository includes demo inference artifacts, test-time pose search, optional expressive-region refinement, simulator-based evaluation, paired synthetic-data generation, training entry points, video preprocessing utilities, and a dry-run trajectory bridge for the Shadow hand.

Installation

python -m venv .venv
source .venv/bin/activate
pip install -e ".[repro]"

The repro extra installs the default CPU PyTorch wheels from PyPI. CUDA users may install the matching PyTorch build first and then rerun the editable install, but the released numerical reference uses CPU. Installation needs network access and git because the CLIP Python source is pinned to a Git commit. Do not rely on CLIP's implicit weight download for reproduction; the pinned artifact download below installs the exact released ViT-B/32 weight file and verifies its SHA256.

Demo Artifacts

The release artifacts are hosted on the GeneralRoboticsLab Hugging Face dataset. Download the pinned release entry:

python scripts/download_artifacts.py --release reproduction-v1 --profile demo

The downloader reads manifest.json, verifies sha256 checksums and file sizes, skips already-valid files, and refuses manifest paths that would write outside the repository. Use --force only when you intentionally want to overwrite a corrupt local artifact.

The demo and reproduction profiles provide the two runtime artifacts:

checkpoints/shadow_dance_forward/shadow_dance_forward.pth
checkpoints/clip/ViT-B-32.pt

Source videos and historical preprocessing intermediates are a separate provenance profile:

python scripts/download_artifacts.py --release reproduction-v1 --profile full-data

These provenance files can be used to inspect or approximately rerun preprocessing. The canonical reproduction does not start from raw video because ROI selection, alignment, and bad-frame removal include manual decisions.

The canonical targets and reference poses are small and checked into Git under datasets/canonical/. The release does not include an inverse checkpoint or the paired datasets needed to retrain the models.

Quick Demo

Run inference on the alphabet and wolf samples:

python scripts/infer.py --config configs/infer.json
python scripts/infer.py --config configs/infer.json --data wolf --input-folder data/samples/wolf

Inference writes a timestamped run under outputs/runs/main_infer/ and atomically updates outputs/runs/main_infer/latest_run.json. The default hillclimber and evaluation configs read this latest pointer automatically.

If you run multiple inference jobs at once, latest points to whichever run finishes last. For controlled experiments, pass the explicit timestamped run directory instead of relying on latest.

Expected Outputs

A successful inference run creates:

outputs/runs/main_infer/<timestamp>_<name>/
  run_manifest.json
  summary_comparison_grid.png
  cases/
    <case_name>/
      final_result/
        final_joints.npy
        target.png
        result.png
        comparison.png
        winner_info.json
        loss_info.json

final_joints.npy stores the reconstructed 21-dimensional joint vector. comparison.png places the target and prediction side by side for quick visual inspection. loss_info.json records the optimization losses for the selected pose.

After inference, you can run simulator-based evaluation:

python scripts/evaluate.py --task sim-loss --config configs/evaluation.json

This adds sim_loss_info.json and sim_result.png to each evaluated final_result bundle.

Reproduce the Released Result from Processed Data

The supported reproduction begins with the checked-in, manually verified 256×256 targets. They have already undergone the historical TTA/crop operations, so canonical runs must not transform them again.

python scripts/download_artifacts.py --release reproduction-v1 --profile reproduction
python scripts/reproduce_subject.py --subject wolf --data-root datasets \
  --output outputs/reproduction/wolf --device cpu --mode solve
python scripts/compare_reproduction.py --subject wolf --run outputs/reproduction/wolf

The pipeline is processed target → gradient pose optimization → collision-aware hillclimber → trajectory. Read the reproduction guide for setup, validation, outputs, supported subjects, and the canonical/non-canonical boundary. Read the usage guide before changing configs or running new images/videos.

Training And Evaluation

The default artifact profile is an inference demo. The current configs/train_forward.json and configs/train_inverse.json files are templates: update the dataset paths before launching supervised training.

1. Optionally generate a small synthetic dataset:

   python scripts/generate_synthetic_data.py --samples 128 --angle 0 --output data/fix_hand_data_0degree

   The generator is headless by default. Pass --gui only when you want the PyBullet window.

2. Validate paired training data:

   python scripts/validate_paired_dataset.py path/to/paired_dataset
   python scripts/validate_paired_dataset.py path/to/fix_hand_data.tar

3. Train the forward decoder after updating configs/train_forward.json:

   python scripts/train_forward.py --config configs/train_forward.json

4. Train the inverse model after providing an inverse paired-data tar and updating configs/train_inverse.json:

   python scripts/train_inverse.py --config configs/train_inverse.json

5. Run main inference:

   python scripts/infer.py --config configs/infer.json

6. Refine with the hillclimber:

   python scripts/run_hillclimber.py --config configs/hillclimber.json

   By default this reads outputs/runs/main_infer/latest/cases.

7. Evaluate simulator loss:

   python scripts/evaluate.py --task sim-loss --config configs/evaluation.json

   By default this reads outputs/runs/main_infer/latest.

The demo configs are intentionally small. For larger experiments, pass larger datasets and longer schedules explicitly with CLI overrides.

Paired Training Data

Forward and inverse supervised training require paired synthetic data:

paired_dataset/
  joint_states.csv
  shadow_output_0_frame_0.png
  shadow_output_1_frame_0.png

joint_states.csv must use the first column as frame_id; the next 21 columns are joint values. For each frame id N, the matching grayscale shadow image must be named shadow_output_N_frame_0.png. The same files may also be packed into a .tar or .tar.gz with joint_states.csv anywhere inside the archive.

Use Your Own Video

Prepare target masks from a video:

python scripts/pipeline/extract_frames.py --video my_video.mp4 --output outputs/my_video/frames --fps 5
python scripts/pipeline/prepare_masks.py --input outputs/my_video/frames --output outputs/my_video/masks
python scripts/pipeline/prepare_expressive_region.py --config configs/expressive_region.json --input outputs/my_video/masks --output outputs/my_video/masks_expressive_region
python scripts/pipeline/cluster_keyframes.py --input outputs/my_video/masks --output outputs/my_video/clusters --clusters 8

The combined runner is:

python scripts/pipeline/run_pipeline.py --video my_video.mp4 --output outputs/my_video --fps 5 --clusters 8

For a released video target, download the pinned full-data profile and pass a sanitized clip under data/full_data/subjects/<subject>/ to the optional preprocessing pipeline. For example:

python scripts/pipeline/run_pipeline.py \
  --video data/full_data/subjects/<subject>/<clip>.mp4 \
  --output outputs/preprocessing/<subject> \
  --fps 5 \
  --clusters 8

Canonical method reproduction starts from the released pretransformed_target_256_v1 targets. They are already transformed and cropped to 256×256 and must not be sent through TTA/crop a second time. See docs/reproduction.md for the frozen state-machine contract and docs/preprocessing.md for provenance.

By default, the combined runner performs frame extraction, mask preparation, and keyframe clustering. Add --expressive-region to also write outputs/my_video/masks_expressive_region; use --expressive-stationary-mode neighbor_frames or --expressive-rescale for optional expressive-region variants.

This pipeline prepares target images for inference and optimization. It does not create 21-dimensional joint labels, so it cannot directly supervise-train or fine-tune the forward/inverse models unless you also provide matching joint angles or generate paired synthetic data.

Configuration

The configuration files in configs/ provide defaults and command-line flags override them. Keep configs/reproduction/default.json and the canonical manifests unchanged when reporting the released result; copy a config for exploratory work. The usage guide explains every user-facing command, its inputs/outputs, and the safe parameters to change.

Hardware Quick Replay

The fastest hardware demo path uses pre-generated, quick-replay-validated reference trajectories in assets/reference_trajectories/. These CSVs let you inspect the motor plan without rerunning the reconstruction or hardware preparation pipeline:

python scripts/hardware/run_motors.py \
  --csv assets/reference_trajectories/camel/interpolated_trajectory.csv

Dry-run is the default: it validates the CSV with quick-replay limits, converts relative joint angles to motor pulses, reports timing and clamp information, and does not import hid or open hardware. Passing --execute is the explicit opt-in path that opens the motor controller, homes the hand, sends trajectory frames, and returns to home on cleanup; execution uses a stricter default adjacent-frame jump limit unless you override it explicitly.

The official quick-replay subjects are listed in assets/reference_trajectories/manifest.json and currently include alphabet, bird, camel, deer, duck, peacock, and wolf. That manifest is the source of truth for which CSVs are release-checked for demo dry-run. The release checker also reports which subjects satisfy stricter hardware-safety limits with zero pulse clamps; that stricter report is guidance for physical execution, not a reason to hide display/demo trajectories.

If you want to regenerate a trajectory instead of using the shipped CSV, download the optional processed hardware package from Hugging Face and run stage 5:

python scripts/hardware/download_hardware_data.py --repo-id GeneralRoboticsLab/shadow-dance-artifacts --subject camel
python scripts/hardware/validate_hardware_data.py --data-root data/hardware_pipeline --subject camel
python scripts/hardware/generate_trajectory.py \
  --subject camel \
  --data-root data/hardware_pipeline \
  --output outputs/hardware/camel/interpolated_trajectory.csv
python scripts/hardware/run_motors.py --csv outputs/hardware/camel/interpolated_trajectory.csv

Stage 1-4 preparation scripts are advanced maintainer/reproduction tools for processed shadow images, binary masks, keyframe clustering, and ordered case metadata. They require raw or historical local inputs and include manual crop/alignment decisions, so cloning the repository is not intended to reproduce the full raw-video-to-trajectory pipeline bit-for-bit. The optional HF hardware package is for regenerating trajectories from cleaned pose outputs; it is not a training dataset.

The physical 21-DoF hand is fully printable. parts/ holds the CAD parts as STEP files plus parts/BOM.md, a bill of materials listing every printed part and quantity, the purchased servos, bearings, and fasteners, and the recommended print materials. STEP files are CAD geometry, so slice (export to mesh) before printing. vertical_fit.step bundles one knuckle set as a single body and must be separated into individual parts first; the other files print as-is.

See docs/hardware.md for the Hugging Face upload layout, privacy-cleaning rules, advanced stage 1-4 preparation commands, and release checks.

Code Organization

src/shadow_dance/
  core/              Shared models, solvers, data, paths, and I/O utilities
  training/          Forward and inverse training code
  inference/         Main reconstruction pipeline
  hillclimber/       Test-time pose refinement
  expressive_region/ Optional expressive-region mask extraction
  data_generation/   Synthetic paired-data generation
  pipeline/          Video-to-mask preprocessing utilities
  evaluation/        Simulator-loss and checkpoint evaluation
  hardware/          Trajectory export and motor bridge
scripts/             User-facing command-line entry points
configs/             Default experiment and utility configs
assets/              URDF, robot assets, and reference trajectories
parts/               Printable hand CAD (STEP) and bill of materials

The canonical URDF is:

assets/urdf/rounded_thin_palm/urdf/thin_palm_urdf_fixed.urdf

Extending The Code

The demo artifact bundle is intentionally small. It is suitable for checking installation, inspecting output bundles, and testing code changes around inference/evaluation. Larger training runs should use paired datasets and schedules appropriate for the target experiment.

For method development, the most relevant entry points are:

src/shadow_dance/training/forward.py      Forward decoder training
src/shadow_dance/inference/main.py        Main pose search pipeline
src/shadow_dance/core/solver.py           Losses, test-time optimization, result bundles
src/shadow_dance/hillclimber/main.py      Simulator-based refinement
src/shadow_dance/evaluation/              Evaluation scripts
src/shadow_dance/pipeline/                Video and mask preprocessing

Result Bundles

All new result bundles write final_joints.npy. shadow_dance.core.io.load_final_joints() can read legacy best_joints.npy during migration.

The public demo decoder uses model: shadow_dance_forward, decoder_arch: resnet8_urdf, and decoder_checkpoint: checkpoints/shadow_dance_forward/shadow_dance_forward.pth. The model label is only a run label; the checkpoint path is always explicit.

Training checkpoint metadata includes created_at, source config, dataset path, model name, and checkpoint name. Pass --save-timestamp-copy during training if you also want a best_model_<timestamp>.pth copy for experiment tracking.

License

This repository is released under the Apache License 2.0. See LICENSE for additional details.

Acknowledgement

This work is supported by DARPA TIAMAT program under award HR00112490419, ARO under award W911NF2410405, NSF ERC PreMiEr under award 2133504, and ARL STRONG program under awards W911NF2320182, W911NF2220113, and W911NF242021.