Learning in Robotics

.gifs and visualizations take a few seconds to load...

About

This repository contains projects completed for ESE 6500: Learning in Robotics (Spring 2026) at the University of Pennsylvania, taught by Prof. Pratik Chaudhari. Course page: pratikac.github.io/pub/25_ese650.pdf

The course covers state estimation, optimal control, and reinforcement learning for robotic systems — from Kalman filtering and particle filters to policy gradients and Q-learning.

---

HW 4 — MuJoCo MPC and PPO for the dm_control Walker

MuJoCo MPC — Real-time Sampling Predictive Control

Goal: Run Google DeepMind's MJPC on classic control tasks. The planner samples thousands of action trajectories per second and executes the best one in a receding horizon — no learning, just fast simulation in the loop.

Cartpole, Acrobot, and Humanoid Stand stabilized in real time by MJPC.

---

PPO for the dm_control Walker

Goal: Train a 2D bipedal walker to walk forward using Proximal Policy Optimization, end-to-end in PyTorch (no SB3/RLlib). Observation: 24-dim (orientations, height, joint velocities). Action: 6-dim continuous joint torques in $[-1, 1]$.

Trained policy walking — deterministic mean action, no Gaussian sampling.

The actor is a Gaussian policy $\pi_\theta(u\mid x) = \mathcal{N}(\mu_\theta(x), \sigma)$ with $\mu$ from a 64-64 MLP and a learnable state-independent log-std. A separate 64-64 critic $V_\phi(x)$ is trained by MSE against TD-$\lambda$ returns. Each PPO update minimizes the clipped surrogate

$$L^{\text{CLIP}}(\theta) = -\mathbb{E}_t \left[ \min \big( r_t(\theta) A_t,\ \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon), A_t \big) \right], \quad r_t(\theta) = \frac{\pi_\theta(u_t\mid x_t)}{\pi_{\theta_{\text{old}}}(u_t\mid x_t)}$$

with advantages from Generalized Advantage Estimation ($\gamma=0.99$, $\lambda=0.95$). KL early-stopping at $1.5 \cdot \kappa_{\text{target}}$ guards against destructive updates. Trained with 8 parallel envs (multiprocessing) on an RTX 3060.

Training over 5M env-steps. Mean batch return rises from ~30 (random init) to a peak of 789; further fine-tuning (halved LR) pushed the deterministic-eval best to 888 on the dm_control walker.walk reward (out of ~1000 max).

Concepts learned: Policy gradients on continuous action spaces, Gaussian policies, the PPO clipped surrogate, GAE advantage estimation, vectorized rollouts via multiprocessing, KL early-stopping, log-std clamping vs. entropy collapse - and the empirical lesson that policy regularization (entropy bonus, action L2) outperforms reward shaping for fixing pathological gaits.

---

HW 3 — NeRF, Particle Filter SLAM, and Policy Iteration

Neural Radiance Field (NeRF) for 3D Scene Reconstruction

Goal: Implement a NeRF from scratch to reconstruct a 3D scene (LEGO bulldozer) from 100 posed 2D images, using COLMAP for camera pose estimation.

Left: Turntable render orbiting the reconstructed scene. Right: Ground truth (left column) vs. NeRF renders (right column) from training viewpoints.

Pipeline: COLMAP sparse reconstruction → camera intrinsics/extrinsics → ray casting through each pixel → stratified depth sampling → positional encoding (10 frequency bands, $\mathbb{R}^3 \to \mathbb{R}^{63}$) → TinyNeRF MLP → volume rendering (alpha compositing with transmittance).

Concepts learned: Pinhole camera model, ray-based rendering, positional encoding for high-frequency detail, volume rendering with alpha compositing, differentiable rendering, COLMAP structure-from-motion.

---

Particle Filter SLAM on KITTI

Goal: Implement SLAM using a particle filter with Velodyne LiDAR and GPS/IMU odometry from the KITTI dataset. Estimate the car's trajectory while simultaneously building an occupancy grid map across 4 driving sequences.

Occupancy maps for KITTI datasets 00 (left) and 02 (right) with SLAM trajectory (red) and odometry (blue). Road boundaries and buildings are clearly visible.

Dataset 00 trajectory comparison — SLAM (red) closely tracks odometry (blue) over a multi-loop urban drive through Karlsruhe.

Pipeline: LiDAR points → Velodyne-to-camera transform ($T_r$) → camera-to-world via particle pose → occupancy grid update (log-odds) → particle reweighting → stratified resampling.

Concepts learned: Particle filters, SE(2) pose composition, LiDAR-to-world coordinate transforms, log-odds occupancy mapping, stratified resampling, KITTI dataset conventions.

---

Policy Iteration for Stochastic Grid Navigation

Goal: Find the optimal policy for a robot navigating a 10×10 grid with stochastic dynamics ($P(\text{intended}) = 0.7$, $P(\text{drift}) = 0.3$), sticky obstacles, and a discounted reward structure ($\gamma = 0.9$).

Left: Value function under the naive "always East" policy — cells west of obstacles are heavily penalized (red). Right: Converged optimal policy (iteration 6) with arrows showing the best action at each cell.

Policy iteration convergence (iterations 0–3). The policy evolves from all-East to routing around obstacles toward the goal. Converged in 6 iterations.

Concepts learned: Markov Decision Processes, Bellman equation, policy evaluation via linear solve, policy improvement, cost-to-go formulation, stochastic transition matrices.

---

HW 2 — Unscented Kalman Filter (UKF) for 3D Orientation Estimation

Goal: Implement an Unscented Kalman Filter (UKF) to track the orientation of an IMU in three dimensions, fusing accelerometer and gyroscope measurements against Vicon motion-capture ground truth. Score: 56/56 on the Gradescope autograder.

3D orientation tracking (Datasets 1–3) — Vicon ground truth (gray) vs. UKF estimate (RGB axes) with 3σ covariance ellipsoid.

The UKF on SO(3)

This is not a standard UKF — the state lives partly on the rotation manifold:

$$x = \begin{bmatrix} q \ \omega \end{bmatrix} \in \mathbb{R}^7$$

where $q$ is a unit quaternion (orientation) and $\omega$ is angular velocity. Because quaternions are constrained to the unit sphere, the covariance is $\Sigma \in \mathbb{R}^{6 \times 6}$ (not $7 \times 7$), using axis-angle error parameterization following the Kraft (2003) formulation. Sigma points are generated in the 6D tangent space, mapped to quaternions via the exponential map, and the quaternion mean is computed via iterative gradient descent (Kraft Sec. 3.4).

Process model: $q_{k+1} = q_k \otimes$ from_axis_angle $(\omega \cdot \Delta t)$, with $\omega$ assumed constant. Measurement model: Accelerometer predicts the gravity vector rotated into the body frame; gyroscope directly measures $\omega$.

Step 1 — IMU Calibration

The raw IMU readings are biased: $\text{value} = \text{raw} + \beta$. I built interactive slider tools using Claude Code to calibrate the accelerometer and gyroscope biases against Vicon ground truth. Roll and pitch are recovered from the accelerometer via gravity direction; gyroscope biases are found by matching integrated angular velocity to Vicon angular rates.

Interactive accelerometer (left) and gyroscope (right) bias calibration against Vicon ground truth.

After tuning, we verify that accelerometer-derived roll/pitch and gyro-integrated orientation both align with Vicon:

Calibrated accelerometer (left) and gyroscope (right) — roll/pitch and angular rates match Vicon.

The combined integration check confirms all 6 biases are consistent:

Joint calibration view: Vicon (blue), gyro-integrated (dark red), and accelerometer (pink/green) orientation.

Step 2 — UKF Noise Parameter Tuning

With calibrated sensors, the UKF has four noise parameters to tune: process noise ($\sigma_q$, $\sigma_\omega$) and measurement noise ($\sigma_{\text{acc}}$, $\sigma_{\text{gyro}}$). I built a real-time slider interface to visualize the effect of each parameter on filter output vs. Vicon.

UKF noise tuner with real-time Euler angle comparison (roll, pitch, yaw) and RMSE readout.

Step 3 — Analysis and Debugging

The UKF outputs are analyzed across four diagnostic views: quaternion components, angular velocity with uncertainty bands, covariance evolution, and Euler angles vs. Vicon.

Euler angles (roll, pitch, yaw) — UKF estimate vs. Vicon ground truth with per-axis RMSE.

Covariance diagonal over time. Notice P[2,2] (yaw orientation) grows while P[0,0] and P[1,1] (roll/pitch) stay bounded — yaw is unobservable from the accelerometer alone.

Key Insights: The Road to 56/56

The journey from 55.25/56 to full marks required three breakthroughs:

1. Numerical hygiene: The provided quaternion.py had subtle issues (unclamped acos, commented-out normalization). I added 5 defensive normalizations and a covariance symmetrization step in the filter. This didn't change the score directly but made the filter respond predictably to parameter changes.

2. Understanding accelerometer-yaw cross-coupling: Accelerometers can only observe roll/pitch (via gravity), not yaw. But in quaternion representation, accelerometer corrections "leak" into yaw through cross-terms. The parameters $\sigma_q$ and $\sigma_{\text{acc}}$ control this leakage.

3. Anisotropic process noise: Different datasets needed conflicting $\sigma_q$ values. The solution: separate $\sigma_q$ for roll/pitch (tight, to minimize yaw contamination) and yaw (loose, to allow gyro-based yaw corrections). This decoupled the tradeoff and achieved full marks.

# Isotropic (couldn't satisfy all datasets):
R = np.diag([sigma_q**2]*3 + [sigma_w**2]*3)

# Anisotropic (56/56):
R = np.diag([sigma_q_rp**2, sigma_q_rp**2, sigma_q_yaw**2] + [sigma_w**2]*3)

Concepts learned: Quaternion-based UKF on SO(3), sigma point generation in tangent space, quaternion averaging via gradient descent, IMU calibration, sensor noise covariance tuning, yaw unobservability from accelerometers, anisotropic process noise design.

---

HW 2 — Extended Kalman Filter for Parameter Estimation

Goal: Use an EKF to estimate an unknown system parameter $a$ from noisy observations of a nonlinear dynamical system.

The system is defined as:

$$x_{k+1} = a \cdot x_k + \epsilon_k, \quad y_k = \sqrt{x_k^2 + 1} + \nu_k$$

where $a = -1$ is the unknown parameter to be estimated, $\epsilon_k \sim \mathcal{N}(0, 1)$, and $\nu_k \sim \mathcal{N}(0, 0.5)$.

The state is augmented to $[x_k, a]^T$ and the EKF linearizes the nonlinear observation model at each step, updating both the hidden state and parameter estimate simultaneously.

Simulated state trajectory x_k and nonlinear observations y_k over 100 time steps.

EKF estimate of the unknown parameter a converges to the true value a = -1, with uncertainty (shaded) shrinking over time.

The EKF successfully recovers $a \approx -1$ with decreasing uncertainty, demonstrating that filtering can jointly estimate latent states and system parameters from indirect, nonlinear measurements.

Concepts learned: EKF derivation for nonlinear systems, joint state-parameter estimation, Jacobian computation for measurement updates, convergence and uncertainty analysis.

---

Topics Covered

The course develops foundations in state estimation, control, and reinforcement learning for robotics:

Module 1 — State Estimation

• Probability background and Bayesian inference
• Markov chains and Hidden Markov Models
• Kalman Filter, Extended Kalman Filter (EKF), and Unscented Kalman Filter (UKF)
• Particle filters and sequential Monte Carlo methods
• Mapping, localization, and SLAM
• Neural Radiance Fields (NeRF) and Gaussian Splatting for SLAM
• Foundation models for robotics

Module 2 — Control

• Linear control and dynamic programming
• Markov Decision Processes (MDPs)
• Value Iteration and Policy Iteration
• Bellman equation and optimality
• Linear Quadratic Regulator (LQR)
• Linear Quadratic Gaussian (LQG)

Module 3 — Reinforcement Learning

• Imitation learning and behavior cloning
• Policy gradient methods (REINFORCE, PPO)
• Q-Learning and Deep Q-Networks (DQN)
• Offline reinforcement learning