Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
Consider whether the change should land upstream in Overlapping files
|
|
|
|
|
…nfig MuJoCo simulation of a Kinova Gen3 (7-DoF) + Robotiq 2F-85 stacking colored cubes, for testing VLA policy execution via ExecutePolicy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xkm9aPVJ33L5QhRYs8M3u6
…aunched bridge - docker/: Dockerfile.serve_policy + docker-compose.yaml + serve_policy.py, an HTTP inference server for lerobot pi0/pi0.5 checkpoints with real-time chunking support, plus unit tests (test_serve_policy.py). External port is driven by SERVE_POLICY_PORT; internal container port stays fixed at 8973. - script/get_action_chunk_adapter.py: ROS bridge between ExecutePolicy's /get_action_chunk service and the docker server, with unit tests under test/ wired into colcon test via CMakeLists.txt/package.xml. - launch/simulated_extras.launch.py: auto-launches the adapter alongside the sim Agent via config.yaml's simulated_hardware_launch_file, so it no longer needs to be run manually. - config/moveit/joint_limits.yaml: correct max_acceleration to match the vendor MoveIt config for this arm+gripper combo (kortex_moveit_config), rather than an uncited value. - objectives/execute_color_stack_policy.xml: lower policy_tracking_weight 300->30. Verified via e2e run with a pi0.5 checkpoint that 300 made the chunk executor's seam-anchor spline overshoot the acceleration limit right after each chunk boundary; 30 runs the full objective without aborting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xkm9aPVJ33L5QhRYs8M3u6
…RTC defaults, drop dead RealSense mesh
- objectives/execute_color_stack_policy.xml: move to the Home waypoint and
open the gripper before switching controllers and running ExecutePolicy,
so a rollout always starts from a known state.
- Default committed_action_steps/guidance_horizon to 20 (matching the
standard RTC execution horizon) and serve_policy.py's --fps/--execution-horizon
defaults to 10/20 to match; verified end to end that the resulting
total_action_steps=250 (25s) budget is sufficient for a full stack.
- description/picknik_kinova_gen3.xacro: drop the unconditional RealSense
D415 mesh instantiation at external_camera_link ("scene_camera") and the
unused wrist_realsense arg. Neither was wired to anything: the real
/scene_camera and /wrist_camera image topics come from mesh-free native
MuJoCo <camera> tags in the hand-authored MJCF scene files, not this URDF.
The mesh only ever showed up as an unexplained floating object in
RViz/planning-scene views.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xkm9aPVJ33L5QhRYs8M3u6
…pper warnings, fix serve_policy docs The xacro dropped the RealSense mesh/scene_camera link but the SRDF (shared verbatim from kinova_sim) still referenced it in disable_collisions entries, and the gripper's non-mimic finger joints were never marked passive or given group_state values. Fork a local SRDF so this config's collision matrix and group_state actually match its own URDF, and drop the now-unused external_camera/wrist_realsense urdf_params. Also document --service-ports on the serve_policy run command; docker compose run does not publish a service's ports by default, so omitting it left the bridge unable to reach the inference server. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ctive Rename cube_stack_scene.xml -> scene.xml and split its keyframes out into scene.xml/keyframes.xml (included via <include>): `default` plus one keyframe per layout (eval_0..eval_149) from the color-cube-stack eval dataset's meta/eval_layouts.jsonl, so the sim can be reset directly to a recorded cube arrangement via ResetMujocoKeyframe. config.yaml needs its own mujoco_model override even though the value now matches the xacro arg's default -- kinova_sim's config.yaml sets its own mujoco_model urdf_param, and dropping the child override lets that parent value win instead of the xacro default, so agent_robot.app tried to load a kinova_sim-only mujoco file that doesn't exist in this package. Add objectives/reset_to_eval_layout.xml as a minimal example (hardcoded to eval_0) of resetting to one of these layouts; a follow-up can parameterize keyframe_name at runtime to loop over all of them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ender Ports the same fix from end-to-end-colorstack's kinova_cube_stack_sim (identical mesh, geom placement, and wrist_camera setup verified before porting). MJWarp's ray tracer decides "is the camera inside this geometry" per triangle, which only works for convex shapes; wrist_camera sits inside the non-convex bracelet_with_vision_link mesh, so every ray hits the nearest interior triangle instead of passing through. Swap in a 19-hull CoACD convex decomposition (render-only geoms, group 2, same compiled placement as the original) and hide the original mesh (group 5). Verified against CPU MuJoCo's native renderer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HMkrP7XWDDkDUNVmQrHzEf
|
|
In a top-down pinch the cube's whole weight is carried by frictional contact
force, and at the default impratio=1 the friction constraint is too compliant
to hold it: the cube slides through the pads for the entire carry and is still
sliding at release. Median cube drift in the PAD frame over 32 held-out
layouts, max / creep:
oracle / GPU 12.7 / 6.2mm -> 2.94 / 0.27mm
oracle / CPU 1.7 / 0.5mm -> 2.93 / 0.22mm
pi0.5 / GPU 8.5 / 4.7mm -> 2.66 / 0.78mm
Load-bearing for policy eval, not just the oracle, and it fails SILENTLY: a
sinking cube still clears the 5cm lift gate, so pick barely moves (76.7 ->
80.0%) while stack halves (23.3 -> 35.0% at impratio=20, a within-condition
A/B). The cube is released ~15mm below where the policy thinks it is, against
an 18mm stack tolerance -- which reads as a bad policy, not a physics setting.
Sweeping impratio knees at ~10 and plateaus after, so 20 sits past the knee
with margin. Ruled out as causes: solver tolerance and iteration count,
narrowphase manifold size, and grasp height (+/-12mm moves slip <0.6mm).
Brings this cell in line with end-to-end-colorstack's kinova_cube_stack_sim,
which carries the full derivation. It was the only functional drift between the
two: the scene XMLs are already physics-identical, and the gripper damping and
actuator kv/forcerange had been ported previously.
Verified: scene.xml loads with impratio=20.0, nq=36.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYRSviSypggS2eCjAj9BxG
Mirrors end-to-end-colorstack's kinova_cube_stack_sim so Pro inference publishes what the square-trained pi0.5 checkpoints consume. pi0.5 letterboxes rather than crops (resize_with_pad, modeling_pi05.py:1191), so a 480x640 frame reaches SigLIP as 168x224 plus black bars and wastes a quarter of the grid. All three cameras: overview and scene in scene.xml, wrist in gen3_7dof.xml. fovy unchanged -- MuJoCo derives horizontal FOV from viewport aspect at fixed fovy, so 480x480 is exactly the centre crop of the old frame, matching how the v7-square training dataset was produced. Changing it would put square checkpoints off-domain. <visual><global> now sets offwidth/offheight=480: extract_cameras requires the offscreen framebuffer to equal every camera's resolution and was silently relying on MuJoCo's matching 640x480 default. Verified: scene loads, three cameras at 480x480, framebuffer matches, nq=36. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JYRSviSypggS2eCjAj9BxG
Standalone `train/` dir (own uv-pinned env, not built by colcon): a stock lerobot-train YAML config, a pi0.5-only LoRA-to-dense checkpoint merger, and a README covering the train -> merge -> serve loop and its --policy.path gotcha.
|
Forge labels the training `action` from a joint command topic, and no Behavior publishes one. joint_command_bridge.py assembles the trajectory controller's reference setpoint with the latched gripper command and republishes both. No Objective sets gripper_command_position yet, so the gripper channel records as a constant. Marked FIXME-CLAUDE and warned at startup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
Ports the scripted color-cube oracle (end-to-end-colorstack scripts/kinova/blockstack/oracle.py) onto Pro motion, so demonstrations can be collected through the Trainer instead of driven from MuJoCo directly. ComputeTopDownKeyposes is the only new Behavior: it builds one segment's Cartesian waypoints above a cube, aligned to whichever of the cube's four equivalent top-down yaws is nearest the wrist. Given the grasped object's transform in the tip frame it positions that object instead of the tip, which is what stacks a carried cube square on a target. Everything else composes from core Behaviors, with the cube poses coming from the `cube_*_tf` sites picknik_mujoco_ros already broadcasts. The oracle drives the MJCF `pinch` site, so the path plans for grasp_link offset 5 mm back onto it. Segments are timed at the oracle's FLOW_SPEED with a constant-speed Cartesian profile. That profile also needs an angular speed, which the oracle's smooth path does not budget; it is set low enough to keep cornering inside the joint acceleration limits. Verified on eval_0: red stacks on green within 1.1 mm laterally and 0.0 mm vertically, against the scorer's 18 mm and 12 mm gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
…ce spline The oracle IK-solves each keypose once, warm-started from the previous solution, then interpolates in joint space; it rejected per-waypoint Cartesian IK because the redundant arm's null space drifts between waypoints. PlanCartesianPath does exactly that, which cost 22.58 rad of joint travel against the oracle's 8.34 and put the release 90 degrees off. PlanJointSplineThroughPoses reproduces the oracle instead: warm-started IK chain, clamped cubic spline through the knots, duration paced by tip speed rather than joint rate. SendGripperCommand sends a goal without awaiting it, matching how ExecutePolicy drives the gripper at deploy time; awaiting stalls 3.9s on a grip that never reaches its commanded position, and cancelling the wait makes the controller rewrite its target to wherever the jaws are, pinning a weaker grip that slips. ComputeTopDownKeyposes now takes reuse_orientation so a grasp orientation is chosen once per cube and held for the carry, instead of re-deriving one that twists the cube. Over all 150 eval keyframes both policies stack 150/150, agreeing on every keyframe, with placement error medians of 0.72 mm (oracle) and 0.74 mm.
`Command Color-Stack Gripper` now sets `gripper_command_position` through `SetRos2Parameter` before every gripper command, so the warning that no Objective sets it, and that collected datasets therefore carry an unusable gripper action, is false and misleading on every launch. Verified by recording /joint_commands across a full stacking run: 186 messages at 10 Hz, gripper last of 8 channels, taking exactly [0.0, 0.6] -- the oracle's open and closed positions.
colcon test hung whenever a backend was running, because rclcpp::init wedged on discovery against the live graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
…in on The Trainer opens an episode some seconds after RecordEpisode returns, and forge drops everything before the start marker, so the oracle's opening motion was silently cut from every episode. WaitForEpisodeStart blocks until the session reports it is recording. Recording /joint_states also gave a fifteen-wide observation.state, eight of those joints passive Robotiq linkage. The bridge now republishes the same eight joints the action uses, and both streams publish from the first tick rather than waiting for the controller to announce its joint order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
360 training layouts, 60 per colour prompt, sampled disjoint from the 150 eval layouts the v7-square dataset holds out. The layout sidecar lives with the dataset, not here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
Record Color-Stack Episode records one demonstration per keyframe, and the per-prompt objectives loop it over their 60 training layouts. The keyframe queue is spelled out in each objective because LoopString's queue port is a deque, and only a literal is converted from the string spelling. Holding before StopRecording keeps the bag's trailing split readable: the snapshot rolls the bag to a fresh split, and stopping before any message reaches it leaves a file the converter aborts the whole dataset on. Seven of sixty episodes hit that without the wait, none of thirty with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
The Trainer's converter records only min/max/mean/std for state and action. pi0.5 normalizes both by quantile and cannot train without q01/q99, so combine_datasets backfills them through LeRobot before merging the per-prompt recordings into one dataset. Image statistics stay absent: LeRobot has no recompute path for them and overwrites them with ImageNet constants anyway, so use_imagenet_stats must be off or it fails writing into a camera entry that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
|
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PkhF7B8wF4M6XvMzw6Sc6r
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.