Ruby failing to bind to port for creation of an rlc node in Cyclone DDS

[AscendedSky]

When launching Gazebo sim with either free fleet adapter or slotcar-based demo adapter using CycloneDDS (recommended DDS for Open-RMF), a crash is observed as shown in the following stack trace:

[gz-16] 1749789976.363501 [30]       ruby: ddsi_udp_create_conn: failed to bind to ANY:14926: Error
[gz-16] [ERROR] [1749789976.363672296] [rmw_cyclonedds_cpp]: rmw_create_node: failed to create domain, error Error
[gz-16] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[gz-16]   what():  failed to initialize rcl node: error not set, at ./src/rcl/node.c:252
[gz-16] Stack trace (most recent call last):
[gz-16] #31   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670c3a049, in
[gz-16] #30   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670b631d6, in rb_protect
[gz-16] #29   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d082d9, in rb_yield
[gz-16] #28   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d03b49, in rb_vm_exec
[gz-16] #27   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d0062b, in
[gz-16] #26   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cfc13e, in
[gz-16] #25   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cf992f, in
[gz-16] #24   Object "/usr/lib/x86_64-linux-gnu/ruby/3.2.0/fiddle.so", at 0x7f266be87b13, in
[gz-16] #23   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cc237b, in rb_nogvl
[gz-16] #22   Object "/usr/lib/x86_64-linux-gnu/ruby/3.2.0/fiddle.so", at 0x7f266be8743b, in
[gz-16] #21   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be580bd, in ffi_call
[gz-16] #20   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be553ee, in
[gz-16] #19   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be58b15, in
[gz-16] #18   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8-gz.so.8.9.0", at 0x7f266b329e62, in runGui
[gz-16] #17   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8-gui.so.8", at 0x7f266b1789ec, in gz::sim::v8::gui::runGui(int&, char**, char const*, char const*, int, char const*, char const*)
[gz-16] #16   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8-gui.so.8", at 0x7f266b177951, in gz::sim::v8::gui::createGui(int&, char**, char const*, char const*, bool, char const*, int, char const*, char const*)
[gz-16] #15   Object "/opt/ros/jazzy/opt/gz_gui_vendor/lib/libgz-gui8.so.8", at 0x7f2669ee6c98, in gz::gui::Application::LoadPlugin(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tinyxml2::XMLElement const*)
[gz-16] #14   Object "/opt/ros/jazzy/opt/gz_plugin_vendor/lib/libgz-plugin2-loader.so.2", at 0x7f26686d8188, in gz::plugin::Loader::Instantiate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const
[gz-16] #13   Object "/opt/ros/jazzy/opt/gz_plugin_vendor/lib/libgz-plugin2.so.2", at 0x7f26686f9eef, in
[gz-16] #12   Object "/opt/ros/jazzy/lib/rmf_building_sim_gz_plugins/libtoggle_floors.so", at 0x7f25f0e5960f, in std::_Function_handler<void* (), gz::plugin::detail::MakeInfo<toggle_floors, gz::gui::Plugin>()::{lambda()#1}>::_M_invoke(std::_Any_data const&)
[gz-16] #11   Object "/opt/ros/jazzy/lib/rmf_building_sim_gz_plugins/libtoggle_floors.so", at 0x7f25f0e4d71f, in toggle_floors::toggle_floors()
[gz-16] #10   Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f25f0ceca8b, in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&)
[gz-16] #9    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f25f0ceb63f, in rclcpp::Node::Node(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rclcpp::NodeOptions const&)
[gz-16] #8    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f25f0ceaf3d, in rclcpp::node_interfaces::NodeBase::NodeBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rclcpp::Context>, rcl_node_options_s const&, bool, bool, std::shared_ptr<rclcpp::CallbackGroup>)
[gz-16] #7    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f25f0cafb57, in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rcutils_error_state_s const*, void (*)())
[gz-16] #6    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc4f0c0, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
[gz-16] #5    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc39a54, in std::terminate()
[gz-16] #4    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc4f0d9, in
[gz-16] #3    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc39ff4, in
[gz-16] #2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f26706fd8fe, in abort
[gz-16] #1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f267071a27d, in gsignal
[gz-16] #0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f2670773b2c, in pthread_kill
[gz-16] Aborted (Signal sent by tkill() 2156 1000)
[rviz2-10] [INFO] [1749789976.726661184] [door_panel_requester_node]: New door [door1] found, refreshing...
[rviz2-10] [INFO] [1749789976.852366768] [door_panel_requester_node]: New door [door7] found, refreshing...
[rviz2-10] [INFO] [1749789977.237750177] [door_panel_requester_node]: New door [door_large1] found, refreshing...
[rviz2-10] [INFO] [1749789977.329662306] [door_panel_requester_node]: New door [door6] found, refreshing...
[gz-16] [GUI] [Msg] Added plugin [0m
[gz-16] [Msg] Publishing dynamic pose messages on [/world/sim_world/dynamic_pose/info]
[fleet_adapter-15] [INFO] [1749789979.304362876] [tinyRobot_fleet_adapter]: Opening door [door1] for [tinyRobot/tinyRobot1]
[gz-16] [INFO] [1749789981.256856354] [rclcpp]: signal_handler(signum=2)
[gz-16] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[gz-16]   what():  failed to create guard condition: the given context is not valid, either rcl_init() was not called or rcl_shutdown() was called., at ./src/rcl/guard_condition.c:67
[gz-16] Stack trace (most recent call last):
[gz-16] #31   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670c3a049, in
[gz-16] #30   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670b631d6, in rb_protect
[gz-16] #29   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d082d9, in rb_yield
[gz-16] #28   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d03b49, in rb_vm_exec
[gz-16] #27   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670d0062b, in
[gz-16] #26   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cfc13e, in
[gz-16] #25   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cf992f, in
[gz-16] #24   Object "/usr/lib/x86_64-linux-gnu/ruby/3.2.0/fiddle.so", at 0x7f266be87b13, in
[gz-16] #23   Object "/lib/x86_64-linux-gnu/libruby-3.2.so.3.2", at 0x7f2670cc237b, in rb_nogvl
[gz-16] #22   Object "/usr/lib/x86_64-linux-gnu/ruby/3.2.0/fiddle.so", at 0x7f266be8743b, in
[gz-16] #21   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be580bd, in ffi_call
[gz-16] #20   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be553ee, in
[gz-16] #19   Object "/lib/x86_64-linux-gnu/libffi.so.8", at 0x7f266be58b15, in
[gz-16] #18   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8-gz.so.8.9.0", at 0x7f266b32f66c, in runServer
[gz-16] #17   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8.so.8", at 0x7f266b035ad7, in
[gz-16] #16   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8.so.8", at 0x7f266b0437f6, in gz::sim::v8::SimulationRunner::Run(unsigned long)
[gz-16] #15   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8.so.8", at 0x7f266b042e81, in gz::sim::v8::SimulationRunner::Step(gz::sim::v8::UpdateInfo const&)
[gz-16] #14   Object "/opt/ros/jazzy/opt/gz_sim_vendor/lib/libgz-sim8.so.8", at 0x7f266b0410d9, in gz::sim::v8::SimulationRunner::UpdateSystems()
[gz-16] #13   Object "/opt/ros/jazzy/lib/rmf_building_sim_gz_plugins/libdoor.so", at 0x7f263dc4b5fc, in
[gz-16] #12   Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263dab2662, in rclcpp::spin_some(std::shared_ptr<rclcpp::Node>)
[gz-16] #11   Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263dab254a, in rclcpp::spin_some(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>)
[gz-16] #10   Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263dab8554, in rclcpp::executors::SingleThreadedExecutor::SingleThreadedExecutor(rclcpp::ExecutorOptions const&)
[gz-16] #9    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263daa99e4, in rclcpp::Executor::Executor(rclcpp::ExecutorOptions const&)
[gz-16] #8    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263dac9888, in rclcpp::GuardCondition::GuardCondition(std::shared_ptr<rclcpp::Context> const&, rcl_guard_condition_options_s)
[gz-16] #7    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0x7f263da9eb57, in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rcutils_error_state_s const*, void (*)())
[gz-16] #6    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc4f0c0, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
[gz-16] #5    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc39a54, in std::terminate()
[gz-16] #4    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc4f0d9, in
[gz-16] #3    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f266bc39ff4, in
[gz-16] #2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f26706fd8fe, in abort
[gz-16] #1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f267071a27d, in gsignal
[gz-16] #0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f2670773b2c, in pthread_kill
[gz-16] Aborted (Signal sent by tkill() 2155 1000)

The issue seems to be somewhat resolved (i.e. decreased frequency of crashes) with the addition of the following command in the ~/.bashrc file:

export CYCLONEDDS_URI='<CycloneDDS><Domain><Discovery><ParticipantIndex>none</ParticipantIndex></Discovery></Domain></CycloneDDS>'

Another option to try if the above seems to have no impact on frequency of crashes is to enable multicast by entering the following commands in your ~/.bashrc file:

export CYCLONEDDS_URI='<CycloneDDS><Domain><General><Interfaces><NetworkInterface name="lo"/></Interfaces><AllowMulticast>true</AllowMulticast></General><Discovery><ParticipantIndex>none</ParticipantIndex></Discovery></Domain></CycloneDDS>'

sudo ip link set lo multicast on

sudo ip route add 224.0.0.0/4 dev lo

Since a guaranteed fix has not been found, this issue will remain unresolved