ROS 2 Hardware Acceleration Working Group

This document defines the scope and governance of the Working Group (WG). The rationale behind some decisions is further justified in REP-2008: ROS 2 Hardware Acceleration Architecture and Conventions (see pending PR).

Item	Description
Mission	Drive creation, maintenance and testing of acceleration kernels on top of open standards (C++ and OpenCL) for optimized ROS 2 and Gazebo interactions over different compute substrates (including FPGAs, GPUs and ASICs).
Scope	hardware acceleration in a) embedded (edge) devices, b) workstations, c) data centers and d) cloud
Objectives	2021 ✅ 1) Design tools and conventions to seamlessly integrate acceleration kernels and related embedded binaries into the ROS 2 computational graphs leveraging its existing build system (ament_acceleration extensions) 1, meta build tools (colcon-acceleration extension) and a new firmware layer (acceleration_firmware) 2.
	2021 ✅ 2) Provide reference examples and blueprints for acceleration architectures used in ROS 2 and Gazebo.
	2022 ✅ 3) Facilitate testing environments that allow to benchmark accelerators with special focus on power consumption and time spent on computations (see HAWG benchmarking approach, community#9, tracetools_acceleration, ros2_kria)
	2022 ⚠️ 4) Survey the community interests on acceleration for ROS 2 and Gazebo (see discourse announcement, survey).
	2022 ⚠️ 5) Produce demonstrators with robot components, real robots and fleets that include acceleration to meet their targets (see acceleration_examples).
	2022 ✅ 7) Acceleration of complete ROS 2 computational graphs ros-acceleration#20
	2022 🆕 8) Merge first hardware accelerators (kernels) into upstream packages (candidate: image_pipeline, see image_pipeline instrumented at #717 )
	2022 🆕 9) Documentation and a "methodology to hardware accelerate a ROS 2 package" (see ✅ #20)
	2022 🆕 10) Organize workshops on robotics and ROS 2 Hardware Acceleration
	2022 🆕 11) Robotic Processing Unit, first demonstrators #25

• ROS 2 Hardware Acceleration Working Group
  • The ROS 2 Hardware Acceleration Stack
  • Reference hardware platforms
    • Official
    • Unofficial
    • Adding your board
  • Subprojects
    • Subproject List
    • Standards for subprojects
    • Adding new subprojects
    • Subproject changes
    • Deprecating subprojects
  • Governance
    • Meetings
    • Communication Channels
    • Backlog Management
    • Membership, Roles and Organization Management
    • Modifying this governance document
  • References

The ROS 2 Hardware Acceleration Stack

The ROS 2 Hardware Acceleration Stack is a series of extensions to ROS 2 which allow to leverage hardware acceleration and create custom compute architectures providing a faster ROS 2 execution and a timing-safe event-driven programming interface. The stack is composed of 4 key elements and aims to be hardware and vendor-agnostic:

#	Standard/convention	Title	Rationale
1	REP 2009	Type Negotiation Feature	Allow ROS 2 Nodes to dynamically negotiate the message types used by publishers and subscriptions, as well adaptively modifying the behavior of publisher and subscriptions to align with accelerators
2	REP 2008	ROS 2 Hardware Acceleration Architecture and Conventions	Architectural pillars and conventions required to introduce hardware acceleration in ROS 2 in a scalable and technology-agnostic manner. Presents one interface for all hardware acceleration vendors.
3	REP 2007	Type Adaptation Feature	An extension to rclcpp that will make it easier to convert between ROS types and custom, user-defined types for Topics, Services, and Actions.
4	REP 2000	ROS 2 Releases and Target Platforms	Production-grade multi-platform ROS support with Yocto

The ROS 2 Hardware Acceleration Working Group contributes and maintains open source implementations of various components of the stack above. A complete implementation of the ROS 2 Hardware Acceleration Stack including professional support, documentation, examples as well as reference designs is available at Acceleration Robotics' ROBOTCORE™. Other solutions that implement part of the stack include AMD's KRS (the Kria Robotics Stack), or NVIDIA's NITROS (NVIDIA Isaac Transport for ROS).

Reference hardware platforms

Official

The following boards are recommended and actively used for development:

Board	Picture	Description	Firmware
Robotic Processing Unit and ROBOTCORE®		A robot-specific processing unit specialized in ROS 2 computations. Features 16x CPUs, a GPU and an FPGA. Maps ROS computational graphs to the various compute units efficiently obtaining an increased throughput, lower latency, less power consumption, and additional real-time capabilities.	reach out for early access
Kria KR260 Robotics Starter Kit		The Kria™ KR260 robotics starter kit is built for robotics and industrial applications, complete with high performance interfaces and native ROS 2 support for ease of development by roboticists and software developers.	acceleration_firmware_kr260
Kria KV260 Vision AI Starter Kit		The Kria™ KV260 starter kit is a development platform for the K26, the first adaptive Single Board Computer. KV260 offers a compact board for edge vision and robotics applications.	acceleration_firmware_kv260

Unofficial

The following list includes boards that have been validated and have unofficial community support. No guarantees provided since no CI jobs are run on on these boards.

Community supported boards

Board	Picture	Description	Firmware
Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit		The ZCU102 Evaluation Kit enables designers to jumpstart designs for automotive, industrial, video, and communications applications. This kit features a Zynq® UltraScale+™ MPSoC with a quad-core Arm® Cortex®-A53, dual-core Cortex-R5F real-time processors, and a Mali™-400 MP2 graphics processing unit	acceleration_firmware_zcu102
AVNET Ultra96-V2		The Ultra96-V2 is an Arm-based, Xilinx Zynq UltraScale+™ MPSoC development board based on the Linaro 96Boards Consumer Edition (CE) specification and designed with a certified radio module from Microchip providing Wi-Fi and Bluetooth. All components are updated to allow industrial temperature grade options. Additional power control and monitoring will be possible with the included Infineon PMICs.	acceleration_firmware_ultra96v2
Zynq UltraScale+ MPSoC ZCU104 Evaluation Kit		The ZCU104 Evaluation Kit enables designers to jumpstart designs for embedded vision applications such as surveillance, Advanced Driver Assisted Systems (ADAS), machine vision, Augmented Reality (AR), drones and medical imaging. This kit features a Zynq® UltraScale+™ MPSoC EV device with video codec and supports many common peripherals and interfaces for embedded vision use case.
NVIDIA Jetson Nano		The Jetson Nano™ Developer Kit is a small, powerful computer that lets you run multiple neural networks in parallel for applications like image classification, object detection, segmentation, and speech processing. All in an easy-to-use platform that runs in as little as 5 watts.	acceleration_firmware_jetson_nano
NVIDIA Jetson AGX Orin		The Jetson AGX Orin Developer Kit is a powerful AI computer with up to 275 Trillion Operation per Second (TOPS).	reach out

Adding your board

The recipe for adding a new board to the community is as follows (reach out 📧 if you need help):

1. Check out the Initial draft of REP-2008 - ROS 2 Hardware Acceleration Architecture and Conventions
2. Create your own firmware repository (e.g. acceleration_firmware_Ultra96V2) for the corresponding board (see acceleration_firmware_kv260 for an example)
3. Once the firmware repo is finalized, assess the capabilities of the hardware according to REP-2008 and create a table in the README.md that argues about it (see example here)
4. Submit a PR to ros-acceleration/community to add your board to the community, with the corresponding support level according to REP-2008

Subprojects

This Working Group owns and maintains the following Subprojects. Its meetings and membership are largely focused on the direction, design, and work on the projects.

Subproject List

The following subprojects are owned by the Working Group:

Core Architecture

• ament_acceleration: CMake macros and utilities to include hardware acceleration in the ROS 2 build system (ament) and its development flows.
• ament_vitis: CMake macros and utilities to include Vitis platform into the ROS 2 build system (ament) and its development flows.
• colcon-hardware-acceleration: An extension for colcon-core to include Hardware Acceleration capabilities.
• ros2acceleration: acceleration command for ROS 2 command line tools.
• tracetools_acceleration: LTTng tracing provider wrapper for ROS 2 packages in the Hardware Acceleration Working Group.
• acceleration_firmware: Base ROS 2 package for hardware acceleration firmware. Used to organize firmware dependencies across vendors.

Examples and utilities for hardware acceleration

• acceleration_examples: ROS 2 package examples demonstrating the use of hardware acceleration.
• adaptive_component: A composable container for Adaptive ROS 2 Node computations. Select between FPGA, CPU or GPU at run-time.
• image_pipeline: A fork for the ros-perception image_pipeline package that includes hardware acceleration capabilities for demonstrations purposes.

Robotic Processing Unit (RPU)

• Robotic Processing Unit

Robotics MCU

• Robotics MCU meta-ticket

Standards for subprojects

Subprojects must meet the following criteria (and the WG agrees to maintain them upon adoption).

• Build passes against ROS 2 Humble
• If packages are part of nightly builds on the ROS build farm, there are no reported warnings or test failures
• Issues and pull requests receive prompt responses
• Releases go out regularly when bugfixes or new features are introduced
• The backlog is maintained, avoiding longstanding stale issues

Adding new subprojects

To request introduction of a new subproject, add a list item to the "Subprojects" section and open a Pull Request to this repository, following the default Pull Request Template to populate the text of the PR.

PR will be merged on unanimous approval from Approvers.

Subproject changes

Modify the relevant list item in the "Subprojects" section and open a Pull Request to this repository, following the default Pull Request Template to populate the text of the PR.

PR will be merged on unanimous approval from Approvers.

Deprecating subprojects

Projects cease to be useful, or the WG can decide it is no longer in their interest to maintain. We do not commit to maintaining every subproject in perpetuity.

To suggest removal of a subproject, remove the relevant list item in the "Subprojects" section and open a Pull Request in this repository, following instructions in the Pull Request Template to populate the text of the PR.

PR will be merged on unanimous approval from Approvers. If the repositories of the subproject are under the WG's GitHub organization, they will be transferred out of the organization or deleted at this time.

Governance

Meetings

Except for vacation periods and other exceptions, regular WG Meeting will generally happen once a month or more often. Meetings are announced in ROS Discourse and minutes are kept.

• Proposal for ROS 2 Hardware Acceleration Working Group (HAWG)
• ROS 2 Hardware Acceleration Working Group, meeting #1 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #2 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #3 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #4 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #5 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #6 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #7 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #8 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #9 (recording)
• ROS 2 Hardware Acceleration Working Group, meeting #10 (recording)

Communication Channels

• Instant messaging: Matrix community (Matrix is an open network for secure, decentralized communication).
• Meeting invite group: Meeting invite group
• Github organization: ros-acceleration
• Discourse tag: wg-acceleration

Backlog Management

Backlog management is performed using Github Projects. So far:

• Phase 1: tools, examples, benchmarking and first demonstrators (:warning: deprecated)
• ROS 2 Hardware Acceleration WG backlog

Membership, Roles and Organization Management

Working Group members may act in one or more of the following roles:

• Member
  • Prerequisite: Attend at least one out of the last three Working Group meetings
  • Responsible for triaging issues
• Reviewer
  • All reviewers are members
  • Prerequisite: Proven track record of high-quality reviews to WG subprojects
  • Responsible for reviewing pull requests
• Approver
  • All approvers are reviewers
  • Prerequisite: Proven track record of high-quality contributions and reviews to WG subprojects
  • Responsible for approving and merging pull requests
  • Responsible for vetting and accepting new projects into the Working Group
• Lead
  • Responsible for organizing and moderating working group meetings
  • Responsible for posting meeting materials (minutes, recordings, etc.)
  • Responsible for breaking ties

To become a member or change role, create an issue in this repository using the appropriate issue template. Such applications are accepted upon unanimous agreement from Approvers, and are typically based on the applicant's history with the subprojects of the Working Group.

Modifying this governance document

Changes to this document will be made via Pull Request. The PR will be merged on unanimous agreement from Approvers.

References

• [1] Mayoral-Vilches, V., & Corradi, G. (2021). Adaptive Computing in Robotics, Leveraging ROS 2 to Enable Software-Defined Hardware for FPGAs. arXiv preprint arXiv:2109.03276.

• [2] Mayoral-Vilches, V., REP-2008 - ROS 2 Hardware Acceleration Architecture and Conventions (see pending PR)

• [3] Mayoral-Vilches, V. (2021). Kria Robotics Stack A ROS 2-centric Approach for Hardware Acceleration in Robotics. Xilinx Inc.

• [4] Lienen, C., Platzner, M., & Rinner, B. (2020, December). ReconROS: Flexible Hardware Acceleration for ROS2 Applications. In 2020 International Conference on Field-Programmable Technology (ICFPT) (pp. 268-276). IEEE.

• [5] Lienen, C., & Platzner, M. (2021). Design of Distributed Reconfigurable Robotics Systems with ReconROS. arXiv preprint arXiv:2107.07208.

• [6] Wan, Z., Yu, B., Li, T. Y., Tang, J., Zhu, Y., Wang, Y., ... & Liu, S. (2021). A survey of fpga-based robotic computing. IEEE Circuits and Systems Magazine, 21(2), 48-74.

• [7] Mayoral-Vilches, V., Neuman, S. M., Plancher, B., & Reddi, V. J. (2022). RobotCore: An Open Architecture for Hardware Acceleration in ROS 2. arXiv preprint arXiv:2205.03929.

• [8] Ichnowski, J., Chen, K., Dharmarajan, K., Adebola, S., Danielczuk, M., Mayoral-Vilches, V., ... & Goldberg, K. (2022). FogROS 2: An Adaptive and Extensible Platform for Cloud and Fog Robotics Using ROS 2. arXiv preprint arXiv:2205.09778.

Footnotes

1. See ament_vitis ↩

2. See acceleration_firmware_kv260 for an exemplary vendor extension of the acceleration_firmware package ↩