ROS distribution files

Abstract

This REP updates the specification of the ROS distribution files facilitated in the building, packaging, testing and documenting process.

The intention is to annotate a ROS distribution with more information, such as whether the distribution should use ROS 1 or ROS 2 semantics, to avoid the need to externally define and update these metadata.

Additionally a flag to enable checking for API/ABI compatibility is introduced.

This REP is an extension to the file format defined in REP 143¹. It currently does not repeat the content of REP 143 but only states the differences.

Motivation

Distribution metadata

For the following use cases additional metadata is currently necessary:

1. The ROS buildfarm has to distinguish ROS 1 distributions from ROS 2 distributions because they require different dependencies to build a workspace. E.g. for ROS 1 workspaces catkin needs to be installed - even for plain CMake packages - since it provides the workspace level setup files. For ROS 2 the package ros_workspace is needed since it provides the setup files for the workspace and therefore the environment necessary to find resources.
2. The ROS wiki shows the list of currently active ROS distributions. Since the rosdistro contains all ROS distros - EOLed ones as well as upcoming distributions before their release date.
3. A ROS distribution uses either Python 2 or Python 3. When a package is bloomed the rosdep keys are mapped to Debian package names and depending on the targeted Python version the conditional dependencies need to be evaluated.

In all cases the missing metadata is currently being hard coded in the source code and needs to be updated with every new ROS release. Therefore this REP aims to add the necessary metadata about a ROS distribution into the rosdistro instead.

API/ABI analysis

The stability of the different ROS distributions with respect to API/ABI stability has been part of the work done by ROS maintainers. Until now the changes are manually reviewed and their impact on the API or ABI is being judged by the reviewer with no automated tool to assist with this task.

The motivation of this REP is to ensure that undesired changes in terms of source/binary stability are caught in the review process and maintainers can make decisions with more information about the effects of changes.

The inclusion of an option for testing the API/ABI changes in ROS packages can cover at least two scenarios:

1. Run an API/ABI analysis together with changes coming from pull requests. Currently the option to use CI in the pull request is covered by the [test_pull_requests]{.title-ref} (as described in REP 143²)
2. Run an API/ABI analysis of changes that appear in the source code of ROS packages. This would be part of the CI cycle present in the devel scripts.

The original target to compare both use cases and to be taken as the stable API/ABI are the latest stable released packages for each ROS distribution under testing.

The new flag to check for ABI compatibility follows the same rationale as the two existing flags to run CI on every commit and/or pull request.

Specification

As of this REP unknown keys in the yaml files should be ignored (instead of resulting in an error). This will allow future additions in a backward compatible way without the need to bump the format version of each file.

Index file

• distributions
  • distribution:
    • distribution_status: an optional string describing the status of a ROS distribution. For the use case 2. the semantic of the following values is defined:
      • prerelease: An upcoming distribution which hasn't been released yet
      • active: A distribution which has been released and is actively being supported
      • end-of-life: A distribution which has reached its end of life
      • rolling: A rolling distribution as described by REP 2002³
    • distribution_type: an optional string describing the type of the ROS distribution. For use case 1. the values ros1 and ros2 will be used to distinguish the major ROS version.
    • python_version: an optional integer describing the major version of Python of the ROS distribution.
• version: version number, this REP describes version 4 (instead of version 3 described in REP 143⁴, version 2 described in REP 141⁵ and version 1 described in REP 137⁶)

Distribution file

• repositories
  • source
    • test_abi: a boolean flag used to enable API/ABI analysis of the code against the latest released packages. When test_commits is enabled the analysis is performed as part of the devel job. When test_pull_requests is enabled the analysis is performed as part of the pull request job.
• version: version number, this REP still describes version 2 (same as REP 143 ⁷). See the compatibility considerations below why a version bump was avoided.

Reference implementation

Distribution metadata

The index file changes have been implemented in version 0.7 of the Python package rosdistro. It serves as a reference implementation for this REP. The draft implementation can be found in⁸.

ABI analysis

The distribution file changes are implemented in version 0.8 of the Python package rosdistro. The draft implementation can be found in⁹.

Compatibility considerations

Distribution metadata

The draft implementation of rosdistro is able to parse multiple index format versions: 2, 3 as well as 4.

If the version of the existing index.yaml file would be bumped that would require every user to update to the latest version of the Python package rosdistro. While generating some friction it is especially a problem on some systems (e.g. Debian) which might not provide a newer version in stable distributions.

Therefore the existing index file isn't being changed but a sibling file named index-v4.yaml is being added. The file has the same content except that is uses version 4 as specified in this document and includes the additional metadata fields.

To make use of the new index file the new version of the Python package will update the default URL to point to the v4 file.

rosdistro version 0.7.5 or newer is necessary to access the python_version key. Older versions of rosdistro will simply ignore the key in the yaml file and not expose it through the API.

This provides a smooth transition for all users: users using the old version of the Python package can continue to use it as is, users updating to the newer version will benefit from the additional metadata. Python code using the rosdistro API can easily check if the metadata is present and if yes use it. If desired other Python packages can explicily depend on the newer version to ensure the v4 index is being used.

API/ABI analysis

The new test_abi could be added in a new format version 3 of the distribution file. That would ensure that implementations of this specification won't break if they decided to result in an error when unknown keys are found. The downside of bumping the distribution version would be that existing clients which haven't updated to the latest version of the rosdistro Python package would fail trying to parse that unknown version. With several EOL ROS distributions not receiving new releases of the Python package rosdistro anymore but still being used by many users this breakage should be avoided.

Therefore the new flag was added to the existing format version 2 instead. The reference implementation is already tolerant to unknown keys and simply ignores them so the newly added flag won't affect existing users.

rosdistro version 0.8.0 or newer is necessary to access the test_abi key. Older versions of rosdistro will simply ignore the key in the yaml file and not expose it through the API.

bloom

Beside using the rosdistro API bloom also includes an explicit check for the version of the index file¹⁰. Therefore a new patch release of bloom is required to also support the new version 4¹¹.

References

Copyright

This document has been placed in the public domain.

1. REP 143: http://www.ros.org/reps/rep-0143.html ↩︎

2. REP 143: http://www.ros.org/reps/rep-0143.html ↩︎

3. REP 2002: http://www.ros.org/reps/rep-2002.html ↩︎

4. REP 143: http://www.ros.org/reps/rep-0143.html ↩︎

5. REP 141: http://www.ros.org/reps/rep-0141.html ↩︎

6. REP 137: http://www.ros.org/reps/rep-0137.html ↩︎

7. REP 143: http://www.ros.org/reps/rep-0143.html ↩︎

8. Patch to python-rosdistro: https://github.com/ros-infrastructure/rosdistro/pull/124 ↩︎

9. Second patch to python-rosdistro: https://github.com/ros-infrastructure/rosdistro/pull/147 ↩︎

10. bloom asserting the index file version: https://github.com/ros-infrastructure/bloom/blob/d8be9d1d3469f00f936ad6e4869b847c5a6f8962/bloom/commands/release.py#L221-L223 ↩︎

11. bloom PR to support v4 https://github.com/ros-infrastructure/bloom/pull/493 ↩︎