Source Package Manager for Rosdep

Abstract

This REP defines a new "source" package manager for rosdep¹ that improves source-based installation of system dependencies. The new source package manager provides scripts for installing dependencies and verifying that they are present. This REP builds on the new rosdep syntax introduced in REP 111².

Motivation

The motivation for this REP is similar to REP 111³. To quote from REP 111:

... the bash script specification was brittle and provided limited information to rosdep itself. The scripts were required to be idempotent because rosdep was unable to detect whether or not the dependency was already installed.

The goal of this REP is to make rosdep more robust and flexible, as well as enable better support of platforms other than Ubuntu.

This REP specifically introduces a custom rosdep package manager that provides a new mechanism for source-based installations of software libraries. The goals for this new package manager are:

• Reliable, source-based installation of thirdparty libraries for any ROS platform.
• Enable rosdep to check that source-based dependencies are present.
• Allow source rosdeps to have prerequisites, i.e. to depend on other rosdeps.
• Enable any community member to contribute rosdep-compatible packaging for libraries.
• Enable automated verification of source-based dependencies without superuser access.
• Reduce duplication in rosdep source installation rules.

There are several anticipated use cases for this new source-based package manager. The first is to provide generic scripts for POSIX-based systems for common thirdparty libraries. These generally can be improved by integrating the native package manager for that platform, but the source mechanism provides an initial working system.

A new, anticipated use case is improving support for ROS on platforms that lack official package managers. For example, support for ROS on Windows is improving and will soon require better infrastructure for satisfying rosdep dependencies. Similarly, although OS X users can use MacPorts for rosdep dependencies, this REP provides an alternative mechanism that is more stable and easier to lock to specific OS X versions.

Specification

rosdep.yaml additions

rosdep.yaml syntax: source PACKAGE_MANAGER, fields

According to REP 111⁴, the basic syntax for a rosdep.yaml rule is:

ROSDEP_NAME:
  OS_NAME1: 
    PACKAGE_MANAGER1:
      PACKAGE_ARGUMENTS_A
  OS_NAME2: 
    PACKAGE_MANAGER2:
      PACKAGE_ARGUMENTS_A

The source package manager defines the PACKAGE_MANAGER and PACKAGE_ARGUMENTS fields as follows:

• PACKAGE_MANAGER is source
• PACKAGE_ARGUMENTS: the source package manager accepts a dictionary with three possible keys:
  • uri (string): the primary download uri for the rosdep manifest (required).
  • alternate-uri (string): a mirror download uri for the rosdep manifest (optional).
  • md5sum (string): md5sum hash of the rdmanifest for security and verification. (optional, recommended)

As defined by REP 111, the use of the source package manager is specific to a OS platform.

The use of the md5sum field is strongly recommended. It provides protection against thirdparty library installs being compromised either due to corruption or malicious intent. It also enables the downloaded file to be verified for completeness.

rosdep.yaml example source rule:

The following example is a plausible rosdep.yaml rule for installing the yaml-cpp library on an Ubuntu system. This rule references an external rosdep manifest file, as well as an mirror of this installation for additional robustness.

yaml-cpp:
  ubuntu:
    source:
      uri: 'https://kforge.ros.org/rosrelease/viewvc/sourcedeps/yaml-cpp/yaml-cpp-0.2.5.rdmanifest'
      alternate-uri: 'https://mirror-kforge.ros.org/rosrelease/viewvc/sourcedeps/yaml-cpp/yaml-cpp-0.2.5.rdmanifest'
      md5sum: 740a2193cea112f5ef682d9f69f9c7b1

rosdep manifests (.rdmanifest)

The rosdep.yaml rule for the source package manager points to an external rosdep manifest (.rdmanifest) file that provides the necessary URLs and scripts for managing a system dependency.

There may one or more than one .rdmanifest file for a given software package dependencing on the installation and platforms needs of the software. For example, a single .rdmanifest file may be sufficient for POSIX-based systems, but a separate .rdmanifest may be necessary for OS X systems. This choice is not enforced by this REP and is up to the developer providing the integration.

.rdmanifest syntax

The rosdep manifest (.rdmanifest) file has 6 fields:

• uri (string): The uri of the tarball to download.
• md5sum (string): The expected md5sum of the tarball (recommended)
• alternate-uri (string): An optional alternate-uri for reliability (recommended)
• check-presence-script (multi-line string): a script that exits with return code zero if a package is already present, non-zero otherwise. This script is downloaded to a temporary file and executed with the user's account.
• install-script (multi-line string): a script that installs software library from tarball. The script is downloaded to a temporary file and executed with the user's account. The working directory set to the exec-path, which is generally the directory where the unpacked tarball is located.
• exec-path (string): relative path used to specify working directory of the install-script (optional). The path is interpreted to be relative to the directory the tarball is downloaded and unpacked to. exec-path is optional and defaults to '.', i.e. the directory of the tarball.
• depends (string list): list of rosdep dependencies that must be installed before checking or installing this rosdep.

check-presence-script and install-script

The check-presence-script and install-script are text-based scripts that are downloaded and executed by rosdep. rosdep will assign these scripts an executable permission and they may be written in any executable text format supported by the target platform.

For example, on POSIX-based systems, scripts can use a shebang to specify the desired interpreter of the script, such as #!/bin/sh or #!/usr/bin/env python.

The scripts are executed using the user's account. In the event that a script requires superuser access, it must explicitly invokes commands like sudo on its own. It is strongly recommended that the check-presence-script not use any superuser privileges as this interferes with automated verification of dependencies. This recommendation is only for the check-presence-script as it is expected that an install-script will need to use superuser permissions for final install steps, e.g. make install.

Source package manager implementation

In the event that rosdep determines that the source package manager is responsible for a rosdep dependency, the following steps are taken by the source package manager to verify a system dependency is installed.

1. Download the .rdmanifset file referenced by the uri rule. If the primary uri is unavailable, attempt to download the alternate-uri.
2. If md5sum is specified, validate the md5sum of the .rdmanifest file.
3. Verify that all rosdep dependencies specified in depends in the .rdmanifest are installed.
4. Download the check-presence-script specified in the .rdmanifest file.
5. Execute the check-presence-script with the current user's account.

If the script returns zero, the installation is assumed to be present. Otherwise, rosdep may determine that it is necessary to perform an installation of this dependency. If the source package manager is asked to install a dependency, the following steps are followed:

1. Download the tarball specified in the uri rule. If the primary uri is unavailable, attempt to download the alternate-uri.
2. If md5sum is specified, validate the md5sum of the tarball.
3. Unpack the tarball.
4. Download the install-script specified in the .rdmanifest file. The working directory of the install-script specified by exec-path, which defaults to the directory the tarball is unpacked to.
5. Execute the install-script with the current user's account.

.rdmanifest example

The following example demonstrates a plausible rosdep manifest file for the yaml-cpp library. This particular rosdep manifest file leverages the dpkg tool and thus will only work on Debian-based systems. A more generic rosdep manifest file could be written to work on a greater variety of platforms, or different rosdep manifest files could be written to support other platforms.

uri: 'https://kforge.ros.org/rosrelease/viewvc/sourcedeps/yaml-cpp/yaml-cpp-0.2.5.tar.gz'
md5sum: b17dc36055cd2259c88b2602601415d9
install-script: | #      #!/bin/bash
  set -o errexit
  mkdir -p build
  cd build
  cmake ..
  make
  echo "About to run checkinstall make install"
  sudo checkinstall -y --nodoc --pkgname=yaml-cpp-sourcedep make install
check-presence-script: | #      #!/bin/bash
  dpkg-query -W -f='${Package} ${Status}\n' yaml-cpp-sourcedep | awk '{\
  if ($4 =="installed")
    exit 0
  else
    print "yaml-cpp-sourcedep not installed"
    exit 1}'
exec-path: yaml-cpp-0.2.5
depends: [checkinstall ]

Rationale

depends

The rosdep manifest file follows the recommendation of REP 111 and provides a dependency specification. The depends syntax in the rosdep manifest provides greater flexibility when integrating software packages. It functions similar to [dpkg]{.title-ref} and [rpm]{.title-ref} dependencies, where it provides an ordering for installing a hierarchical set of libraries. It also enables developers to use multiple package managers to satisfy dependencies. For example, the library could specify dependencies that resolve to apt or yum installations, and then perform a source-based install on top.

rosdep manifest files

The new rosdep manifest files provides for a separation of roles when integrating libraries into rosdep: those that provide wrappers for thirdparty libraries to be used in ROS, and those that integrate these into rosdep.yaml files in a ROS stack hierarchy. Authors of thirdparty libraries can provide integration with rosdep without having to release separate ROS stacks.

This approach does comes with some disadvantages. In the previous implementation of source-based installs for rosdep, the entire installation script was contained within the rosdep.yaml rule. This enabled the rosdep.yaml file to be complete: no additional specifications were necessary, other than implicit resources referenced by the script. REP 111 instead requires developers to also to create and host a separate rosdep manifest file.

Although this separation has additional overhead, it has benefits beyond the separation of roles described above. First, the inline syntax leads to more duplication. Two platforms wishing to both used a source-based install must repeat the entire rule separately, even if the rules are identical.

The old inline approach also led to repetition in cases where two ROS stacks, without a common ancestor, wished to integrate the same rosdep library. As these stacks do not have a common ancestor, each stack must add the rosdep library in its own rosdep.yaml file. This REP reduces the duplication as each rosdep.yaml file only needs to point at the external rosdep manifest file.

Finally, the inline syntax of the old rosdep format decreased the readability of the rosdep.yaml files. which is an important concern when maintaining these files over time.

References and Footnotes

Copyright

This document has been placed in the public domain.

#####

Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

1. rosdep documentation (http://www.ros.org/wiki/rosdep) ↩︎

2. REP 111: Multiple Package Manager Support for Rosdep (http://www.ros.org/reps/rep-0111.html) ↩︎

3. REP 111: Multiple Package Manager Support for Rosdep (http://www.ros.org/reps/rep-0111.html) ↩︎

4. REP 111: Multiple Package Manager Support for Rosdep (http://www.ros.org/reps/rep-0111.html) ↩︎