RL-Scope Documentation

RL-Scope is a cross-stack profiler for deep reinforcement learning workloads.

For a tutorial on reproducing figures in the RL-Scope paper, see RL-Scope Artifact Evaluation.

For information on running the RL-Scope development docker container, see Host Configuration.

RL-Scope Artifact Evaluation

This is a tutorial for reproducing figures in the RL-Scope paper. To ease reproducibility, all experiments will run within a Docker development environment.

1. Machine configuration

Generally speaking, RL-Scope works on multiple GPUs models. The only limitation is that you need to use a GPU that supports the newer “CUPTI Profiling API”. NVIDIA’s documentation states that Volta and later GPU architectures (i.e., devices with compute capability 7.0 and higher) should support this API. If you attempt to use a GPU that is unsupported, the Docker build will fail, since we check for CUPTI Profiling API compatibility using a sample program (see dockerfiles/sh/test_cupti_profiler_api.sh).

The machine we used in the RL-Scope paper had a NVIDIA 2080Ti GPU. We have also reproduced results on an AWS g4dn.xlarge instance which contains a T4 GPU.

AWS

As mentioned above, we have reproduced results on an AWS g4dn.xlarge instance which contains a single T4 GPU. Please note that the other AWS instances that have more than one GPU are also fine (e.g., g4dn.12xlarge, p3.8xlarge), and will simply run the experiments faster by using multiple GPUs in parallel.

To make setup as simple as possible, we used NVIDIA’s Deep Learning AMI to create an VM instance, which comes preinstalled with Ubuntu 18.04, Docker, and an NVIDIA driver. If you wish to use a different OS image, just make sure you install the NVIDIA driver and Docker.

Regardless of the starting OS image you use, there is still some host setup that is required (which will be discussed in the next section).

2. Running the Docker development environment

In order to run the Docker development environment, you must first perform a one-time configuration of your host system, then use run_docker.py to build/run the RL-Scope container. To do this, follow all the instructions at Host Configuration. Afterwards, you should be running inside the RL-Scope container, which looks like this:

All remaining instructions will run commands inside this container, which we will emphasize with [container]$.

3. Building RL-Scope

RL-Scope uses a C++ library to collect CUDA profiling information (librlscope.so), and offline analysis of collected traces is performed using a C++ binary (rls-analyze)

To build the C++ components, run the following:

[container]$ build_rlscope

4. Installing experiments

The experiments in RL-Scope consist of taking an existing RL repository and adding RL-Scope annotations to it. In order to clone these repositories and install them using pip, run the following:

[container]$ install_experiments

5. Running experiments

The RL-Scope paper consists of several case studies. Each case study has its own shell script for reproducing figures from that section. The shell script will collect traces from each relevant algorithm/simulator/framework, then generate a figure seen in the paper in a corresponding subfolder output/artifacts/* of the RL-Scope repository.

RL Framework Comparison

This will reproduce results from the “Case Study: Selecting an RL Framework” section from the RL-Scope paper; In particular, the “RL framework comparison” figures, shown below for reference:

To run the experiment and generate the figures, run:

[container]$ experiment_RL_framework_comparison.sh

Figures will be output to output/artifacts/experiment_RL_framework_comparison/*.pdf.

RL Algorithm Comparison

This will reproduce results from the “Case Study: RL Algorithm and Simulator Survey” section from the RL-Scope paper; In particular, the “Simulator choice” figures, shown below for reference:

To run the experiment and generate the figures, run:

[container]$ experiment_algorithm_choice.sh

Figures will be output to output/artifacts/experiment_algorithm_choice/*.pdf.

Simulator Comparison

This will reproduce results from the “Case Study: Simulator Survey” section from the RL-Scope paper; In particular, the “Simulator choice” figures, shown below for reference:

To run the experiment and generate the figures, run:

[container]$ experiment_simulator_choice.sh

Figures will be output to output/artifacts/experiment_simulator_choice/*.pdf.

NOTE: Your reproduced graph will have a slightly different breakdown for Pong than seen above from the RL-Scope paper; in particular the simulation time will be closer to HalfCheetah. This is likely due to a difference in library version for the atari-py backend simulator used by Pong.

Host Configuration

In order to run the Docker development environment, you must perform a one-time configuration of your host system. In particular:

1. Install docker-compose: install docker and docker-compose.

2. NVIDIA driver: allow non-root users to access GPU hardware counters.

3. Docker default runtime: make GPUs available to all containers by default.

After you’ve configured your host system, you can launch the RL-Scope docker container:

4. Running the Docker development environment: build and run the container.

1. Install docker-compose

If your host does not yet have docker installed yet, follow the instructions on DockerHub for Ubuntu.

Make sure you are part of the docker UNIX group:

[host]$ sudo usermod -aG docker $USER

NOTE: if you weren’t already part of the docker group, you will need to logout/login for changes to take effect.

Next, we need to install docker-compose. To install docker-compose into /usr/local/bin/docker-compose, do the following:

[host]$ DOCKER_COMPOSE_INSTALL_VERSION=1.27.4
[host]$ sudo curl -L "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_INSTALL_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
[host]$ sudo chmod ugo+rx /usr/local/bin/docker-compose

2. NVIDIA driver

By default, the nvidia kernel module doesn’t allow non-root users to access GPU hardware counters. To allow non-root user access, do the following:

1. Paste the following contents into /etc/modprobe.d/nvidia-profiler.conf:

   options nvidia NVreg_RestrictProfilingToAdminUsers=0
   

2. Reboot the machine for the changes to take effect:

   [host]$ sudo reboot now
   

3. Docker default runtime

By default, GPUs are inaccessible during image builds and within containers launched by docker-compose. To fix this, we can make --runtime=nvidia the default for all containers on the host. To do this, do the following:

1. Stop docker and any running containers:

   [host]$ sudo service docker stop
   

2. Paste the following contents into /etc/docker/daemon.json:

   {
     "default-runtime": "nvidia",
     "runtimes": {
       "nvidia": {
         "path": "/usr/bin/nvidia-container-runtime",
         "runtimeArgs": []
       }
     }
   }
   

3. Restart docker:

   [host]$ sudo service docker start
   

4. Running the Docker development environment

The run_docker.py python script is used for building and running the docker development environment. In order to run this script on the host, you need to install some minimal “deployment” pip dependencies (requirements.deploy.txt).

First, on the host run the following (replacing [rlscope-root] with the directory of your RL-Scope repository):

# Install python3/virtualenv on host
[host]$ sudo apt install python3-pip python3-virtualenv

# Create python3 virtualenv on host
[host]$ cd [rlscope-root]
[host]$ python3 -m virtualenv -p /usr/bin/python3 ./venv
[host]$ source ./venv/bin/activate
[host (venv)]$ pip install -r requirements.deploy.txt

# Build and run RL-Scope the docker development environment
[host (venv)]$ cd [rlscope-root]
[host (venv)]$ python run_docker.py

After the container is built, it will run and you should be greeted with the welcome banner:

If you wish to restart the container in the future, you can do:

[host]$ cd [rlscope-root]
[host]$ source ./venv/bin/activate
[host (venv)]$ python run_docker.py

Unit tests

Running unit tests

RL-Scope has both python and C++ unit tests, which can be run either separately or all together.

To run all unit tests (i.e., both python and C++):

[container]$ rls-unit-tests

To run only C++ unit tests:

[container]$ rls-unit-tests --tests cpp

Output should look like:

To run only python unit tests:

[container]$ rls-unit-tests --tests py

Output should look like:

Python unit tests

Python unit tests made are using the pytest testing framework. Unit tests are written in the same module as the function they are testing. To locate unit tests, search for def test_ in a file.

C++ unit tests

C++ unit tests are made using the gtest testing framework. Unit tests are any/all files rooted under test whose filename matches test_*.[cc|cpp]. All unit tests are compiled into the rls-test binary.

Source documentation

• Index

• Module Index

• Search Page

• rlscope

Documentation TODOs