User Manual

User Manual:

Open the PDF directly: View PDF .
Page Count: 67

Download
Open PDF In Browser	View PDF

MONROE
Measuring Mobile Broadband Networks in Europe
H2020-ICT-11-2014
Project number: 644399

Deliverable User manual
MONROE Platform User Manual

Editor(s):
Contributor(s):

Miguel Peón-Quirós, Özgü Alay, Vincenzo Mancuso
Miguel Peón-Quirós, Thomas Hirsch, Ali Safari Khatouni

Work Package:
Revision:
Date:
Deliverable type:
Dissemination level:

5.2 / User Support
1.0
March 14, 2019
R (Report)
Public

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Abstract
This document describes the processes that MONROE experimenters need to follow to create,
run, monitor and collect results from their experiments.

Participant organisation name

Short name

SIMULA RESEARCH LABORATORY AS (Coordinator)
CELERWAY COMMUNICATION AS
TELENOR ASA

SRL
Celerway
Telenor

NETTET SVERIGE AB

NET1

NEXTWORKS

NXW

FUNDACION IMDEA NETWORKS

IMDEA

KARLSTADS UNIVERSITET

KaU

POLITECNICO DI TORINO

POLITO

2 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Contents
1 Introduction

6

1.1 MONROE nodes hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

1.1.1 System design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Overview of the node configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6
7

1.3 Overview of the experimental workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

2 Experiment preparation

9

2.1 General experiment notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

2.2 Container preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

2.2.1

Package and tool installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2.2

Band Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2.3

Virtual Machine support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2.4

NEAT support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3 Optional interactive debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Mandatory certification process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.6 Life cycle of monroe/base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Resource allocation, and experiment scheduling and monitoring

16

3.1 User login and certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.1

Installation of user certificates in Chrome . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.2 Resource allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.1

Eduroam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3 Experiment scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3.1

Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.3.2

Checking availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.3.3

First availability scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.4 Experiment monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5 Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.5.1

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.5.2

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 Retrieval of metadata and experiment results

32

4.1 User experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2 MONROE metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5 Run-time considerations for experimenters

33

5.1 Node identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 Communication during the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 Interface naming and default route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.4 Interface binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.5 Metadata at run-time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.5.1

Example: Correlate experiment results with metadata at run-time . . . . . . . . . . . . . 35

5.5.2

Metadata information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3 of 67

Project no. 644399

User manual
MONROE Platform User manual

5.5.3

Public
Rev. 1.0/ March 14, 2019

Metadata format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.6 Tstat at run-time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.6.1

Tstat Round Robin Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.6.2

Tstat logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.7 Access to user-owned development nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.7.1

Accessing user-owned development nodes . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6 Monitoring node status

42

7 MONROE templates, examples and default experiments

42

7.1 Example template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.1.1

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

7.1.2

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

7.1.3

Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

7.1.4

Overview of the code structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

7.2 Docker miscellaneous usage notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
7.3 Experiment: ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.3.1

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.3.2

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.3.3

Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

7.4 Experiment: http_download . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7.5 Experiment: Tstat & mPlane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7.5.1

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.5.2

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.6 MONROE example: helloworld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.6.1

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7.6.2

Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7.6.3

Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7.7 MONROE example: paris-traceroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.7.1

Usage (inside a MONROE container) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.7.2

Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.7.3

Additional remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7.8 MONROE example: headlessbrowsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.8.1 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.9 MONROE example: pReplay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.9.1

Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

7.10 MONROE example: astream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.10.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.10.2 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.11 MONROE example: udpbwestimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.11.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.11.2 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.12 MONROE example: traceroute_background_experiment . . . . . . . . . . . . . . . . . . . . . . 53
7.12.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.13 Other containers in the repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

7.13.1 Container: metadata-subscriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.13.2 Container: tunnelbox-server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.13.3 Container: monroe_base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8 List of known bugs and issues

54

A List of packages installed in monroe/base

55

B Description of metadata fields

62

C How to map container folders to Windows paths

64

5 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 1: System design. Left: Tail node with one LTE Cat6 modem and one WiFi adaptor. Right: Head node
with two LTE Cat6 modems.

1

Introduction

The purpose of this document is to guide MONROE experimenters through the process of creating, running
and monitoring their experiments, and the subsequent collection of results. It first explains how the experiments must be prepared inside Docker containers, the testing process they must undergo before they can
be deployed into MONROE’s nodes, and how they must be uploaded to a repository for deployment into the
nodes. Then, it explains the basics of the web interface that allows provision of resources and the scheduling
of experiment executions. Finally, it shows how the experiment results can be retrieved either directly from
the experiment itself or from the repository provided by MONROE.

1.1

MONROE nodes hardware

The MONROE platform has gone through a complete process of analysis and redesign to adapt to the new
hardware available in the market and overcome some of the issues encountered in the first design. The
following paragraphs explain the main characteristics of the current design.
1.1.1

System design

The current MONROE design presents a heterogeneous set of nodes grouped in pairs:
• “Head,” with two Sierra Wireless LTE Cat6 modems.
• “Tail,” with one Sierra Wireless LTE Cat6 modem and one WiFi adaptor.
Figure 1 shows the current node design. Both types of nodes are based on a PC Engines APU2D4 motherboard with the following characteristics:
• 1 GHz 64-bit quad core AMD Geode APU.
• 4 GiB RAM.
• 16 GiB SDD.
• Three miniPCIe slots, two of which support a 3G/4G modem.

6 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

The current node design does not support a dedicated management MBB interface. Thus, additional
measures have been taken to minimize the interference of background traffic with user experiments. In
particular, most maintenance operations (except optionally the transfer of user results) are paused during
experiment execution. Also, a fix hour is reserved for maintenance in all nodes every day.

1.2

Overview of the node configuration

MONROE nodes have been designed to have minimal impact on the experiments that run on them. Therefore, only one experiment can run at a given time in a node. Although the experiments are executed inside a
Docker container, they have no quotas on CPU or memory usage, subject only to available node resources.
Container image size and temporary storage in the node may be restricted, though.
Every MONROE node runs, in addition to user experiments, the following background processes:
• The experiment scheduler, which arbitrates the execution of user experiments in the node. The scheduler runs permanently in the background and contacts periodically the scheduling server, sending
“heartbeats” and checking for new schedules for the node. When an experiment is not running, the
scheduler may start the deployment of the containers for one or several experiments scheduled to be
run in the immediate future, so that they are prepared on advance. The scheduler checks the duration of the slot assigned to an experiment; if the experiment does not finish on time, it stops the whole
container.
• Synchronization (rsync) services to copy data files to the MONROE repository. This service copies user
experiment results, the data collected by passive experiments and assorted metadata measurements.
It runs continuously, transferring files to the server as they appear in the corresponding folders. This
service uses the management interface, which is different from the interfaces available for the experiments. However, the management interface may share in some cases the same subscriber contract
with one of the experiment interfaces; operators might restrict the total bandwidth available for all the
SIMs linked to the same contract. Additionally, two modems (management plus experiment) using the
same operator antenna may somehow affect the bandwidth available for the experiment. Therefore,
experimenters should be aware of the small amount of data that can be transferred by this service in
parallel to their experiments.
• Several systems run continuously in the background gathering information on the status of diverse
components. Examples include a service to read the signal strength and network configuration of each
of the experiment modems, the GPS data and various node parameters such as CPU load, memory
usage or CPU temperature. These services run continuously in the background with a frequency that
varies from one second up to several minutes. Although their impact on user experiments should be
minimal, their existence must be known by the experimenters.
• In addition to the services that gather metadata, MONROE nodes keep several containers active all the
time. These containers run experiments that are deemed basic for the MONROE platform and include:
– A ping experiment. Container number 1 executes continuously an ICMP ping operation to a fixed
external server (currently, Google’s DNS at 8.8.8.8). The RTT values are collected and transferred
to the servers. The ping experiment runs continuously with a frequency of one second, for every
interface.

7 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Design experiment

Schedule container
in testing mode

Schedule
experiment

Configure container
for experiment

Deployment + test

Deployment /
execution

Store container in
repository

MONROE
certification

Retrieval of results
(per node / per
schedule)

Experiment design
phase

Testing phase

Experimentation
phase

Figure 2: Experimental workflow.
– A container that runs Tstat, the passive mPlane monitoring probe that collects, for each interface,
detailed flow level statistics. The Tstat container generates no traffic; flow level data is synchronized to the MONROE repository using the standard synchronization process described above.
• Finally, some built-in MONROE experiments run as scheduled containers. These experiments will not
run at the same time than user experiments:
– A bandwidth measurement test, which periodically downloads an object using the HTTP protocol
to measure the achievable bandwidth. The test runs on each interface. The periodicity of this
experiment and whether it can be run while user experiments are being executed are yet to be
decided.
– A container that periodically executes a paris-traceroute to several popular websites recording
information about all the intermediate hops. This container will in principle be run several times
per day, but the interactions with user experiments are yet to be determined.

1.3

Overview of the experimental workflow

Experiments conducted in the MONROE platform follow the workflow shown in Figure 2, which consists of
three phases: Experiment design, testing and experimentation. During the experiment design phase, the
experiment goals and properties are defined and the container required to deploy it in MONROE nodes is
configured. During the testing phase, the container is executed on nodes specifically devoted to testing new
experiments. If the experiment passes all the safety and behavior tests, a MONROE manager will digitally
sign the container image. Signed containers cannot be further modified without running again through the
testing phase. Finaly, the experimenter is free to schedule the experiment container on any nodes, subject
to the specific quotas assigned to their project.

8 of 67

Project no. 644399

User manual
MONROE Platform User manual

2

Public
Rev. 1.0/ March 14, 2019

Experiment preparation

2.1

General experiment notes

MONROE experiments run under the root user of a Docker container. Therefore, experimenters can design
any kind of experiment within the security restrictions of the platform, including the configuration of routing
tables, stopping or starting interfaces and executing any kind of applications. We assume the reader is familiar with the Docker technology. Otherwise, we suggest getting used to it by accessing the documentation at
https://docs.docker.com/engine/understanding-docker/ .
Creating and using containers is a two-step process. At design time, the experimenters create the image
for the container in their local machine using a container-creation script. If necessary, they can install new
packages (e.g., via apt-get) or copy libraries. The docker tools read the script and create the final image for
the experiment, which will then have to be uploaded to a repository. At run-time, the nodes retrieve the
container image from the repository and start it as scheduled.
During execution, the experiment should not install additional applications or download any data that
is not part of the experiment itself (e.g., if the experiment uploads a file to a server to test upstream speed,
either include the file to be uploaded in the container at design time or create it locally).
⇒ Experiments will under no circumstances allow direct ssh access to the node or any other form of running interactive commands from outside the container that can pose a security risk for the platform. ⇐

2.2

Container preparation

MONROE experiments are deployed in Docker containers (https://www.docker.com/). Preparing a new
container from MONROE’s base image is an easy process:
1. Install Docker in your machine. Do it preferably downloading the installation script from the web page,
rather than through a package manager such as apt-get:
$ wget https://get.docker.com -O install.sh
$ chmod u+x install.sh
$ ./install.sh

You will have to run docker as root unless you add yourself to the docker group.
Mac users: Download and install “Docker for MAC”
(https://www.docker.com/products/docker#/mac)
or the “Docker Toolbox”
(https://docs.docker.com/toolbox/overview/), according to your OS version.
2. Test the Docker installation with the ‘Hello world!’ example:
$ sudo docker run hello-world
Unable to find image ’hello-world:latest’ locally
latest: Pulling from library/hello-world
03f4658f8b78: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:xxxxxxxxxxxxxxxxxxxxxxxx
Status: Downloaded newer image for hello-world:latest

If everything has worked correctly up to here, you will see a welcome message similar to the following:

9 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Hello from Docker.
This message shows that your installation appears to be working correctly.
...

You can check which images are locally installed with:
$sudo docker images
REPOSITORY
TAG
hello-world
latest

IMAGE ID
690ed74de00f

CREATED
4 months ago

SIZE
960 B

3. Now you are ready to download the MONROE base image:
git clone https://github.com/MONROE-PROJECT/Experiments.git

This will fetch the repository with MONROE’s example containers.
4. Head to Experiments/experiments/template/. Here, you will find the required files to prepare your
image based on MONROE’s base. You should care about four things: a) The contents of the files/
folder, b) the build.sh file, c) the push.sh file and d) the template.docker script file that describes
how to create your container. In the directory files/ you can put all the files that are part of your
experiment. As a simple example, we can use the following script:
$vi files/myscript.sh
#!/bin/bash
ls -lah > /monroe/results/listing.txt

Any files that your experiment creates in /monroe/results are delivered to the repository, where you
will be able to retrieve them. Writes to any other part of the filesystem will be lost once the experiment is
finished. In periodic schedules, no data will survive from one execution to the next (i.e., the container
is loaded fresh before each execution). If result persistence is needed, the experimenter will have to
supply it by downloading the needed files from the network during the experiment itself.
5. You should not need to modify the build.sh file. The name of the container is the name of the current
directory, and it must match the name of the .docker file (e.g., template.docker as we are in a folder
named template/).
6. The file template.docker is the script used to build your container. You can modify it to:
• Define the entry point of your experiment (“ENTRYPOINT”).
• Change the base image of the container, e.g., monroe/base.
• Install additional packages or libraries.
For example:
FROM monroe/base
MAINTAINER your-email-address
COPY files/* /opt/monroe/
#Default cmd to run.
ENTRYPOINT ["dumb-init", "--", "/bin/bash", "/opt/monroe/myscript.sh"]

10 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

This example will copy the files in the files/ directory to the one you specify inside the docker container (e.g., /opt/monroe).
TIP: If you need to install additional packages in the container, be sure to clean any temporary files
from the image. Also, notice that the Docker creation script analyses the contents of the container
filesystem after every line in the .docker script is executed. That means that, even if you delete files at
the end, Docker will create intermediate “layers” that will be downloaded and applied sequentially to
build the final image of your container. Consider instead using one-liners such as the following:
RUN apt-get update && apt-get install -y vim && apt-get clean

This will ensure that the files are deleted before Docker analyses the filesystem.
7. Modify the file push.sh to reflect the name of your repository:
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CONTAINER=${DIR##*/}
CONTAINERTAG=myuser/myrepo # --> Modify to your own dockerhub user/repo
docker login && docker tag ${CONTAINER} ${CONTAINERTAG} && docker push ${CONTAINERTAG} && \
echo "Finished uploading ${CONTAINERTAG}"

During the development phase of your experiment, follow these steps to make your container accessible for the testing nodes:
• Create an account at Docker Hub.
• Create your own repository (you can create one container as private; no limits for public ones).
Containers for deployment on MONROE nodes must be public.
• In your development machine, run: docker login. It will ask you for your credentials. •
8. After populating the files/ directory, modifying the .docker file and updating the push.sh file, you are
ready to create the image:
$sudo ./build.sh
Using default tag: latest
latest: Pulling from monroe/base
Digest: sha256:6df1195a3cc3da2bfe70663157fddc42e174ec88761ead7c9a3af591e80ebbd5
Status: Image is up to date for monroe/base:latest
Sending build context to Docker daemon 11.26 kB
Step 1 : FROM monroe/base
---> d1b4f4baa60d
Step 2 : MAINTAINER mikepeon@imdea.org
---> Using cache
---> 0b05b5c453c7
Step 3 : COPY files/* /opt/monroe/
---> acc2df443070
Removing intermediate container 66a666516a27
Step 4 : ENTRYPOINT dumb-init -- /bin/bash /opt/monroe/myscript.sh
---> Running in f4b7a1ee804a
---> 096c7a56ff1c
Removing intermediate container f4b7a1ee804a
Successfully built 096c7a56ff1c
Finished building template

11 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

9. Test that your new docker container is available:
$sudo docker images
REPOSITORY
hello-world
your_docker_account/your_experiment
monroe/base

TAG
latest
latest
latest

IMAGE ID
690ed74de00f
xxxxxxxxxxxx
xxxxxxxxxxxx

CREATED
4 months ago
32 seconds ago
12 days ago

SIZE
960 B
626.6 MB
626.6 MB

Exact image ids and sizes will vary.
10. Push the container image to the repository:
$ sudo ./push.sh
Username (your-Docker-user-name):
Password: (type your DockerHub password)
WARNING: login credentials saved in /home/your-username/.docker/config.json
Login Succeeded
The push refers to a repository [docker.io/mikepeon/template]
5f339bfdaae2: Pushed
486ab26686cc: Layer already exists
034f70c0d9cd: Layer already exists
86b5acd8772a: Layer already exists
f03317610243: Layer already exists
50f6c1bd7ce6: Layer already exists
aec5953bffa2: Layer already exists
507169b05eea: Layer already exists
5d799297d10c: Layer already exists
759d76df9ac7: Layer already exists
5f70bf18a086: Layer already exists
12e469267d21: Layer already exists
latest: digest: sha256:c855de65307191b4832b2ec60a4401c1b63424827c29149703c5d7ef07b519f7
size: 3001
Finished uploading your-username/template

11. You can now test that your image runs correctly, even on your own PC (if the experiment logic and
resource demands allow for it).
$mkdir /run/shm/myresults
$sudo docker run -v /run/shm/myresults:/monroe/results your_docker_account/your_experiment
--> The output of your experiment will be in /run/shm/myresults/listing.txt

The docker command line allows you to specify a mapping between a directory inside the docker image and one in the host system. In this case, we have mapped /monroe/results from the container to
/run/shm/myresults. This is useful if you are running the container locally in a normal PC for debug-

ging purposes.
IMPORTANT: This process shows how to build and run a container locally in your workstation. However, experimenters do not have direct access to the MONROE nodes. Therefore, to execute your experiment in a MONROE node, you will follow the process just up to the sudo ./push.sh step and then
use the web interface to upload and schedule the container into the nodes.
You may check the contents of experiments/* for more useful examples.
The following is a list of useful common Docker commands:
• To list installed/built images (and get their ids):
12 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

docker images
• To list running containers and get their tags:
docker ps
• To stop running containers:
docker kill container-tag
• To delete images:
docker rm -f image_id
• To retrieve the latest version of an image (e.g., monroe/base):
docker pull monroe/base
• To attach to a running container and get an interactive shell:
docker exec -i -t container-tag bash
2.2.1

Package and tool installation

If you have to install extra packages, libraries or tools, do it from the my_experiment.docker file. You should
never pull repositories or download libraries from inside your experiment as this will count against your data
quota (and execution slot) for every instance of your experiment. Instead, modify the container configuration file as in the following example:
FROM monroe/base
MAINTAINER your-email-address
RUN apt-get update && apt-get install -y \
python \
python-pip \
traceroute \
&& apt-get clean
RUN pip install pygame
RUN mkdir -p /opt/yourname
COPY files/* /opt/yourname/
#Default cmd to run
ENTRYPOINT ["dumb-init", "--", "/bin/bash", "/opt/yourname/myscript.sh"]

You may also download any files using wget, but you may simply put them in the files/ folder as well.
Remember, this happens during container creation on your PC, not during experiment execution on the
nodes.
If you find the need for big libraries that you think should go into the base image, please contact MONROE’s administrators.
TIP: The easiest way to find out which packages and versions are available in the MONROE base image is
to create a simple container and run an interactive batch session inside it in your workstation. For example,
assuming that you have a basic container that simply waits when run, you may follow the following steps:
mkdir /run/shm/myresults
docker run -v /run/shm/myresults:/monroe/results repository/your_container &

13 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

LockType
4
8
16

DESCRIPTION
Lock in MODE_GSM
Lock in MODE_UMTS
Lock in MODE_LTE

Table 1: LockTypes

docker ps --> Look for the tag of your running container
docker exec -i -t container_tag bash
--> Here you are inside the container
dpkg -l > /monroe/results/package-listing.txt
exit
--> You’ll find the output at /run/shm/myresults/package-listing.txt

For easier reference, Table 10 in Appendix A gives a detailed listing of packages available in monroe/base
at the time of writing this text.
2.2.2

Band Locking

This option allows experimenters to lock the monroe node modems in LTE, 3G etc... by sending a PUT
request from within a container using different lock types see Table 1 using curl similar to:
curl -X POST -d ’{"lockType": 16}’ http://172.17.0.1:80/modems/1/lock
HTTP/1.0 200 OK

Where "lockType": 16 indicates that it has been locked on 4G LTE and modems/#/lock indicates the
modems ip4table to check ip4table first, or that the modem is locked run:
curl -s http://172.17.0.1:80/modems/ | jq
[
{
"ispName": "YOIGO",
"apnId": 77,
"apn": "internet",
"mode": "LTE",
"deviceType": 0,
"disconnects": 0,
"ip4table": 1,
"modemState": 7,
"deviceKind": 1,
"ifname": "wwan0",
"deviceMode": 5,
"deviceSubMode": 10,
"signalStrength": -56,
"modemName": "MC7455",
"locking": 16,
"imei": "359072060710833",
"iccid": "8934041514050773954",
"usb_vid": 4505
}
]

2.2.3

Virtual Machine support

This option allows experimenters to use a different kernel or kernel options than installed on the MONROE
node. To develop and deploy the experiments to the nodes we utyilize the docker container system. The
14 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

container will on the node be converted to a virtual machine (kvm) and executed in a environment similar to
a regular experiment.
A example on how to configure and run a virtual machine experiment can be found here: https://
github.com/MONROE-PROJECT/Experiments/tree/master/experiments/nodetest .
There are a couple of things to consider when deploying a experiment as a virtual machine;
• Base your docker image on monroe/base:virt
– monroe/base:virt is a stripped down version of monroe/base: https://github.com/MONROE-PROJECT/
Experiments/blob/master/monroe_base/10_virt_docker .
• Include a kernel
• Set the parameter "vm":1 when scheduling the experiment.
• Be very conservative on diskspace, eg installed packages etc.
• Define as the last line of /opt/monroe/user-experiment.sh how to start your experiment.
• Issue poweroff as last command of the experiment script.
2.2.4

NEAT support

This option allows experimenters to route all TCP traffic through a NEAT proxy https://github.com/
NEAT-project/neat . To enable this feature set the parameter "neat":1 when scheduling the experiment.
The NEAT rest api can be reached from inside a experiment container on url 172.17.0.1:45888, eg 172.
17.0.1:45888/pib and 172.17.0.1:45888/cib.

2.3

Optional interactive debugging

To speed up the process of debugging experiments in the nodes, three debugging paths are provided.
First, experimenters can order (buy) a number of “development” nodes to be hosted locally in their
premises. These nodes, which will not be accessible through the standard scheduler and user interface,
can be accessed through local interfaces (eth, serial console) and provide root access.
Second, and only for “testing” nodes, the user interface includes an option to provide an SSH public key
to the container. Once the container starts, experimenters can connect to monitor experiment progress. The
SSH session can extend until the container finishes or is stopped.
Finally, a virtual machine containing a “virtual MONROE node” has been designed to ease development
and debugging on a local PC. This virtual node replays metadata previously recorded from a real one.
CAUTION: Enabling ssh changes the entry point to the container, for this matter it is best not to use ssh
when you have designed your container just to make sure that the container starts automatically in the UI.

2.4

Mandatory certification process

MONROE experiments have to be certified before they can be executed by deployed nodes. A small number
of nodes are available through the user interface so that experimenters can test their experiments before
starting the certification process.
The certification process consists of the following steps:
1. The experimenter contacts their patron to inform them that a new version of their experiment is ready
for certification.

15 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

2. The experimenter has to provide a summary of the experiment, i.e., overall purpose, design and implementation (reasonable length around 0.5 to 1 A4 pages).
3. The container should be submitted as source (i.e., build scripts for Docker, not tools source code) for
easy inspection by the patron. Additionally, this will allow the MONROE administrators to update the
containers when a new version of monroe/base is available.
4. The patron, or the maintenance team, will then build (or pull) the container, tag it as partnername/
experimentname, and push it to the deployed docker repository.

Every experiment submitted to the MONROE testbed must first pass through a testing process to receive
manual approval by a MONROE administrator. To submit your experiment for testing, you have to use the
web interface specifying “testing” as the required node type.

2.5

Deployment

MONROE’s scheduling system will automatically deploy experiments to the nodes before their execution
time. The nodes will fetch the container image from the Docker repository, and the size of the download will
be accounted in your data quota. Notice that in the case of periodic experiments, each time an experiment
is run, the Docker container may have to be re-downloaded and its costs will be accounted in your quota.

2.6

Life cycle of monroe/base

The current version of monroe/base deployed on nodes is tagged as “latest.” New versions will be tagged as
“staging;” their existence will be announced on the experimenters mail list. Experimenters must check their
experiments against the new stagging version to verify that no incompatibilities appear. Any relevant issues
can be discussed with the MONROE administrators. After a reasonable period of time, the new version will
be retagged as “latest,” and deployed into the nodes. All the containers should have been built against the
new version at this time to avoid wasting quota resources when they are deployed in the nodes.

3

Resource allocation, and experiment scheduling and monitoring

Once a experiment is configured as a Docker container, it can be scheduled multiple times under different
conditions using the user client web located at https://www.monroe-system.eu .

3.1

User login and certificates

User identification in MONROE is achieved through client certificates. Every experimenter has their own
certificate compatible with the FED4FIRE1 federation. User certificates are issued by iMinds through the
following URL: https://authority.ilabt.iminds.be/. New users must create a new account (“sign
up”). Be sure to select the option “Join Existing Project” and type the name “Monroe” in the project name
field (Figure 3). The authorization process involves a manual verification step by one of the MONROE administrators, so it will probably take one or two days.
Please, notice that the current policy for MONROE is to use one user certificate per project, shared between all the experimenters belonging to that project.
1 http://www.fed4fire.eu/

16 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 3: iMinds registration page to obtain FED4FIRE-compatible certificates for use with the MONROE
platform.
Once the identity of the experimenter is approved, they will receive an informative email. They should
then log into the iMinds webpage to download the certificate files (PKCS12). These files must be installed in
the experimenter browser. After that, the user should be able to access the user web directly. Upon request
of the main (index.html) file, the browser will contact MONROE servers to verify that the user credentials
are correct. In the case of any problems, the user will be presented with instructions on how to obtain a
certificate. If the client certificate is verified successfully, they will be automatically redirected to the listing
of their experiments.
NOTE: User certificates are manually activated in the scheduling software. To use your certificate, please
send its SSL ID (“fingerprint”) to one of the MONROE administrators (e.g., mailto:mikepeon@imdea.org,
mailto:mohamed.moulay@imdea.org). You may find it in the screen after pressing the “Try me” button,
once the certificate is correctly installed in your browser:
{
"fingerprint": "c79f1967aea17811a1ebed39b7d718430904338a",
"user": {
"id": 3,
"name": "MONROE Test admin",
"quota_data": 50000000000,
"quota_storage": 500000000,
"quota_time": 500000000,
"role": "admin",
"ssl_id":"c79f1967aea17811a1ebed39b7d718430904338a",
},
"verified": "SUCCESS"
}

⇒ We have identified some common issues that are not yet solved. Below are some workarounds:
• For the first login, you may be asked for your user certificate and then your browser may show a security
warning. This is due to the use of a self-signed server certificate. Please ask your browser to proceed.

17 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Then, you will probably see an error page from MONROE. Please, click the red button labeled “Try me”
and check that you get a successful data output. Finally, please proceed again to the main page of the
project. From that point, you should be able to access the system without further problems in future
sessions. (Pointers on how to simplify this issue are welcome!)
• Firefox on OSX has an issue with CORS headers. Although the web and scheduling servers are running
now on the same machine, you may still encounter this problem.
⇒ Reminder: All user certificates will end on the 31st of July 2018.
3.1.1

Installation of user certificates in Chrome

This section explains how to install the FED4FIRE-compatible user certificates used by the MONROE platform in Google Chrome for Windows. The procedure for other browsers and platforms should be similar.
1. Go to your browser settings page:

2. Display the advanced configuration settings:

18 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

3. Go to the section labeled “HTTPS/SSL” and click the button “Manage certificates...”:

4. The dialog box for managing your certificates will be displayed. Press the button “Import...” to import
19 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

your certificate:

5. In the new dialog, click “Next”:

6. In the file-selection dialog that appears next, change the file type from “X.509 Certificate (*,cer;*.crt)”
to “Personal Information Exchange”:

20 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

7. And select the file containing your certificate:

8. In the next dialog box, enter your certificate password:

21 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

9. If the import is successful, your certificate will be imported to your “Personal Store” and you will be
able to access the MONROE user interface by selecting it when prompted by your browser. Notice that
you may still get a warning about the validity of the server certificate.

3.2

Resource allocation

The “New” tab allows assigning resources and scheduling new experiments. Here, the user will be presented
with a page similar to Figure 4.
To create a new experiment, at least the following parameters must be specified:
Name: A representative experiment description.
Script: A Docker hub path for the experiment container. In the previous example, it would be your_docker_
account/my_experiment. Experiments on deployed nodes must be lodged in MONROE’s repository:
docker.monroe-system.eu/... (the final URL is communicated when the container is certified).

Number of nodes: The number of nodes that must execute the experiment.
Duration: Length of the experiment execution, in seconds (excluding the time required to deploy the container). The node will kill the experiment after this time. The minimum slot that can be reserved is 5 min
and the maximum, 24 h. However, because of the scheduling of MONROE experiments, the maximum
possible duration is in practice slightly less than three hours.
If the starting date is fixed, the user can introduce it in the field “Start.” All dates are introduced as UTC
times; the interface presents alongside the corresponding local time for the user’s browser. The scheduler
will then try to satisfy the requirements.
Alternatively, if the starting date is not relevant, the user may leave this field empty and press the button
“Check availability” to check the earliest available slot (add at least ten minutes to the proposed time to
allow for container deployment into the nodes). If the user just wants to submit the experiment as soon as
possible, they can just mark the option “As soon as possible” and leave the other fields empty when pressing
the “Submit” button.
Additionally, the user may specify the following restrictions (Figure 5):
22 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 4: Example for the creation of a new experiment.

23 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Country filter: The user may select nodes located in one or several countries, or they may choose to use
nodes from any country indistinctly.
Node type: Currently available node types are deployed or testing. The testing nodes are reserved for experiments that must still be verified by a MONROE administrator. Experiments on deployed nodes must
be lodged in MONROE’s repository: docker.monroe-system.eu/.
Node model: Until complete retirement of the old MONROE nodes, experiments can run on old or new
nodes. Eventually, all experiments will be run on new nodes.
Number of interfaces: New MONROE nodes come in pairs of co-located nodes, where one node (“head”)
has two 4G interfaces, and the other (“tail”) has one 4G interface and one WiFi interface. Specifying the
number of required interfaces for the experiment restricts the type of nodes that can be selected by the
scheduler:
• One interface: The scheduler chooses only nodes with one 4G interface and WiFi (tails).
• Two interfaces: The scheduler chooses only nodes with two 4G interfaces (heads).
• Three interfaces: The scheduler chooses pairs of co-located nodes. In that case, the number of
requested nodes must be even, as each pair is counted as two nodes. The assignment is atomic,
i.e., either the complete pairs will be secured or the complete assignment will fail.
The numbering of nodes follows the convention that the head in a pair receives number n and the
co-located tail receives number n + 1, where n is even.
Node IDs: If the experimenter wants to use a set of specific nodes, for example, to repeat one experiment
under the very same conditions, it is possible to introduce a comma-separated list of required nodes,
instead of accepting any available ones.
Active-data quota: The experimenter must specify the active-data quota for each interface, that is, the maximum amount of data that each interface can use. The scheduler checks this value against the quota
available for the user.
Log files quota: The user may want to place an estimate on the maximum amount of data that may be generated as result files in /monroe/results. This is important because the size of the results is also counted
against the user quota.
Deployment-storage quota: This is the size allocated for the container file system in the node. Bigger sizes
require more time to deploy. The maximum limit is 1 GB.
Additional options: The user may provide a set of comma separated key-value pairs. These options will be
appended to the JSON-formatted configuration file that can be read at run-time by the container at
/monroe/config. This mechanism enables experiment parameterization.

3.2.1

Eduroam

In the additional option tab experimenters can use eduroam, which is a roaming service for researchers or
students in universities across Europe, to use eduoroam see Figure 6 the key values are similar to:
"_eduroam": { "identity": "100368007@****",

"hash": "YourpasswdHash" }

When the experiments starts the docker container will initialize wpa_supplicant to bring up the Wlan0 interface up, as it will be available
in the network namespace alongside op0, op1. The start log and the container network namespace will be similar to:

24 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 5: Filters for node selection.

Figure 6: Eduroam configuration.

25 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

$Start Log$
Starting container... Successfully initialized wpa_supplicant
Killed old client process
Loading iptables rules
ok.
Starting accounting.
Loading iptables rules
Started docker process xx.
Startup finished Tue Month x xx:xx:xx UTC 20xx.
Started allocator client
JSON request: {"address":"xxx.xxx.xx.xx","ifname":"wlan0","addr_family":x,"cmd":0,"version":1}
Sent 80 bytes
Server: tas_socket Table: xxxxx Lease: xxxxxx Ifname: wlan0 Address: xxx.xxx.xx.xx Family: x
Table: xxxxx

root@xxxxxxx:/# ifconfig
eth0: flags=xx mtu 1500
inet xxx.xx.x.x netmask xxx.xxx.xxx.x broadcast x.x.x.x
lo: flags=xx mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
op0: flags=xx mtu 1500
inet xxx.xx.x.x netmask xxx.xxx.xxx.xxx broadcast x.x.x.x
wlan0: flags=4xxx mtu 1500
inet xxx.xx.xx.xx netmask xxx.xxx.xxx.xxx broadcast x.x.x.x

Noting that the eduroam parameters are hidden parameters, but not encrypted and they are still stored in the scheduler database.

3.3

Experiment scheduling

When all the requirements are specified, the user needs to click the “Submit experiment” button to submit to the scheduler. The
experiments must respect several restrictions to be successfully scheduled:
• The starting time must be at least 10 min in the future, to allow time for container deployment.
• No experiment can be scheduled more than one month in advance.
• Periodic experiments must have a period greater than 3600 s. The finishing time must also obey the previous rule, that is, the last
experiment instance in the recurrence must be scheduled in less than a month from the current time.
• No experiment (or instance in a series) can last more than one day. In practice, the longest period that an experiment will be
awarded is less than 3 h.
• If a list of specific nodes and a starting date are given, the scheduler may be unable to grant the required resources.

3.3.1

Recurrence

MONROE’s scheduler allows to specify experiments that need to be repeated periodically. In that case, the user has to specify the
repetition period (≥ 3600 s) and the final stopping date. The scheduler will treat each repetition as a different experiment and will try to
satisfy the requirements for each of them consecutively. However, the operation is atomic: Either all the repetitions are scheduled, or
none are.

3.3.2

Checking availability

If the exact starting time is not relevant, the user can press the “Check availability” button. If the requirements can be satisfied, a
message explaining when the experiment might be started will be displayed. Additionally, it will also inform of the maximum number
of nodes that can be used during this period, and the maximum ending time. With these data, the experimenter may decide to increase

26 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

the number of nodes that run the experiment, or increase its duration until the time that the scheduler is likely being able to grant.
Figure 7 shows the answer of the scheduler for an availability query.

3.3.3

First availability scheduling

The MONROE scheduler allows users to define experiments with an execution window instead of a fixed starting time. This option is
particularly useful with mobile nodes whose active times are difficult to foresee. With first availability scheduling, the user specifies
requirements as usual, except the starting date; instead, a temporal window within which the experiment can be scheduled is defined.
The scheduler will assign a node or set of nodes to the experiment, without committing a specific time slot. When the node contacts
the scheduler (e.g., when a mobile bus becomes online), if it has no pending tasks, the scheduler will assign one of the tasks in the firstavailability queue. Figure 8 shows the definition of an experiment for first availability scheduling. In the time between the experiment
definition and the assignment of an execution window to each of its tasks, the status of the experiment tasks shows no start and stop
times, as shown in Figure 9. The start and stop times of the experiment show the boundaries of the execution window for all the tasks.
First availability scheduling presents some particularities that users should consider:
• If several nodes are requested, execution on each node is asynchronous and thus not guaranteed to be simultaneous.
• Experiments may not be executed if the end of the scheduling window is reached without a chance to execute them.
• Tasks in the first-availability queue are not deployed on advance to the nodes. The user should take this into consideration,
increasing the “duration” of the experiment to include long deployment times.
• The scheduler checks if the first-availability request has a chance to be fulfilled at all, under the current circumstances.
• By default, the execution window ends 24 h after the current time.

3.4

Experiment monitoring

Once an experiment is successfully submitted, the user can check its progress under the “Status” tab. Figure 10 shows an example of a
list of experiments.
All the active (i.e., not completed) experiments for the user are shown. Experiments that have not yet been started can be canceled
and deleted. However, the scheduler will try to stop experiments that have already started, but they will not be deleted from the list.
Clicking on any experiment displays the details for its individual schedules. There, the number of schedules that are defined but
not yet deployed, the ones that are deployed and ready to be started, the ones that are currently running, etc., is summarized. One line
is presented for each individual schedule on each MONROE node. Table 2 explains the states in which an individual task may be.
Some experiments may be designed to finish after completion. For those ones, the correct finishing state is “Finished.” If they are
stopped by the scheduler, they probably exceeded the execution time foreseen by the experimenter. However, other experiments may
be designed to run continuously for a period of time. In those cases, the “Stopped” state could actually be the correct ending state
as intended by the experimenter. Moreover delayed, and failed states can have different meanings From insufficient disk space to the
docker container not being found as explained in Table 3.

27 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

NewExperimentCheckAvailability.png

Figure 7: The scheduler may supply hints on the scheduling availability, including the earliest starting date
that is possible, the end of the availability period for the required resources and the maximum number of
nodes, with the specified requirements, that the experiment could reserve. In this example, the experiment
can start on “2017-02-28 16:04:08 UTC” and can last until “2017-02-28 16:45:02 UTC.” The experiment can
be scheduled with up to 36 nodes during this period.

28 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 8: Example of a first availability scheduling.

Figure 9: Status of an experiment in the fist availability queue without defined starting and stopped times.

29 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 10: List of user experiments.

30 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 2: Experiment states

STATE

DESCRIPTION

(Ongoing states)
Defined
Requested
Deployed
Delayed
Started
Restarted

The experiment is created in the scheduler. If a task remains in this state past its starting time, the node was probably shut down and the task will not be executed anymore.
The node has requested the container and is deploying it.
The node has already deployed the container and is waiting for its starting time.
The scheduling process failed temporarily.
The container is being executed in the designated node. The “download” link for the
task results is already available.
The node has restarted the experiment after a node failure.

(Final states)
Finished
Stopped

Failed
Canceled
Aborted

The task was correctly executed and it finished on its own before consuming the complete time slot.
The task was correctly executed, but it was stopped by the scheduler at the end of the
execution slot (correct for tasks designed to remain in execution until the end of their
time slot).
The task stopped abnormally.
The task was canceled by the user before being started (but other tasks in the experiment were already started).
The task was aborted by the user after being started.

Table 3: Experiment problems

STATE

DESCRIPTION

Deployed

•
•
•
•

Container does not exist, or i/o timeout.
Insufficient disk space.
Cannot deploy while node is in maintenance mode.
Container downloading in background.

Failed

•
•
•
•
•
•
•
•
•

Storage quota exceeded during deployment.
Container exited immediately.
Network namespace does not exist.
Container image not found.
Started into maintenance mode.
Docker run command failed.
Docker run exit code 127: entry point not found?
Deployment time exceeded stop time, skipped start hook.
Deployment time exceeded stop time.

31 of 67

Project no. 644399

User manual
MONROE Platform User manual

3.5

Public
Rev. 1.0/ March 14, 2019

Command Line Interface

The MONROE scheduler REST API is normally used through the provided WEB interface. However, experimenters can use it directly
to improve task automation. A complete command-line tool is available at https://github.com/ana-cc/monroe-cli.2 The following paragraphs describe how to install and use the tool. Note that due to a certificate issue the tool will only run on Debian based
systems.

3.5.1

Installation

Installation prerequisites:

sudo apt install git python3-dev python3-setuptools build-essential libffi-dev libssl-dev python3-straight.plugin
From the directory that contains the tool sources:
python3 setup.py develop
The user certificate can be imported with:
$ monroe setup MyCert.p12
Enter passphrase:
Your certificate files were stored in ~/.monroe
Alternatively, the cert/key files can be extracted manually and placed under ∼/.monroe/ with the names mnrCrt.pem and

mnrKey.pem.

3.5.2

Usage

The tool has integrated help:
$ monroe -h
usage: monroe [-h] Command ...
Monroe Cli
optional arguments:
-h, --help
show this help message and exit
Experiment:
The following commands can be used to create and submit experiments
Command
Description
create
Creates an experiment
whoami
Displays MONROE user details
quota
Displays MONROE quota details
experiments
Display recent experiments
setup
Specifies MONROE user certificate to use for accessing the
scheduler
delete
Deletes an experiment
results
Downloads the results for an experiment
Correct installation of the tool can be tested trying to retrieve the user identification:
$ monroe whoami
Authentication ID: 2, Name: MONROE Test user, Storage Quota remaining: 49597346816 bytes
User quotas can be easily retrieved:
2 The

command-line interface tool has been provided by Ana Custura, from the University of Aberdeen Court, in the context of the
MONROE-PREC project.

32 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 11: Folder containing the results of an individual schedule, transferred to MONROE’s servers.

$ monroe quota
2017-05-05 : Remaining time is 138888.00 hours.
2017-05-05 : Remaining storage quota is 46.00 GB.
2017-05-05 : Remaining data quota is 46.00 GB.
Experiments can be easily submitted, and customized with various options:
$ monroe create monroe/base --nodecount 5 --duration 600
Allocated task 1289.
The tool allows retrieving the list of recent experiments:
$ monroe experiments
Experiment ID: 6798 Name: mike test on new nodes Script: steven76/headless Summary: {u’aborted’: 1}
Experiment ID: 6799 Name: mike test on new nodes Script: steven76/headless Summary: {u’aborted’: 1}
Experiment ID: 6800 Name: mike test on new nodes Script: steven76/headless Summary: {u’aborted’: 1}
The results of an experiment can also be automatically retrieved:
$ monroe results 6479
This command downloads the experiment results into 6479/sched_id, for all the task IDs associated to that experiment.
For experiment creation, the tool supports defining SSH access to the container (in testing nodes), additional options and recurrence. Finally, the tool can be used as a library:
from monroe.core import *

4

Retrieval of metadata and experiment results

MONROE repositories contain two types of data: MONROE metadata itself and the results of user experiments.

4.1

User experiment results

Any files written during the experiment to the /monroe/results directory will be synchronized to the experiment repository. This
operation happens continuously during experiment execution and then upon its finalization. Therefore, it is advisable that only final
files ready to be transferred are copied (indeed, mv’ed) to that location to avoid the system to sync temporary files and consume your
quota or produce invalid results. This recommendation means that files in the results folder should not be updated; experimenters are
encouraged to copy intermediate result files as soon as they are ready so they can retrieve partial results if the experiment fails in the
middle of its execution.
The result files can be accessed through the user interface: For experiments that have already been started, the interface presents a
link under the column “Results” that redirects the user to the HTTP folder (Figure 11) that contains the files already synchronized from
the node where the experiment runs to the repository. In this way, the experimenter can retrieve result files even for partial experiments
that fail or are canceled.
In addition, the experiment may use any network functionalities to communicate with outside servers as needed (e.g., scp some
files to an external server). In order to improve safety, private keys should be restricted to the experiments and discarded after a reasonable time. Additionally, instead of saving your keys in the container itself, you may want to pass them as additional options during
experiment scheduling. The values will be available during container execution as a JSON file at /monroe/config. Notice that this
file is created by the node scheduler. The same effect is achievable when the containers are run manually in user development nodes

33 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

adding the -v option to the command line. To map both a locally created config file and the results folder of the container to a node
folder, in development nodes without a scheduler, the following command line fragment may be used:3
-v /monroe/results:/monroe/results -v /monroe/config:/monroe/config:ro

4.2

MONROE metadata

MONROE metadata can be freely accessed by two means. First, a CSV dump of all database tables is generated daily. The files can
be accessed at the following URL (a valid user certificate is needed): https://www.monroe-system.eu/user/dailyDumps/. The
dump files should be available every day after 12:00 CET (24-hour format). Each file covers the period [00:00, 00:00) GMT.
Our servers run on CET time, but metadata timestamps use GMT. Therefore, to cover “a day” of metadata using, e.g., the local time
in Norway, two CSV files need to be combined. During Winter time, the needed metadata is in the period [day0 :01:00, day1 :01:00).
During Summer time, the needed metadata is in the period [day0 :02:00, day1 :02:00).
Alternatively, metadata can be accessed directly in a replica of MONROE’s Cassandra database, which is updated daily approximately at noon with the data from the previous (GMT) day. Access credentials for this server will be provided as requested. MONROE
repositories include several examples on how to access the database.

5

Run-time considerations for experimenters

This section discusses several considerations that experimenters must take into account when designing and running their experiments
on the MONROE platform.

5.1

Node identification

An experiment can identify the node it is running on by reading the contents of the /nodeid file:
cat /nodeid
54

5.2

Communication during the experiment

During execution, the experiment is free to establish any network communications through the available interfaces. The user can
choose to bind explicitly from each command or application to a specific interface, or they may define default routes during the experiment:
route add default gw 172.16.0.1 eth0

5.3

Interface naming and default route

To offer a consistent view of the platform resources, whereas allowing flexibility for future changes in the platform configuration, the
following naming scheme is used for each of the interfaces available for the experiments:
op0:
op1:
op2:
eth0:

First mobile interface.
Second mobile interface.
Only for old nodes, third mobile interface.
Ethernet (wired) network connection, when available.

The platform guarantees that a given opi corresponds to the same operator during experiment execution. However, the assignment
may change between nodes in the same country or even between successive executions in the same node. Therefore, experiments must
check the metadata stream to select the correct interface associated to the desired operator.
Under some circumstances, the mobile devices used in the MONROE nodes may loose connectivity, reset themselves or undergo
any other process that makes them temporarily unavailable for the experiments. To identify and tackle with these situations, experimenters are encouraged to build “robust” experiments subscribing to the corresponding metadata streams.
If the experimenter writes their own code:
1. Subscribe to the metadata broadcast.
2. Wait for a MODEM.*.UPDATE message for the modem(s)/operators of interest.
3 Thanks to Eneko Atxutegi Narbona and Jonas Karlsson for pointing this out.

34 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

3. Once this information is obtained, use the desired interface and store the results with the corresponding ICCID or operator
name.
4. Should the interface disappear (ENODEV error, “no such device”), start over at 2.
When using an external tool that does not handle ENODEV (e.g., “fping”), replace step 4 by:
• Monitor the metadata for a MODEM.*.CONNECTIVITY message indicating that connectivity was lost, or monitor the interface
list to check if the device disappears. Upon either event, start over at 2.
Experimenters should take notice that an interface may not only go down, but it may actually disappear from the list of available
interfaces (e.g., if the modem has to be restarted). Even if it reappears soon after, any existing network connections on the old interface
will fail with ENODEV.
It is also possible to skip steps 1 and 2 when reconnecting to an interface after a failure, as the interface name corresponding to the
desired operator is already known. It is still necessary to keep retrying to connect to the interface, until it comes up.

5.4

Interface binding

Experiments running in MONROE nodes have access to several network interfaces. By default, that is, if the experiment does not
take any special configuration actions, the default route will be configured to one of the mobile broadband interfaces, if available.
However, experimenters have the possibility of explicitly binding external tools or their programs to specific interfaces. Several options
are available to bind an experiment to an interface.
1. Most standard tools can be instructed to use an specific interface:
ping -I op0 host_name
tcpdump --I op0 target
wget --bind-address ...
curl --interface ...
2. Explicit binding in the source code.
• In C:
snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "op0");
setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, (void *)&ifr, sizeof(ifr))
localaddr.sin_addr.s_addr = inet_addr("192.168.1.100");
bind(sockfd, (struct sockaddr *)&localaddr, sizeof(localaddr));
• In Python:
s = socket.socket()
s.bind(’192.168.1.152’, 0))
3. Library overloading the bind() and connect() functions through LD_PRELOAD:
http://www.ryde.net/code/bind.c.txt
4. Changing the default route:
route del default gw ...; route add ...

5.5

Metadata at run-time

MONROE nodes retrieve constantly some metadata information concerning their own state and the network conditions. This information is continuously uploaded to the MONROE servers and stored in a database. One of the main goals of the MONROE project is to
make all that information freely accessible. Therefore, experimenters may perform an off-line correlations of events in their experiment
with the information in the MONROE database.
MONROE experimenters can also access all the metadata information at run-time from their experiments to achieve easy correlation of events or modify the behavior of the experiment during its execution. For example:
• Experiments that depend on external factors (location):
–
–
–
–
–

Round trip time vs. location.
Proactive HTTP caching according to location.
Round trip time vs. base station.
Round trip time vs. signal strength.
Route selection according to current conditions.

• Experiment validation:

35 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

– Verify that node temperature is/was within limits.
– Verify that system load is/was below threshold.
The metadata is broadcast locally using ZeroMQ. The following excerpt in Python shows how an application can subscribe to the
metadata stream:
import zmq
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect ("tcp://172.17.0.1:5556")
# An empty string subscribes to everything:
topicfilter = ’’
# E.g., use ’MONROE.META.DEVICE.GPS’ for GPS-only metadata
socket.setsockopt(zmq.SUBSCRIBE, topicfilter)
while True:
string = socket.recv()
print string

5.5.1

Example: Correlate experiment results with metadata at run-time

The following example shows how to create an application that executes a ping to an external machine and saves the results alongside
the node location:
• Pipe the ping command through a “ping formatter.”
• The “ping” formatter subscribes to a zmq socket and topic:
– Socket : ’tcp://172.17.0.1:5556’
– Topic : ’MONROE.META.DEVICE.GPS’
• Cache the GPS position received.
• Wait for output from the ping command (stdin).
• Store experiment information including the GPS position:
– Use the “library” monroe_exporter (python only).
– Call the monroe_exporter script via the command line.
Below is the corresponding source code:
socket.connect(’tcp://localhost:5557’)
socket.setsockopt(zmq.SUBSCRIBE, ’MONROE.META.DEVICE.GPS’)
LAST_GPS_FIX = None
monroe_exporter.initalize(’MONROE.EXP.PING’, 1, 5.0)
’’’fork and wait for for gps messages’’’
while True:
(topic, msgdata) = socket.recv_multipart()
LAST_GPS_FIX = json.loads(msgdata)
’’’main process waits for ping experiment output ’’’
while line:
exp_result = r.match(line).groupdict()
msg = {
’InterfaceName’: interface,
’Bytes’: int(exp_result[’bytes’]),
’Host’: exp_result[’host’],
’Rtt’: float(exp_result[’rtt’]),
’SequenceNumber’: int(exp_result[’seq’]),
’TimeStamp’: float(exp_result[’ts’])

36 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

}
if LAST_GPS_FIX != None:
msg.update(
{
’GPSTimeStamp’: LAST_GPS_FIX[’TimeStamp’],
’Latitude’: LAST_GPS_FIX[’Latitude’],
’Longitude’: LAST_GPS_FIX[’Longitude’],
’Altitude’: LAST_GPS_FIX[’Altitude’],
’NumberofSatellites’: LAST_GPS_FIX[’NumberofSatellites’]
})
monroe_exporter.save_output(msg)
line = sys.stdin.readline()

5.5.2

Metadata information

Currently, the collected metadata includes:
•
•
•
•

Node GPS.
Node sensors (CPU temp) and probes (load, memory usage).
Modem status and events.
Continuous and scheduled internal experiments:
– RTT (through ping).
– Bandwidth (through HTTP download).

The following and some examples of the information received in the metadata stream:
• RTT experiment:
{"DataId": "MONROE.EXP.PING", "Bytes": 84, "NodeId": "54",
"SequenceNumber": 301, "DataVersion": 1, "Timestamp": 1465805479.747943,
"Rtt": 71.2, "Host": "8.8.8.8", "Operator": "Orange",
"Iccid": "8934014251541036013", "Guid":
"sha256:a9f9fb2c04bba3782ef2624e118faa18f16b08c826155cae5e1ea7e1d88832b5.0.54.3791"}
• Sensors, where each message may contain information about a different set of measurements:
{"DataId": "MONROE.META.NODE.SENSOR", "softirq": "205270", "SequenceNumber": 48581,
"DataVersion": 1, "b": "1059270", "b": "4885494", "guest": "0", "NodeId": "54",
"idle": "42657942", "user": "10480984", "irq": "0", "steal": "0",
"Timestamp": 1465786966.123456, "nice": "3063"}
{"DataId": "MONROE.META.NODE.SENSOR", "SequenceNumber": 48567, "DataVersion": 1,
"Timestamp": 1465786961.123456, "percent": "65.98", "NodeId": "54", "current": "302234",
"start": "1465484726", "total": "5246545.72", "id": "39"}
{"DataId": "MONROE.META.NODE.SENSOR", "SequenceNumber": 48460, "DataVersion": 1,
"Timestamp": 1465786926.123456, "apps": "3632746496", "NodeId": "54", "free": "483119104",
"swap": "0"}
• Modem events:
{"DataId": "MONROE.META.DEVICE.MODEM", "InterfaceName": "usb2", "CID": 72209509,
"DeviceState": 3, "SequenceNumber": 33548, "DataVersion": 1,
"Timestamp": 1465803136.123456,
"NWMCCMNC": 21404, "Band": 3, "RSSI": -80, "IPAddress": "10.33.101.173",
"IMSIMCCMNC": 21404, "DeviceMode": 5, "NodeId": "54", "IMEI": "864154023645179",
"RSRQ": -8, "RSRP": -85, "LAC": 28014, "Frequency": 1800,
"InternalIPAddress": "192.168.0.153", "Operator": "YOIGO",
"ICCID": "8934041514050774002", "IMSI": "214040113950108"}
• GPS:
{"DataId": "MONROE.META.DEVICE.GPS", "SequenceNumber": 34164, "DataVersion":
"Timestamp": 1465805718.123456, "Altitude": -1455.900024, "NodeId": "63",

37 of 67

1,

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 4: Metadata topics.

TOPIC

DESCRIPTION

*.DEVICE.MODEM.iccid.UPDATE
*.DEVICE.MODEM.iccid.MODE
*.DEVICE.MODEM.iccid.SIGNAL
*.DEVICE.MODEM.iccid.LTEBAND
*.DEVICE.MODEM.iccid.ISPNAME
*.DEVICE.MODEM.iccid.IPADDR
*.DEVICE.MODEM.iccid.LOCCHANGE
*.DEVICE.MODEM.iccid.NWMCCMNCCHANGE
*.DEVICE.GPS
*.NODE.SENSOR.sensor_name
*.NODE.EVENT

Temp sensor, running experiments, quotas, . . .
Power up events, etc, . . .

"Longitude": -3.777019, "NMEA":
"$GPGGA,081518.0,4020.002011,N,00346.621107,W,1,02,500.0,-1455.9,M,53.0,M„*5D\r\n",
"SatelliteCount": 2, "Latitude": 40.333366}

5.5.3

Metadata format

Metadata and internal experiment results follow a JSON structure, as detailed in https://github.com/MONROE-PROJECT/Experiments/
wiki :
• All Metadata messages have a topic according to Table 4. Appendix B gives the complete description of the meaning of all the
metadata fields.
• All metadata topics are prefixed with “MONROE.META.”
• All internal experiments are prefixed with “MONROE.EXP.”
• Experiments receive metadata messages only for topics to which they subscribe.
• An empty string ("") subscribes to all topics.

5.6

Tstat at run-time

The Tstat (http://www.tstat.polito.it/) runs on all nodes in the mPlane container as one of the basic MONROE containers.
The Tstat is a passive probe able to provide several insight on the traffic patterns at both the network and the transport levels. The Tstat
generates two different types of logs.

5.6.1

Tstat Round Robin Database

The RRD (Round Robin Database) logs is an average of samples of each packet in 5 minutes, it imposes at least 5 minutes delay to visualize RRDs. The detail description of the RRD logs is available on Tstat documentation(http://tstat.polito.it/HOWTO.shtml#
RRD). RRD are available via the (http://monroe-repository.polito.it:8080/) and the Graphite GUI provides some tool to
present RRD logs and save the interested plots. Fig. 12 shows the bit rate of the ICMP packet for the node #38 on interface op0 over the
last 24 hours. There is possibility to create a dashboard to monitor the experiments and interfaces’ status. Fig. 13 illustrates an example
of saved dashboard to monitor the volume of traffic on one node.

5.6.2

Tstat logs

Tstat generates detailed flow level logs for TCP, UDP, and HTTP flows. These are text file with more than 100 metrics, containing information about the client and server addresses, network and application level metrics, and DNS queries. The description of the metrics
presents (http://tstat.polito.it/measure.shtml#LOG). In MONROE, the Tstat configure to generate 4 different logs as following:
Logs are available in two ways,

38 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 12: Graphite GUI of the Tstat RRD logs.
Table 5: Tstat log types.

TYPE

DESCRIPTION

log_tcp_complete
log_tcp_nocomplete
log_udp_complete
log_http_complete

Every TCP connection that has been tracked
All the connections for which the three way handshake is not properly seen
Every tracked UDP flow pair
Information from every HTTP request and response

1. Real time access on the node, logs for the last three generated are shared with MONROE experimenters on the "/monroe/tstat",
it helps the MONROE users to use passive traces collected by Tstat during their experiment. The three logs can cover at most the
last three hours.
2. On demand, all logs are imported into MONROE database for all node. The schema of the tables are available on github (https:
//github.com/MONROE-PROJECT/Database/blob/master/db_schema.cql). Three columns, (NodeId,Iccid,DataId) added
to each table bring the possibility to join with metadata and collected data.
It is recommended to check the description of the logs on (http://tstat.polito.it/measure.shtml#LOG). Tables 6, 7 and 8
present the table describing of some interesting metrics in tcp and http logs.

5.7

Access to user-owned development nodes

This section refers to development nodes owned by external users under the dispositions of their specific MONROE agreement. Two
options are possible for the management of those nodes:
1. The nodes join the pool of MONROE nodes. Experiments are scheduled through the MONROE scheduler and users, including
the node owners, do not have direct SSH access to them. The metadata produced by these nodes will join the rest of the MONROE
databases.
2. The nodes are considered “development nodes” for private use of their owners. In that case, they will not join the MONROE
platform and will not be accessible through the MONROE scheduler, neither for their owners nor for other users. The nodes will
be marked as “storage” or “development.” Thus, users (again, only their owners) must log locally into the nodes to manually

39 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 6: Core TCP Set.

C2S

S2C

Short description

Unit

Long description

1
2
3
4
5
6
7
8
9
10
11
12
13
14
29
30
31
32
33
34
35
36
37
38
39
40
41
42

15
16
17
18
19
20
21
22
23
24
25
26
27
28

Client/Server IP addr
Client/Server TCP port
packets
RST sent
ACK sent
PURE ACK sent
unique bytes
data pkts
data bytes
rexmit pkts
rexmit bytes
out seq pkts
SYN count
FIN count
First time abs
Last time abs
Completion time
C first payload
S first payload
C last payload
S last payload
C first ack
S first ack
C internal
S internal
C anonymized
S anonymized
Connection type

–
–
–
0/1
–
–
B
–
B
–
B
–
–
–
ms
ms
ms
ms
ms
ms
ms
ms
ms
0/1
0/1
0/1
0/1
–

P2P type
HTTP type

–
–

IP addresses of the client/server
TCP port addresses for the client/server
total number of packets observed from the client/server
0 = no RST segment has been sent by the client/server
number of segments with the ACK field set to 1
number of segments with ACK field set to 1 and no data
number of bytes sent in the payload
number of segments with payload
number of bytes transmitted in the payload, including retransmissions
number of retransmitted segments
number of retransmitted bytes
number of segments observed out of sequence
number of SYN segments observed (including rtx)
number of FIN segments observed (including rtx)
Flow first packet absolute time (epoch)
Flow last segment absolute time (epoch)
Flow duration since first packet to last packet
Client first segment with payload since the first flow segment
Server first segment with payload since the first flow segment
Client last segment with payload since the first flow segment
Server last segment with payload since the first flow segment
Client first ACK segment (without SYN) since the first flow segment
Server first ACK segment (without SYN) since the first flow segment
1 = client has internal IP, 0 = client has external IP
1 = server has internal IP, 0 = server has external IP
1 = client IP is CryptoPAn anonymized
1 = server IP is CryptoPAn anonymized
Bitmap stating the connection type as identified by TCPL7 inspection engine (see
protocol.h)
Type of P2P protocol, as identified by the IPP2P engine (see ipp2p_tstat.h)
For HTTP flows, the identified Web2.0 content (see the http_content enum in
struct.h)

43
44

Table 7: TCP End to End Set.

C2S

S2C

Short description

Unit

Long description

45

52

Average rtt

ms

46
47
48
49
50
51

53
54
55
56
57
58

rtt min
rtt max
Stdev rtt
rtt count
ttl_min
ttl_max

ms
ms
ms
–
–
–

Average RTT computed measuring the time elapsed between the data segment and the
corresponding ACK
Minimum RTT observed during connection lifetime
Maximum RTT observed during connection lifetime
Standard deviation of the RTT
Number of valid RTT observation
Minimum Time To Live
Maximum Time To Live

40 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 13: An example of dashboard on Tstat RRD GUI.
schedule their containers using Docker commands. The nodes will not run the base experiments; no metadata, or any other
information produced by them will join the MONROE databases.
In essence, “managed” nodes are part of the testbed and work as any other ones, whereas “development” nodes are for private use
of their owners. The following paragraphs provide relevant information for the use of development nodes.

5.7.1

Accessing user-owned development nodes

Development nodes can be accessed either through the management interface (the black wire connected to eth2) via SSH, or directly
via the serial console (DB9 connector, using a null-modem cable). The necessary passwords will be provided on request through a
secure channel. Table 9 explains the uses of each available user.
Do not distribute passwords or keys to unauthorized personnel. Do not send passwords or keys over insecure channels. Use of
the administrator user ’monroeSA’ is allowed only for development on local nodes, unless granted permission to perform a specific
task requiring this user. Creation of user accounts on nodes is forbidden. Modifying user accounts on nodes is forbidden. Modifying
authorized_keys on nodes is forbidden. Be VERY careful if you change any firewall settings, and only do this on development
nodes. Be smart.
Local access, which allows password authentication, can be achieved through the serial port and the management interface.
The APU’s third ethernet port (eth2), nearest the USB ports, has a default IP address, 172.16.254.1/24 for the Head and 172.16.254.2/24
for the Tail. The nodes can be accessed by setting up a static IP address in the 172.16.254.0/24 network span (e.g., 172.16.254.20) on the
developer side of the link and establishing an SSH connection.
The DB-9 serial port (console) allows direct terminal access. The boot process and grub menu are visible and interactive through
this connection; some kernel messages will be printed as well while connected. Connecting to the console port requires a null modem
cable. In the following examples, /dev/ttyS0 has to be substituted with the device path for the developer’s cable. A typical case for
USB-to-serial adapters is /dev/ttyUSB0:
minicom -D /dev/ttyS0

6

---or---

screen /dev/ttyS0 115200

Monitoring node status

The state of the nodes can be checked under the tab “Resources.” Figure 14 shows an example of the supplied information. Experimenters can use the operator codes and names to manually pick nodes with concrete operators.

41 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 8: Core HTTP Set.

C2S

S2C

Short description

Unit

Long description

1
2
3
4
5
6
7
8
9
10
11
12
13

1
2
3
4
5

Client IP addr
Client TCP port
Server IP addr
Server TCP port
Segment time abs
Request method
Hostname
FQDN
URL Path
Referer
User agent
Cookie
Do Not Track
Response string
Response code
Content len
Content type
Server
Range
Location
Set Cookie

–
–
–
–
s
–
–
–
–
–
–
–
–
–
–
B
–
–
–
–
–

IP addresses of the client (sending the request/receiving the response)
TCP port addresses for the client
IP addresses of the server (receiving the request/sending the response)
TCP port addresses for the server
Absolute time [s] (epoch) of the request/response
Request method (GET/POST/HEAD) [*]
Value fo the “Host:” HTTP request field
DN-Hunter cached DNS name [ˆ]
URL request path
Value of the “Referer:” HTTP request field
Value of the “User-Agent:” HTTP request field
Value of the “Cookie:” HTTP request field
Value of the “DNT:” HTTP request field
Response identifier (always “HTTP”) [*]
HTTP response code (2xx/3xx/4xx/5xx)
Value of the “Content-Length:” HTTP response field
Value of the “Content-Type:” HTTP response field
Value of the “Server:” HTTP response field
Value of the “Content-Range:” HTTP response field for partial content (Code 206)
Value of the “Location:” HTTP response field for redirected content (Code 302)
Value of the “Set-Cookie:” HTTP response field

6
7
8
9
10
11
12
13

Table 9: Node users

USER

PASSWORD

SUDO

USES

monroe
monroeSA

[redacted]
[redacted]

reboot
yes

maintenance, troubleshooting
administration, development

42 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Figure 14: Status of the MONROE nodes. The screen capture shows all types of nodes in Spain. The green approval sign (“thumbs-up”) close to the node IDs indicates that this node is capable of executing experiments.
Clicking on the “Location” link for a node opens a Google Maps page showing the location of the node. Finally, the bottom part of the screen shows a “map” of nodes that allows users and MONROE administrators
to quickly identify available (and problematic) nodes in the platform.

Column “Location” opens a Google Maps window with the last known position of the node, when available. Similarly, column
“Graphs” opens the visualization page for the selected node. There, experimenters can see the last known RTT and RSSI measures for
the node, its current location and state.

7

MONROE templates, examples and default experiments

This section details the template for building MONROE experiments, the experiments that run as part of the default MONROE platform
and several additional examples that can be directly used or that can serve as the basis for new ones. The source code for the examples
is publicly available at https://github.com/MONROE-PROJECT/Experiments.

7.1

Example template

This experiment template provides an extensive example to show the capabilities of the MONROE platform. The experiment will download a url (file) over http using curl from a configurable operator while at the same time recording the GPS positions of the node. If
the operator is not available at the time of execution, the experiment will fail.

43 of 67

Project no. 644399

User manual
MONROE Platform User manual

7.1.1

Public
Rev. 1.0/ March 14, 2019

Usage

The configuration values can be supplied as a JSON string in the “Additional options” field of the web user interface. This allows to
specify a different set of parameters for each execution of the experiment.
The values of the configuration parameters can be read by the experiment from the /monroe/config file. The following text
shows a configuration file with per-execution (“additional options” field) options:4
{
"stop": 1486653420,
"start": 1486653120,
"traffic": 1048576,
"script": "mikepeon/mike-depurar",
"shared": 0,
"storage": 134217728,
"resultsQuota": 0,
"guid": "sha256:3796f833f55c8dbca7e9845ea06120ccebec85c2770c0de2deb57509300efa44.165695.48.1",
"option1": "value1",
"option2": "value2",
"nodeid": "48"
}
The default configuration values are as follows:
{
# The following values are specific to the monroe platform
"guid": "no.guid.in.config.file",
# Created by the scheduler
"nodeid": "no.nodeid.in.config.file",
# Created by the scheduler
"storage": 104857600,
# Created by the scheduler
"traffic": 104857600,
# Created by the scheduler
"script": "jonakarl/experiment-template",
# Created by the scheduler
"zmqport": "tcp://172.17.0.1:5556",
"modem_metadata_topic": "MONROE.META.DEVICE.MODEM",
"gps_metadata_topic": "MONROE.META.DEVICE.GPS",
# "dataversion": 1,
# Version of the experiment
# "dataid": "MONROE.EXP.JONAKARL.TEMPLATE",
# Name of the experiement
"meta_grace": 120,
# Grace period to wait for interface metadata
"exp_grace": 120,
# Grace period before killing experiment
"meta_interval_check": 5,
# Interval to check if interface is up
"verbosity": 2,
# 0="Mute", 1=error, 2=information, 3=verbose
"resultdir": "/monroe/results/",
# These values are specic for this experiment
"operator": "Telenor SE",
"url": "http://193.10.227.25/test/1000M.zip",
"size": 3*1024 - 1,
# The maximum size in Kbytes to download
"time": 3600
# The maximum time in seconds for a download
}
The download will abort when either size OR time OR actual size of the “url” is downloaded. All debug/error information will be
printed on stdout, depending on the “verbosity” variable.

7.1.2

Requirements

The following directories and files must exist and have read and write permissions for the user/process running the container:
• /monroe/config, supplyed by the scheduler in the nodes.
• “resultdir,” according to the values supplied in the configuration string or the default ones (Section 7.1.1).
4 Entries in the /monroe/config file may appear in different order.

44 of 67

Project no. 644399

User manual
MONROE Platform User manual

7.1.3

Public
Rev. 1.0/ March 14, 2019

Output

The experiment will execute a statement similar to running curl with the following command line:
curl -o /dev/null --raw --silent --write-out "{ remote: %{remote_ip}:%{remote_port},
size: %{size_download}, speed: %{speed_download}, time: %{time_total},
time_download: %{time_starttransfer} }" --interface eth0 --max-time 100 --range 0-100
http://193.10.227.25/test/1000M.zip
The experiment will produce a single-line JSON object similar to this (pretty printed to improve readability):
{
"Bytes": 30720000,
"DataId": "313.123213.123123.123123",
"DataVersion": 1,
"DownloadTime": 2.716,
"GPSPositions": [
{
"Altitude": 225.0,
"DataId": "MONROE.META.DEVICE.GPS",
"DataVersion": 1,
"Latitude": 59.404697,
"Longitude": 13.581558,
"NMEA": "$GPGGA,094832.0,5924.281896,N,01334.893500,E,1,05,1.6,225.0,M,35.0,M,,*5D\r\n",
"SatelliteCount": 5,
"SequenceNumber": 14,
"Timestamp": 1465551728
},
{
"DataId": "MONROE.META.DEVICE.GPS",
"DataVersion": 1,
"Latitude": 59.404697,
"Longitude": 13.581558,
"NMEA": "$GPRMC,094832.0,A,5924.281896,N,01334.893500,E,0.0,,100616,0.0,E,A*2B\r\n",
"SequenceNumber": 15,
"Timestamp": 1465551728
}
],
"Guid": "sha256:15979bc2e2449b0011826c2bb8668df980da88221af3fc7916cb2eba4f2296c1.0.45.15",
"Host": "193.10.227.25",
"Iccid": "89460850007006922138",
"InterfaceName": "usb0",
"NodeId": "45",
"Operator": "Telenor SE",
"Port": "80",
"SequenceNumber": 1,
"SetupTime": 0.004,
"Speed": 11295189.0,
"TimeStamp": 1465551458.099917,
"TotalTime": 2.72
}

7.1.4

Overview of the code structure

The experiment consists of one main process and two sub processes, where one process listens to modem and gps information, and
the other executes the experiment. The main process supervises the execution of its two children.

45 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Information sharing between processes. Information is shared between processes via two thread-safe data structures (i.e.,
a Python “Manager” object). Regarding modem information, the latest metadata update (for the specified operator) is stored in a
dictionary. The GPS information is continuously appended to a list as it is received.

The metadata sub-process. This process listens to GPS and modem messages sent on the ZeroMQ bus and updates the shared
data structures.

The experiment sub-process. This process reads entries from the shared data structures, runs the experiment and saves its
result when finished.

7.2

Docker miscellaneous usage notes

• List running containers:
docker ps
• Debug shell:
docker run -i -t --entrypoint bash --net=host template
• Normal execution with output to stdout:
docker run -i -t --net=host template
• Attach to a running container (with shell):
docker exec -i -t [container runtime name] bash
• Get container logs (stderr and stdout):
docker logs [container runtime name]

7.3

Experiment: ping

This background experiment runs continuously an RTT estimate on each MBB operator on the node (one independent experiment
is run per interface). The experiments measure IP RTT by continuously sending ping packets to a configurable server (by default
8.8.8.8, Google’s public DNS server). The experiment will send one “Echo Request” (ICMP type 8) packet per second over the
specified interface until aborted. RTT is measured as the time between the echo request is sent and the echo reply (ICMP type 0) is
received from the server. The experiment runs on all interfaces in parallel.

7.3.1

Usage

The experiment is designed to run as a Docker container and will not attempt to do any active network configuration. If the specified
interface does not exist (i.e., is not up) when the experiment starts, it will immediately exit.
The default parameter values are:
{
"guid": "no.guid.in.config.file",
# Created by the scheduler
"zmqport": "tcp://172.17.0.1:5556",
"nodeid": "fake.nodeid",
"modem_metadata_topic": "MONROE.META.DEVICE.MODEM",
"server": "8.8.8.8",
# ping target
"interval": 1000,
# time in ms between successive packets
"dataversion": 2,
"dataid": "MONROE.EXP.PING",
"meta_grace": 120,
# Grace period to wait for interface metadata
"ifup_interval_check": 5,
# Interval to check if interface is up
"export_interval": 5.0,
"verbosity": 2,
# 0="Mute", 1=error, 2=Information, 3=verbose
"resultdir": "/monroe/results/",
"modeminterfacename": "InternalInterface",
"interfacename": "eth0",
# Interface to run the experiment on
"interfaces_without_metadata": ["eth0", "wlan0"] # Manual metadata on these interfaces
}

46 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

All debug/error information will be printed on stdout depending on the value of the “verbosity” parameter.

7.3.2

Requirements

The following directories and files must exist and have read and write permissions for the user/process running the container:
• /monroe/config, supplyed by the scheduler in the nodes.
• “resultdir,” according to the values supplied in the configuration string or the default ones (Section 7.1.1).

7.3.3

Output

The experiment will execute a statement similar to running fping with the following command line:
fping -I eth0 -D -c 1 -p 1000 -l 8.8.8.8
The experiment will produce one of the two following single-line JSON objects, depending on whether it got a reply form the server
or not. If a reply was received:
{
"Guid": "313.123213.123123.123123", # exp_config[’guid’]
"Timestamp": 23123.1212, # time.time()
"Iccid": 2332323, # meta_info["ICCID"]
"Operator": "Telia", # meta_info["Operator"]
"NodeId" : "9", # exp_config[’nodeid’]
"DataId": "MONROE.EXP.PING",
"DataVersion": 2,
"SequenceNumber": 70,
"Rtt": 6.47,
"Bytes": 84,
"Host": "8.8.8.8",
}
If the reply was not received (Bytes and RRR values are not present):
{
"Guid": "313.123213.123123.123123", # exp_config[’guid’]
"Timestamp": 23123.1212, # time.time()
"Iccid": 2332323, # meta_info["ICCID"]
"Operator": "Telia", # meta_info["Operator"]
"NodeId" : "9", # exp_config[’nodeid’]
"DataId": "MONROE.EXP.PING",
"DataVersion": 2,
"SequenceNumber": 71,
"Host": "8.8.8.8",
}

7.4

Experiment: http_download

This is a periodically scheduled experiment that monitors the download speed of each MBB operator on the node. The experiment
will, over each MBB operator in sequence, download the specified url (file) with curl (http), presenting one result per interface. The
MONROE experiment template described in Section 7.1 corresponds to this experiment, therefore, it is not further detailed here.

7.5

Experiment: Tstat & mPlane

The mPlane protocol provides control and data interchange for passive and active network measurement tasks. It is built around a
simple workflow that can interact with different frameworks to provide the results of the measurements. This package includes an
mPlane proxy and generic configuration files for Tstat.
mPlane captures traffic flow on all interfaces with the Tstat (http://tstat.polito.it/) probe. The mPlane container is always running as one of the default experiments on all MONROE nodes. The Tstat passive traces are stored locally on the node and

47 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

are accessible by the experimenters. A detailed description and the source code are available on github (https://github.com/
MONROE-PROJECT/mPlane).
Tstat RRD logs and the compressed log are stored in the node at /experiments/monroe/mplane. Tstat logs are transfered
to the MONROE server and imported into MONROE’s (Cassandra) database. The structure of the database tables is available on github
(https://github.com/MONROE-PROJECT/Database/blob/master/db_schema.cql).
During experiment execution, the last three Tstat logs are shared with the experiment at /monroe/tstat. Therefore, MONROE
users can access the passive traces collected by Tstat during their experiments.
The data collected for a subset of the most relevant metrics for the HTTP experiments are visualized by the MONROE visualization
tool. And example of the metrics contained in the Tstat logs can be seen here: http://213.182.68.136:8080/#/experiment/
tstat.

7.5.1

Requirements

The script must have access to /nodeid and run get_nodeid.

7.5.2

Usage

Create your docker image normally and execute the container with the following command line:
docker run -i -t --net=host -d -v /mplane:/monroe/results -v /tstat:/monroe/tstat
-v /etc/nodeid:/nodeid:ro monroe/mplane

7.6

MONROE example: helloworld

This experiment provides an easy example for using the configuration options from the scheduler, listen to and record the metadata
stream (e.g., GPS and operator information), and show the experiment log functionality on a MONROE node. The experiment listens
to the metadata stream and records the nr_of_messages first messages. The metadata messages are saved in JSON format with
a custom field (“Hello”) in the output directory. Additionally, the experiment prints out some debugging messages to show how these
messages are logged and later retrieved via the web user interface.

7.6.1

Usage

The experiment is configured with a JSON string introduced via the “Additional options” field in the web user interface. The configurable
parameters and their default values are:
{
"zmqport": "tcp://172.17.0.1:5556",
"nodeid": "fake.nodeid",
"metadata_topic": "MONROE.META",
"verbosity": 2,
"resultdir": "/monroe/results/",
"nr_of_messages": 3
}

7.6.2

# Needs to be overriden
# 0 = "Mute", 1=error, 2=Information, 3=verbose

Requirements

The following directories and files must exist and have read and write permissions for the user/process running the container:
• /monroe/config, supplyed by the scheduler in the nodes.
• “resultdir,” according to the values supplied in the configuration string or the default ones (Section 7.6.1).

7.6.3

Output

The experiment will produce a single-line JSON object similar to the following ones, depending on the metadata received (“pretty
printed” here to improve readability):

48 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

{
"DataId": "MONROE.META.NODE.SENSOR",
"DataVersion": 1,
"SequenceNumber": 58602,
"Timestamp": 1465888420,
"NodeId": "9",
"Hello": "World"
}
The log file will contain records similar to these ones:
[2017-02-07 09:53:27.190338] Hello: Default config {
"metadata_topic": "MONROE.META",
"nodeid": "fake.nodeid",
"nr_of_messages": 3,
"resultdir": "/monroe/results/",
"verbosity": 2,
"zmqport": "tcp://172.17.0.1:5556"
}
[2017-02-07 09:53:27.20000] Hello: Start recording messages with configuration {
"metadata_topic": "MONROE.META",
"nodeid": "fake.nodeid",
"nr_of_messages": 3,
"resultdir": "/monroe/results/",
"verbosity": 2,
"zmqport": "tcp://172.17.0.1:5556"
}
[[2017-02-07 09:53:27.30000] Received message 1 with topic : MONROE.META.NODE.SENSOR
{
"DataId": "MONROE.META.NODE.SENSOR",
"DataVersion": 1,
"SequenceNumber": 58602,
"Timestamp": 1465888420,
"NodeId": "9",
"Hello": "World"
}
. # And so on for each metadata message received until the configured value of metadata messages
.
.
[2017-02-07 09:53:27.40000] Hello : Finished the experiment

7.7

MONROE example: paris-traceroute

This example showcases how to use the MONROE-modified version of paris-traceroute inside a container. The binary of this tool is
included in the base image of MONROE.
The original version of paris-traceroute has no option to choose which interface should be used. In this version, flags to set the
interface and source IP of the transmitted packets have been added. Setting the interface is obligatory; if it is not set, the program will
crash (by design), since if the interface were chosen automatically, it would probably not be what the experimenter intended to use.
The source IP flag is optional. Just setting the IP flag to the IP of an interface without setting the interface flag will not work either. This
is done on purpose as well, as it might be possible for multiple interfaces to have the same IP within the MONROE network namespace.
If the IP flag is not set, the source IP is set to the IP of the chosen interface.

7.7.1

Usage (inside a MONROE container)

The parameters of this experiment are provided as “Additional options” in the scheduling web interface. The following JSON string is
an example of the additional options that can be passed to this container:

49 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

"interfaces": ["op1", "op2"], "targets": ["8.8.8.8", "www.uc3m.es"],
"traceAlgos": ["exh"], "protocol": "udp"
Flags:
-C
-O

--nodeIPArgument
--nodeInterfaceArgument

Source IP
Source interface (mandatory)

The paris-traceroute binary can be executed (as any normal Linux command) either without specifying a traceroute algorithm to
perform a “simple” traceroute (similar to the output of the ordinary traceroute command), or with the flags -n -a exh, to perform
an exhaustive traceroute. Exhaustive traceroutes provide more detailed and accurate paths between the host (MONROE node) and the
target server that are able to detect, among others, the presence of load balancers, which create multiple paths between host and target.

7.7.2

Output

The experiment output is a text file:
root@b59e69a56297:/# paris-traceroute -O op2 -C 192.168.1.127 8.8.8.8
traceroute [(192.168.1.127:33456) -> (8.8.8.8:33457)], protocol udp, algo hopbyhop, duration 18 s
1 192.168.1.1 (192.168.1.1) 2.946 ms
0.553 ms
0.559 ms
2 * * *
3 10.133.17.29 (10.133.17.29) 83.259 ms
136.577 ms
82.050 ms
4 10.133.17.14 (10.133.17.14) 78.783 ms
131.510 ms
79.231 ms
5 10.133.17.236 (10.133.17.236) 84.243 ms
133.024 ms
79.785 ms
6 10.133.17.3 (10.133.17.3) 81.543 ms
139.381 ms
100.263 ms
7 83.224.40.186 (83.224.40.186) 89.319 ms
188.926 ms
179.963 ms
MPLS Label 24703 TTL=254
8 83.224.40.185 (83.224.40.185) 82.710 ms
172.438 ms
147.020 ms
9 85.205.14.105 (85.205.14.105) 85.179 ms
137.514 ms
125.869 ms
10 72.14.223.169 (72.14.223.169) 85.609 ms
137.363 ms
118.063 ms
11 216.239.47.128 (216.239.47.128) 79.567 ms
146.356 ms
145.285 ms
12 209.85.243.33 (209.85.243.33) 129.615 ms
198.938 ms
269.407 ms
MPLS Label 568892 TTL=1
13 64.233.174.143 (64.233.174.143) 108.599 ms
185.810 ms
246.661 ms
MPLS Label 692130 TTL=1
14 108.170.234.47 (108.170.234.47) 111.645 ms
825.615 ms
1424.942 ms
15 * * *
16 google-public-dns-a.google.com (8.8.8.8) 103.087 ms !T2
166.279 ms !T2
224.649 ms !T2

7.7.3

Additional remarks

Paris-traceroute instances should be run sequentially and preferably when the node is generating little traffic in general because it uses
raw packet capture to detect the replies from intermediate nodes and background traffic might interfere with this process.

7.8

MONROE example: headlessbrowsing

This experiment evaluates the performance of different HTTP protocols (HTTP1.1, HTTP1.1/TLS, HTTP2) using the headless Firefox
browser. It uses the Selenium browser-automation framework, which enables execution of web-browsing automation tests in different
browsers such as Firefox and Chrome. The Selenium web-driver is used for Firefox. For a given url, HTTP protocol and source network
interface, Selenium launches the native Firefox browser to visit that url.

7.8.1

Output

This experiment generates an HTTP ARchive (HAR) file during the download of a target url that helps to find afterwards the impact of
different web-page features on its overall Page Load Time (PLT).
The experiment generates a single JSON file such as:

50 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

{
"DataId":"MONROE.EXP.FIREFOX.HEADLESS.BROWSING",
"ping_min":" 55.6",
"ping_max":"56.8",
"NumObjects":6,
"InterfaceName":"usb2",
"Web load time":196,
"PageSize":35641,
"DataVersion":1,
"Timestamp":1481536829.0814,
"NWMCCMNC":22210,
"Objects":[
{
"objectSize":1951,
"mimeType":"image/png",
"startedDateTime":"2016-12-12T10:00:21.293+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/img/Wikipedia_wordmark.png",
"timings":{"receive":1, "send":0, "connect":1, "dns":0, "blocked":0, "wait":60},
"time":62
},
{
"objectSize":13196,
"mimeType":"image/png",
"startedDateTime":"2016-12-12T10:00:21.294+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/img/Wikipedia-logo-v2.png",
"timings":{"receive":52, "send":3, "connect":59, "dns":2, "blocked":0, "wait":53 },
"time":169
},
{
"objectSize":9425,
"mimeType":"application/javascript",
"startedDateTime":"2016-12-12T10:00:21.295+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/js/index-abc278face.js",
"timings":{"receive":3, "send":0, "connect":120, "dns":0, "blocked":0, "wait":65},
"time":188
},
{
"objectSize":1164,
"mimeType":"application/javascript",
"startedDateTime":"2016-12-12T10:00:21.296+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/js/gt-ie9-c84bf66d33.js",
"timings":{"receive":0, "send":1, "connect":64, "dns":2, "blocked":0, "wait":70 },
"time":137
},
{
"objectSize":1590,
"mimeType":"image/png",
"startedDateTime":"2016-12-12T10:00:21.381+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/img/sprite-icons.png?
27378e2bb51199321b32dd1ac3f5cd755adc21a5",
"timings":{"receive":1, "send":0, "connect":1, "dns":0, "blocked":0, "wait":49 },
"time":51
},
{
"objectSize":8315,
"mimeType":"image/png",

51 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

"startedDateTime":"2016-12-12T10:00:21.425+00:00",
"url":"https://www.wikipedia.org/portal/wikipedia.org/assets/img/sprite-project-logos.png?
dea6426c061216dfcba1d2d57d33f4ee315df1c2",
"timings":{"receive":2, "send":0, "connect":8, "dns":0, "blocked":0, "wait":54 },
"time":64
} ],
"IPAddress":"2.43.181.254",
"IMSIMCCMNC":22210,
"tracedRoutes":["192.168.96.1", "193.10.227.25", "xx.xx.xx.xx" ..... "192.168.96.1" ],
"InternalInterface":"op0",
"NodeId":"41",
"ping_exp":1,
"Protocol":"HTTP1.1",
"SequenceNumber":1,
"url":"www.wikipedia.org",
"ping_avg":"56.2",
"InternalIPAddress":"192.168.96.123",
"Operator":"voda IT",
"Iccid":"8939104160000392116"
}

7.9

MONROE example: pReplay

The pReplay experiment replays the dependency graph of a web site.
The traversal begins with the first activity: Loading the root HTML. After building the dependency graph, it acts for each task whose
dependencies have already been met. For network tasks, it makes a request for the corresponding url; correspondingly, for computation
activities, it waits for the amount of time mentioned in the graph. Once a particular activity is finished, pReplay checks if any activities
depending on that one have already met all of their dependencies and must thus be triggered. pReplay walks through the dependency
graph until all activities in the graph have been visited.

7.9.1

Usage

Execute pReplay on a command line inside a container as with any other Linux command:
./pReplay interface_name server testfile [http|https|http2] [max-connections] [cookie-size]
Parameters:
•
•
•
•

interface_name: Source interface for outgoing traffic.
server: DNS name or IP address.
testfile: Relative path to test file in JSON format.
protocol:
– http: http 1.1
– https: http 1.1 with SSL
– http2: http 2

• max-connections: Maximum amount of concurrent connections.
• cookie-size: Size of cookie — works with http1 only.

7.10

MONROE example: astream

AStream is a Python based emulated video player to evaluate the performance of the DASH bitrate adaptation algorithms. The supported rate adaptation algorithms are:
• Basic adaptation.
• Segment Aware Rate Adaptation (SARA) [2].
• Buffer-Based Rate Adaptation (Netflix) [1].

52 of 67

Project no. 644399

User manual
MONROE Platform User manual

7.10.1

Public
Rev. 1.0/ March 14, 2019

Usage

The experimenter can choose the rate adaptation algorithm passing a JSON string to the scheduler through the user interface (e.g.,
"playback":"NETFLIX"). The default is the basic adaptation scheme. Additionally, the user can specify the target MPD file to play
(e.g., "mpd_file":"http://128.39.37.161:8080/BigBuckBunny_4s.mpd") and the number of segments to retrieve
(e.g., "segment_limit":10).

7.10.2

Output

The astream container outputs two log files:
1. Buffer logs: Epoch time, current playback time, current buffer size (in segments), current playback state.
2. Playback logs: Epoch time, playback time, segment number, segment size, playback bitrate, segment duration, weighted harmonic mean average download rate.

7.11

MONROE example: udpbwestimator

Udpbwestimator is an experiment setup to estimate available bandwidth for a particular network interface. It consists of two applications, a receiver and a traffic generator (server). The receiver initiates connections and requests the server for traffic. Then, every
second, the server sends a burst of UDP packets back to back to the receiver, which follows the packet arrival times and estimates the
available bandwidth.

7.11.1

Usage

The receiver accepts the following command line parameters:
-c : Number of back-to-back packets to be sent in each second.
-b : Number of bursts to be sent.
-l : Payload length in bytes.
-s : Source IP to bind to.
-o : Source port.
-d : Destination IP.
-p : Destination port.
-w : Optional, filename for writing the packet arrival times.

7.11.2

Output

The experiment will produce a single-line JSON object similar to the following:
{
"CID" : 33346602,
"DataId" : "MONROE.EXP.UDPBWESTIMATOR",
"DataVersion" : 1,
"DeviceMode" : 5,
"DeviceState" : 3,
"Guid" : "sha256:872af8c8b8f1635be6936a111b5fa838071e6f42cb317e9db1d9bb0c7db31425.93321.204.1",
"IMEI" : "864154023639966",
"IMSI" : "240016025247086",
"IMSIMCCMNC" : 24001,
"IPAddress" : "78.79.63.124",
"Iccid" : "89460120151010468086",
"InterfaceName" : "usb0",
"InternalIPAddress" : "192.168.68.118",
"InternalInterface" : "op1",
"LAC" : 2806,
"NWMCCMNC" : 24202,
"NodeId" : "204",
"Operator" : "NetCom",
"RSRP" : -72,

53 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

"RSRQ" : -7,
"RSSI" : -49,
"SequenceNumber" : 1,
"Timestamp" : 1479312368.633218,
"bw" : "48.41 38.98 36.44 50.00 30.20 45.21 47.02 37.89 44.37 28.90 25.91 38.57 48.74
39.94 43.37 37.94 43.81 39.60 52.00 47.55 48.20 34.85 41.44 47.60 57.26 46.11
45.66 52.04 37.43 49.67 33.56 50.35 41.11 51.63 45.33 104.01 45.73 49.95 50.37
38.57 29.45 50.95 54.95 45.42 47.13 34.30 46.10 103.68 79.75 45.72 52.03 30.38
50.21 36.96 71.51 54.66 39.26 44.12 45.18 39.93"
}

7.12

MONROE example: traceroute_background_experiment

Performs traceroute periodically to various targets. This experiment is meant to be run in the background and can be run in parallel
with experiments of other users. It uses the default traceroute binary distributed by the Debian repositories.
Each traceroute produces a text file that is parsed by outputParser.py to generate the JSON output of this experiment. The
JSON file is then imported into the MONROE database.

7.12.1

Usage

To reduce experiment duration, the traceroutes can be run in parallel. The number of parallel traceroute instances is dictated by the
maxNumberOfTotalTracerouteInstances parameter. It is possible to parallelize on a per-interface basis (i.e., maxNumberOfTotalTracer
per interface) or per the whole experiment (i.e., maxNumberOfTotalTracerouteInstances total in the experiment instance
spread among all the interfaces). This behavior is controlled by the executionMode parameter. The available options are: serially,
serialPerInterface and parallel.
Additionally, a flag can be provided to choose the protocol of the probes: default, udp, tcp and icmp.
The parameters of this experiment are provided as “Additional options” in the web user interface. An example JSON string that can
be used with this container as additional options is:
"interfaces": ["op0", "op1", "op2"], "targets": ["www.ntua.gr", "www.uc3m.es", "Google.com",
"Facebook.com", "Youtube.com", "Baidu.com", "Yahoo.com", "Amazon.com", "Wikipedia.org",
"audio-ec.spotify.com", "mme.whatsapp.net", "sync.liverail.com", "ds.serving-sys.com",
"instagramstatic-a.akamaihd.net"], "maxNumberOfTotalTracerouteInstances": 5,
"executionMode": "parallel"

7.13

Other containers in the repositories

Our public repositories contain the source code for other Docker containers that perform varied tasks in the nodes. Although they are
not intended as examples, users can take a look into them to gain a deeper understanding of the platform configuration.

7.13.1

Container: metadata-subscriber

The subscriber is designed to listen to ZMQ messages send out by the metadata-multicaster. The subscriber attaches to a configurable ZeroMQ socket and listens to all messages that begin with the topic “MONROE.META,” except the ones whose topic ends with
“.UPDATE” (rebroadcasts) and/or begins with “MONROE.META.DEVICE.CONNECTIVITY.” as these are redundant. All messages are
updated with NodeId, but are otherwise saved verbatim as a JSON formatted file suitable for later import in the MONROE databases.

7.13.2

Container: tunnelbox-server

This container acts as an SSH reverese tunnel endpoint that clients can use to directly connect to their experiment containers (on any
MONROE node). The purpose is to provide experimenters an interactive way of accessing an experiment running on a real MONROE
node during development or debugging. The client has to supply its own public SSH key to the experiment container using the web
user interface. The web user interface provides further instructions (SSH command line) to connect to the experiment container using
the provided key.

54 of 67

Project no. 644399

User manual
MONROE Platform User manual

7.13.3

Public
Rev. 1.0/ March 14, 2019

Container: monroe_base

This is the container upon which all user experiments must be built. The container is based on Debian “jessie” with (MONROE) common experiment tools added. For a list of the tools currently installed see the folder monroe_base.docker in our repositories.
This base image has been extended to support layer depending on the kind of experiments the experimenter wants to run. The new
monroe_base tags are as follow:
• monroe/base This is for all experiments except the one defined below.
• monroe/base:web This tag was specifically designed for experiments that use headless browsing such as Chrome or Firefox.
• monroe/base:virt This tag is needed for experiments that need to run in a virtual machine.
Some of the packages moved from monroe_base to monroe_base:web such as:
• moved Firefox and support packages (xvfb, xauth, libgtk-3-0, libasound2, dbus, libdbus-glib-1-2, libgtk2.0-0, libgtk2.0-common,
pyvirtualdisplay, selenium, python-dateutil, geckodriver)
• Added support for Firefox (56.0.1) and support packages Google chrome(64.*)

8

List of known bugs and issues
• In general, Firefox does not render the date-time picker correctly. You will have to either enter the dates and times manually or
use Chrome.
• Container deployment can take several minutes, particularly for nodes without an Ethernet management connection (e.g., mobile nodes in trains or buses). When scheduling an experiment, the user has to take into account the time needed for the deployment. The system will not automatically take care of this at this moment.
• Similarly, the button “Check availability” returns the earliest available slot. However, it does not account for the time needed to
deploy the container. The user must manually account for that.
• Checking the option “ASAP” to schedule an experiment as soon as possible may fail due to lack of time to deploy the container.
The system does add some slack in this case, but its length may need some adjustment according to the type of nodes and MBB
characteristics.

A

List of packages installed in monroe/base
Table 10: List of packages installed in monroe/base as of 2017-02-27.

Name

Version

Architecture

Description

acl
adduser
adwaita-icon-theme
apt
base-files
base-passwd
bash
bsdutils
bzip2
ca-certificates
ca-certificates-java
coreutils
curl
d-itg
dash
dbus
dconf-gsettingsbackend:amd64
dconf-service
debconf
debconf-i18n
debian-archive-keyring
debianutils
default-jre-headless

2.2.52-2
3.113+nmu3
3.14.0-2
1.0.9.8.4
8+deb8u6
3.5.37
4.3-11+b1
1:2.25.2-6
1.0.6-7+b3
20141019+deb8u1
20140324
8.23-4
7.38.0-4+deb8u5
2.8.1-r1023-3
0.5.7-4+b1
1.8.20-0+deb8u1
0.22.0-1

amd64
all
all
amd64
amd64
amd64
amd64
amd64
amd64
all
all
amd64
amd64
amd64
amd64
amd64
amd64

Access control list utilities
add and remove users and groups
default icon theme of GNOME
commandline package manager
Debian base system miscellaneous files
Debian base system master password and group files
GNU Bourne Again SHell
basic utilities from 4.4BSD-Lite
high-quality block-sorting file compressor - utilities
Common CA certificates
Common CA certificates (JKS keystore)
GNU core utilities
command line tool for transferring data with URL syntax
Distributed Internet Traffic Generator
POSIX-compliant shell
simple interprocess messaging system (daemon and utilities)
simple configuration storage system - GSettings back-end

0.22.0-1
1.5.56
1.5.56
2014.3
4.4+b1
2:1.7-52

amd64
all
all
all
amd64
amd64

simple configuration storage system - D-Bus service
Debian configuration management system
full internationalization support for debconf
GnuPG archive keys of the Debian archive
Miscellaneous utilities specific to Debian
Standard Java or Java compatible Runtime (headless)

55 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

dh-python

1.20141111-2

all

diffutils
dmsetup
dpkg
dumb-init
e2fslibs:amd64
e2fsprogs
findutils
flent
fontconfig
fontconfig-config
fonts-dejavu-core
fping
gcc-4.8-base:amd64
gcc-4.9-base:amd64
glib-networking:amd64
glib-networking-common
glib-networking-services
gnupg
gpgv
gpsd
gpslogger-oml2
grep
gsettings-desktop-schemas
gzip
hicolor-icon-theme
hostname
httperf-oml2
httping
inetutils-ping
init
init-system-helpers
initscripts
insserv

1:3.3-1+b1
2:1.02.90-2.2+deb8u1
1.17.27
1.2.0
1.42.12-2
1.42.12-2
4.4.2-9+b1
0.15.0-1
2.11.0-6.3+deb8u1
2.11.0-6.3+deb8u1
2.34-1
3.10-2
4.8.4-1
4.9.2-10
2.42.0-2
2.42.0-2
2.42.0-2
1.4.18-7+deb8u3
1.4.18-7+deb8u3
3.11-3
2.11.0-mytestbed2
2.20-4.1
3.14.1-1
1.6-4
0.13-1
3.15
2.11.0-mytestbed2
1.5.8-1
2:1.9.2.39.3a460-3
1.22
1.22
2.88dsf-59
1.14.0-5

amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
all
all
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
all
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64

iperf
iperf-oml2
iperf3
iproute2
iptables
java-common
jq
libacl1:amd64
libapt-pkg4.12:amd64
libasound2:amd64
libasound2-data
libasyncns0:amd64
libatk-bridge2.0-0:amd64
libatk1.0-0:amd64
libatk1.0-data
libatspi2.0-0:amd64
libattr1:amd64
libaudit-common
libaudit1:amd64
libavahi-client3:amd64
libavahi-commondata:amd64
libavahi-common3:amd64
libblas-common
libblas3
libblkid1:amd64
libbluetooth3:amd64

2.0.5+dfsg1-2
2.11.0-mytestbed2
3.0.7-1
3.16.0-2
1.4.21-2+b1
0.52
1.4-2.1
2.2.52-2
1.0.9.8.4
1.0.28-1
1.0.28-1
0.8-5
2.14.0-2
2.14.0-1
2.14.0-1
2.14.0-1
1:2.4.47-2
1:2.4-1
1:2.4-1+b1
0.6.31-5
0.6.31-5

amd64
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
all
amd64
amd64
all
amd64
amd64
amd64

Debian helper tools for packaging Python libraries and applications
File comparison utilities
Linux Kernel Device Mapper userspace library
Debian package management system
wrapper script which proxies signals to a child
ext2/ext3/ext4 file system libraries
ext2/ext3/ext4 file system utilities
utilities for finding files–find, xargs
The FLExible Network Tester
generic font configuration library - support binaries
generic font configuration library - configuration
Vera font family derivate with additional characters
sends ICMP ECHO_REQUEST packets to network hosts
GCC, the GNU Compiler Collection (base package)
GCC, the GNU Compiler Collection (base package)
network-related giomodules for GLib
network-related giomodules for GLib - data files
network-related giomodules for GLib - D-Bus services
GNU privacy guard - a free PGP replacement
GNU privacy guard - signature verification tool
Global Positioning System - daemon
Record and store GPS measurements using OML
GNU grep, egrep and fgrep
GSettings desktop-wide schemas
GNU compression utilities
default fallback theme for FreeDesktop.org icon themes
utility to set/show the host name or domain name
HTTP server performance tester, with OML support
ping-like program for http-requests
ICMP echo tool
System-V-like init utilities - metapackage
helper tools for all init systems
scripts for initializing and shutting down the system
boot sequence organizer using LSB init.d script dependency information
Internet Protocol bandwidth measuring tool
Internet Protocol bandwidth measuring tool, with OML support
Internet Protocol bandwidth measuring tool
networking and traffic control tools
administration tools for packet filtering and NAT
Base of all Java packages
lightweight and flexible command-line JSON processor
Access control list shared library
package management runtime library
shared library for ALSA applications
Configuration files and profiles for ALSA drivers
Asynchronous name service query library
AT-SPI 2 toolkit bridge - shared library
ATK accessibility toolkit
Common files for the ATK accessibility toolkit
Assistive Technology Service Provider Interface - shared library
Extended attribute shared library
Dynamic library for security auditing - common files
Dynamic library for security auditing
Avahi client library
Avahi common data files

0.6.31-5
1.2.20110419-10
1.2.20110419-10
2.25.2-6
5.23-2+b1

amd64
amd64
amd64
amd64
amd64

Avahi common library
Dependency package for all BLAS implementations
Basic Linear Algebra Reference implementations, shared library
block device id library
Library to use the BlueZ Linux Bluetooth stack

56 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

libbsd0:amd64
libbz2-1.0:amd64
libc-bin
libc6:amd64
libcairo-gobject2:amd64
libcairo2:amd64
libcap-ng0:amd64
libcap2:amd64
libcap2-bin
libcgi-fast-perl
libcgi-pm-perl
libcolord2:amd64
libcomerr2:amd64
libconfig-grammar-perl
libcroco3:amd64
libcryptsetup4:amd64
libcups2:amd64
libcurl3:amd64
libdatrie1:amd64
libdb5.3:amd64
libdbi1:amd64
libdbus-1-3:amd64
libdbus-glib-1-2:amd64

0.7.0-2
1.0.6-7+b3
2.19-18+deb8u6
2.19-18+deb8u6
1.14.0-2.1+deb8u1
1.14.0-2.1+deb8u1
0.7.4-2
1:2.24-8
1:2.24-8
1:2.04-1
4.09-1
1.2.1-1+b2
1.42.12-2
1.10-2
0.6.8-3+b1
2:1.6.6-5
1.7.5-11+deb8u1
7.38.0-4+deb8u5
0.2.8-1
5.3.28-9
0.9.0-4
1.8.20-0+deb8u1
0.102-1

amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
all
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64

libdconf1:amd64
libdebconfclient0:amd64

0.22.0-1
0.192

amd64
amd64

libdevmapper1.02.1:amd64
libdigest-hmac-perl
libdrm2:amd64
libedit2:amd64
libencode-locale-perl
libexpat1:amd64
libfcgi-perl
libffi6:amd64
libfile-listing-perl
libflac8:amd64
libfontconfig1:amd64
libfontenc1:amd64
libfreetype6:amd64
libgcc1:amd64
libgcrypt20:amd64
libgdbm3:amd64
libgdk-pixbuf2.0-0:amd64
libgdk-pixbuf2.0-common
libgfortran3:amd64
libgl1-mesa-glx:amd64
libglapi-mesa:amd64
libglib2.0-0:amd64
libgmp10:amd64
libgnutls-deb0-28:amd64
libgpg-error0:amd64

2:1.02.90-2.2+deb8u1
1.03+dfsg-1
2.4.58-2
3.1-20140620-2
1.03-1
2.1.0-6+deb8u3
0.77-1+b1
3.1-2+b2
6.04-1
1.3.0-3
2.11.0-6.3+deb8u1
1:1.1.2-1+b2
2.5.2-3+deb8u1
1:4.9.2-10
1.6.3-2+deb8u2
1.8.3-13.1
2.31.1-2+deb8u5
2.31.1-2+deb8u5
4.9.2-10
10.3.2-1+deb8u1
10.3.2-1+deb8u1
2.42.1-1+b1
2:6.0.0+dfsg-6
3.3.8-6+deb8u3
1.17-3

amd64
all
amd64
amd64
all
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64

libgps21:amd64
libgraphite2-3:amd64
libgssapi-krb5-2:amd64
libgtk-3-0:amd64
libgtk-3-bin
libgtk-3-common
libgtk2.0-0:amd64
libgtk2.0-common
libharfbuzz0b:amd64
libhogweed2:amd64

3.11-3
1.3.6-1 deb8u1
1.12.1+dfsg-19+deb8u2
3.14.5-1+deb8u1
3.14.5-1+deb8u1
3.14.5-1+deb8u1
2.24.25-3+deb8u1
2.24.25-3+deb8u1
0.9.35-2
2.7.1-5+deb8u1

amd64
amd64
amd64
amd64
amd64
all
amd64
all
amd64
amd64

utility functions from BSD systems - shared library
high-quality block-sorting file compressor library - runtime
GNU C Library: Binaries
GNU C Library: Shared libraries
Cairo 2D vector graphics library (GObject library)
Cairo 2D vector graphics library
An alternate POSIX capabilities library
POSIX 1003.1e capabilities (library)
POSIX 1003.1e capabilities (utilities)
CGI subclass for work with FCGI
module for Common Gateway Interface applications
system service to manage device colour profiles – runtime
common error description library
grammar-based user-friendly config parser
Cascading Style Sheet (CSS) parsing and manipulation toolkit
disk encryption support - shared library
Common UNIX Printing System(tm) - Core library
easy-to-use client-side URL transfer library (OpenSSL flavour)
Double-array trie library
Berkeley v5.3 Database Libraries [runtime]
DB Independent Abstraction Layer for C – shared library
simple interprocess messaging system (library)
simple interprocess messaging system (GLib-based shared library)
simple configuration storage system - runtime library
Debian Configuration Management System (C-implementation
library)
Linux Kernel Device Mapper userspace library
module for creating standard message integrity checks
Userspace interface to kernel DRM services – runtime
BSD editline and history libraries
utility to determine the locale encoding
XML parsing C library - runtime library
helper module for FastCGI
Foreign Function Interface library runtime
module to parse directory listings
Free Lossless Audio Codec - runtime C library
generic font configuration library - runtime
X11 font encoding library
FreeType 2 font engine, shared library files
GCC support library
LGPL Crypto library - runtime library
GNU dbm database routines (runtime version)
GDK Pixbuf library
GDK Pixbuf library - data files
Runtime library for GNU Fortran applications
free implementation of the OpenGL API – GLX runtime
free implementation of the GL API – shared library
GLib library of C routines
Multiprecision arithmetic library
GNU TLS library - main runtime library
library for common error values and messages in GnuPG components
Global Positioning System - library
Font rendering engine for Complex Scripts – library
MIT Kerberos runtime libraries - krb5 GSS-API Mechanism
GTK+ graphical user interface library
programs for the GTK+ graphical user interface library
common files for the GTK+ graphical user interface library
GTK+ graphical user interface library
common files for the GTK+ graphical user interface library
OpenType text shaping engine (shared library)
low level cryptographic library (public-key cryptos)

57 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

libhtml-parser-perl
libhtml-tagset-perl
libhtml-tree-perl
libhttp-cookies-perl
libhttp-date-perl
libhttp-message-perl
libhttp-negotiate-perl
libice6:amd64
libicu52:amd64
libidn11:amd64
libio-html-perl
libio-socket-ssl-perl

3.71-1+b3
3.20-2
5.03-1
6.01-1
6.02-1
6.06-1
6.00-2
2:1.0.9-1+b1
52.1-8+deb8u4
1.29-1+deb8u2
1.001-1
2.002-2+deb8u1

amd64
all
all
all
all
all
all
amd64
amd64
amd64
all
all

libiperf0
libjasper1:amd64
libjbig0:amd64
libjpeg62-turbo:amd64
libjs-cropper
libjs-prototype
libjs-scriptaculous
libjson-c2:amd64
libjson-glib-1.0-0:amd64
libjson-glib-1.0-common
libk5crypto3:amd64
libkeyutils1:amd64
libkmod2:amd64
libkrb5-3:amd64
libkrb5support0:amd64
liblcms2-2:amd64
libldap-2.4-2:amd64
liblinear1:amd64
liblocale-gettext-perl
liblua5.2-0:amd64
liblwp-mediatypes-perl
liblwp-protocol-https-perl
liblzma5:amd64
libmount1:amd64
libmpdec2:amd64
libncurses5:amd64
libncursesw5:amd64
libnet-http-perl
libnet-ssleay-perl
libnettle4:amd64

3.0.7-1
1.900.1-debian1-2.4+deb8u1
2.1-3.1
1:1.3.1-12
1.2.2-1
1.7.1-3
1.9.0-2
0.11-4
1.0.2-1
1.0.2-1
1.12.1+dfsg-19+deb8u2
1.5.9-5+b1
18-3
1.12.1+dfsg-19+deb8u2
1.12.1+dfsg-19+deb8u2
2.6-3+b3
2.4.40+dfsg-1+deb8u2
1.8+dfsg-4
1.05-8+b1
5.2.3-1.1
6.02-1
6.06-2
5.1.1alpha+20120614-2+b3
2.25.2-6
2.4.1-1
5.9+20140913-1+b1
5.9+20140913-1+b1
6.07-1
1.65-1+deb8u1
2.7.1-5+deb8u1

amd64
amd64
amd64
amd64
all
all
all
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
all
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64

libnfnetlink0:amd64
libnspr4:amd64
libnss3:amd64
libocomm
libocomm-dev
libocomm1
libogg0:amd64
liboml2
liboml2-9
liboml2-dev
libp11-kit0:amd64

1.0.1-3
2:4.12-1+debu8u1
2:3.26-1+debu8u1
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
1.3.2-1
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
0.20.7-1

amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64

libpam-modules:amd64
libpam-modules-bin
libpam-runtime
libpam0g:amd64
libpango-1.0-0:amd64
libpangocairo-1.0-0:amd64
libpangoft2-1.0-0:amd64

1.1.8-3.1+deb8u1+b1
1.1.8-3.1+deb8u1+b1
1.1.8-3.1+deb8u1
1.1.8-3.1+deb8u1+b1
1.36.8-3
1.36.8-3
1.36.8-3

amd64
amd64
all
amd64
amd64
amd64
amd64

collection of modules that parse HTML text documents
Data tables pertaining to HTML
Perl module to represent and create HTML syntax trees
HTTP cookie jars
module of date conversion routines
perl interface to HTTP style messages
implementation of content negotiation
X11 Inter-Client Exchange library
International Components for Unicode
GNU Libidn library, implementation of IETF IDN specifications
open an HTML file with automatic charset detection
Perl module implementing object oriented interface to SSL
sockets
Internet Protocol bandwidth measuring tool (runtime files)
JasPer JPEG-2000 runtime library
JBIGkit libraries
libjpeg-turbo JPEG runtime library
JavaScript image cropper UI
JavaScript Framework for dynamic web applications
JavaScript library for dynamic web applications
JSON manipulation library - shared library
GLib JSON manipulation library
GLib JSON manipulation library (common files)
MIT Kerberos runtime libraries - Crypto Library
Linux Key Management Utilities (library)
libkmod shared library
MIT Kerberos runtime libraries
MIT Kerberos runtime libraries - Support library
Little CMS 2 color management library
OpenLDAP libraries
Library for Large Linear Classification
module using libc functions for internationalization in Perl
Shared library for the Lua interpreter version 5.2
module to guess media type for a file or a URL
HTTPS driver for LWP::UserAgent
XZ-format compression library
device mounting library
library for decimal floating point arithmetic (runtime library)
shared libraries for terminal handling
shared libraries for terminal handling (wide character support)
module providing low-level HTTP connection client
Perl module for Secure Sockets Layer (SSL)
low level cryptographic library (symmetric and one-way cryptos)
Netfilter netlink library
NetScape Portable Runtime Library
Network Security Service libraries
OComm: O? Communications Library (metapackage)
OML measurement library headers
OComm: O? Communications Library
Ogg bitstream library
OML: The O? Measurement Library (metapackage)
OML: The O? Measurement Library
OML measurement library headers
Library for loading and coordinating access to PKCS#11 modules - runtime
Pluggable Authentication Modules for PAM
Pluggable Authentication Modules for PAM - helper binaries
Runtime support for the PAM library
Pluggable Authentication Modules library
Layout and rendering of internationalized text
Layout and rendering of internationalized text
Layout and rendering of internationalized text

58 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

libpcap0.8:amd64
libpcre3:amd64
libpcsclite1:amd64
libpgm-5.1-0
libpixman-1-0:amd64
libpng12-0:amd64
libpopt0:amd64
libpq5:amd64
libprocps3:amd64
libproxy1:amd64
libpsl0:amd64
libpulse0:amd64
libpython-stdlib:amd64

1.6.2-2
2:8.35-3.3+deb8u4
1.8.13-1
5.1.118-1 dfsg-1
0.32.6-3
1.2.50-2+deb8u2
1.16-10
9.4.9-0+deb8u1
2:3.3.9-9
0.4.11-4+b2
0.5.1-1
5.0-13
2.7.9-1

amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64

libpython2.7minimal:amd64
libpython2.7-stdlib:amd64

2.7.9-2+deb8u1

amd64

system interface for user-level packet capture
Perl 5 Compatible Regular Expression Library - runtime files
Middleware to access a smart card using PC/SC (library)
OpenPGM shared library
pixel-manipulation library for X and cairo
PNG library - runtime
lib for parsing cmdline parameters
PostgreSQL C client library
library for accessing process information from /proc
automatic proxy configuration management library (shared)
Library for Public Suffix List (shared libraries)
PulseAudio client libraries
interactive high-level object-oriented language (default python
version)
Minimal subset of the Python language (version 2.7)

2.7.9-2+deb8u1

amd64

libpython3-stdlib:amd64

3.4.2-2

amd64

libpython3.4minimal:amd64
libpython3.4-stdlib:amd64

3.4.2-1

amd64

3.4.2-1

amd64

libquadmath0:amd64
libreadline6:amd64
librest-0.7-0:amd64
librrd4
librrds-perl

4.9.2-10
6.3-8+b3
0.7.92-3
1.4.8-1.2
1.4.8-1.2

amd64
amd64
amd64
amd64
amd64

librsvg2-2:amd64
librsvg2-common:amd64
librtmp1:amd64
libruby2.1:amd64
libsasl2-2:amd64
libsasl2-modules-db:amd64
libsctp1:amd64
libselinux1:amd64
libsemanage-common
libsemanage1:amd64
libsepol1:amd64
libsigar
libslang2:amd64
libsm6:amd64
libsmartcols1:amd64
libsndfile1:amd64
libsnmp-session-perl
libsodium13:amd64
libsoup-gnome2.4-1:amd64
libsoup2.4-1:amd64
libsqlite3-0:amd64
libss2:amd64
libssh2-1:amd64
libssl1.0.0:amd64
libstdc++6:amd64
libsystemd0:amd64
libtasn1-6:amd64
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl
libthai-data
libthai0:amd64
libtiff5:amd64

2.40.5-1+deb8u2
2.40.5-1+deb8u2
2.4+20150115.gita107cef-1
2.1.5-2+deb8u3
2.1.26.dfsg1-13+deb8u1
2.1.26.dfsg1-13+deb8u1
1.0.16+dfsg-2
2.3-2
2.3-1
2.3-1+b1
2.3-2
1.6.5-1ppa1o
2.3.0-2
2:1.2.2-1+b1
2.25.2-6
1.0.25-9.1+deb8u1
1.13-1.1
1.0.0-1
2.48.0-1
2.48.0-1
3.8.7.1-1+deb8u2
1.42.12-2
1.4.3-4.1+deb8u1
1.0.1t-1+deb8u5
4.9.2-10
215-17+deb8u5
4.2-3+deb8u2
0.04-7+b3
1.7-5+b2
0.06-7
0.1.21-1
0.1.21-1
4.0.3-12.3+deb8u1

amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
all
amd64
amd64

59 of 67

Interactive high-level object-oriented language (standard library, version 2.7)
interactive high-level object-oriented language (default
python3 version)
Minimal subset of the Python language (version 3.4)
Interactive high-level object-oriented language (standard library, version 3.4)
GCC Quad-Precision Math Library
GNU readline and history libraries, run-time libraries
REST service access library
time-series data storage and display system (runtime library)
time-series data storage and display system (Perl interface,
shared)
SAX-based renderer library for SVG files (runtime)
SAX-based renderer library for SVG files (extra runtime)
toolkit for RTMP streams (shared library)
Libraries necessary to run Ruby 2.1
Cyrus SASL - authentication abstraction library
Cyrus SASL - pluggable authentication modules (DB)
user-space access to Linux Kernel SCTP - shared library
SELinux runtime shared libraries
Common files for SELinux policy management libraries
SELinux policy management library
SELinux library for manipulating binary security policies
System Information Gatherer And Reporter
S-Lang programming library - runtime version
X11 Session Management library
smart column output alignment library
Library for reading/writing audio files
Perl support for accessing SNMP-aware devices
Network communication, cryptography and signaturing library
HTTP library implementation in C – GNOME support library
HTTP library implementation in C – Shared library
SQLite 3 shared library
command-line interface parsing library
SSH2 client-side library
Secure Sockets Layer toolkit - shared libraries
GNU Standard C++ Library v3
systemd utility library
Manage ASN.1 structures (runtime)
get display widths of characters on the terminal
converts between character sets in Perl
internationalized substitute of Text::Wrap
Data files for Thai language support library
Thai language support library
Tag Image File Format (TIFF) library

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

libtimedate-perl
libtinfo5:amd64
libtrace3

2.3000-2
5.9+20140913-1+b1
3.0.21-1

all
amd64
amd64

libudev1:amd64
liburi-perl
libusb-0.1-4:amd64
libusb-1.0-0:amd64
libustr-1.0-1:amd64
libuuid1:amd64
libvorbis0a:amd64
libvorbisenc2:amd64
libwandio1
libwayland-client0:amd64
libwayland-cursor0:amd64
libwrap0:amd64
libwww-perl
libwww-robotrules-perl
libx11-6:amd64
libx11-data
libx11-xcb1:amd64
libxau6:amd64
libxaw7:amd64
libxcb-dri2-0:amd64
libxcb-dri3-0:amd64
libxcb-glx0:amd64
libxcb-present0:amd64
libxcb-render0:amd64
libxcb-shm0:amd64
libxcb-sync1:amd64
libxcb1:amd64
libxcomposite1:amd64
libxcursor1:amd64
libxdamage1:amd64
libxdmcp6:amd64
libxext6:amd64
libxfixes3:amd64
libxfont1:amd64
libxi6:amd64
libxinerama1:amd64
libxkbcommon0:amd64
libxkbfile1:amd64
libxml2:amd64
libxmu6:amd64
libxmuu1:amd64
libxpm4:amd64
libxrandr2:amd64
libxrender1:amd64
libxshmfence1:amd64
libxt6:amd64
libxtables10
libxtst6:amd64
libxxf86vm1:amd64
libyaml-0-2:amd64
libzmq3:amd64
login
lsb-base
mawk
mgen
mime-support
mount
multiarch-support
nano

215-17+deb8u5
1.64-1
2:0.1.12-25
2:1.0.19-1
1.0.4-3+b2
2.25.2-6
1.3.4-2
1.3.4-2
3.0.21-1
1.6.0-2
1.6.0-2
7.6.q-25
6.08-1
6.01-1
2:1.6.2-3
2:1.6.2-3
2:1.6.2-3
1:1.0.8-1
2:1.0.12-2+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1.10-3+b1
1:0.4.4-1
1:1.1.14-1+b1
1:1.1.4-2+b1
1:1.1.1-1+b1
2:1.3.3-1
1:5.0.1-2+b2
1:1.5.1-1
2:1.7.4-1+b2
2:1.1.3-1+b1
0.4.3-2
1:1.0.8-1
2.9.1+dfsg1-5+deb8u3
2:1.1.2-1
2:1.1.2-1
1:3.5.11-1+b1
2:1.4.2-1+b1
1:0.9.8-1+b1
1.1-4
1:1.1.4-1+b1
1.4.21-2+b1
2:1.2.2-1+b1
1:1.1.3-1+b1
0.1.6-3
4.0.5+dfsg-2+deb8u1
1:4.2-3+deb8u1
4.1+Debian13+nmu1
1.3.3-17
5.02+dfsg2-3
3.58
2.25.2-6
2.19-18+deb8u6
2.2.6-3

amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
all
amd64
all
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
amd64
all
amd64
amd64
all
amd64
amd64
amd64

collection of modules to manipulate date/time information
shared low-level terminfo library for terminal handling
network trace processing library supporting many input formats
libudev shared library
module to manipulate and access URI strings
userspace USB programming library
userspace USB programming library
Micro string library: shared library
Universally Unique ID library
decoder library for Vorbis General Audio Compression Codec
encoder library for Vorbis General Audio Compression Codec
multi-threaded file compression and decompression library
wayland compositor infrastructure - client library
wayland compositor infrastructure - cursor library
Wietse Venema’s TCP wrappers library
simple and consistent interface to the world-wide web
database of robots.txt-derived permissions
X11 client-side library
X11 client-side library
Xlib/XCB interface library
X11 authorisation library
X11 Athena Widget library
X C Binding, dri2 extension
X C Binding, dri3 extension
X C Binding, glx extension
X C Binding, present extension
X C Binding, render extension
X C Binding, shm extension
X C Binding, sync extension
X C Binding
X11 Composite extension library
X cursor management library
X11 damaged region extension library
X11 Display Manager Control Protocol library
X11 miscellaneous extension library
X11 miscellaneous ’fixes’ extension library
X11 font rasterisation library
X11 Input extension library
X11 Xinerama extension library
library interface to the XKB compiler - shared library
X11 keyboard file manipulation library
GNOME XML library
X11 miscellaneous utility library
X11 miscellaneous micro-utility library
X11 pixmap library
X11 RandR extension library
X Rendering Extension client library
X shared memory fences - shared library
X11 toolkit intrinsics library
netfilter xtables library
X11 Testing – Record extension library
X11 XFree86 video mode extension library
Fast YAML 1.1 parser and emitter library
lightweight messaging kernel (shared library)
system login tools
Linux Standard Base 4.1 init script functionality
a pattern scanning and text processing language
packet generator for IP network performance tests
MIME files ’mime.types’ & ’mailcap’, and support programs
Tools for mounting and manipulating filesystems
Transitional package to ensure multiarch compatibility
small, friendly text editor inspired by Pico

60 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

ncurses-base
ncurses-bin
net-tools
netbase
netperf
nmap
nmetrics-oml2

5.9+20140913-1
5.9+20140913-1+b1
1.60-26+b1
5.3
2.7.0-1
6.47-3+deb8u2
2.11.0-mytestbed2

all
amd64
amd64
all
amd64
amd64
amd64

oml2
oml2-apps
oml2-proxy-server
oml2-proxycon
oml2-server
openjdk-7-jreheadless:amd64
openssh-client
openssh-server

2.11.1 rc-mytestbed1
2.11.0-mytestbed2
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
2.11.1 rc-mytestbed1
7u111-2.6.7-2 deb8u1

amd64
amd64
amd64
amd64
amd64
amd64

basic terminal type definitions
terminal-related programs and man pages
NET-3 networking toolkit
Basic TCP/IP networking system
Network performance benchmark
The Network Mapper
Measure and record system information from libsigar using
OML
OML: The O? Measurement Library Suite (Metapackage)
Standalone OML2 applications (metapackage)
OML proxy server
OML proxy server control script
OML measurement server
OpenJDK Java runtime, using Hotspot JIT (headless)

1:6.7p1-5+deb8u3
1:6.7p1-5+deb8u3

amd64
amd64

openssh-sftp-server

1:6.7p1-5+deb8u3

amd64

openssl
otg2-oml2
passwd
perl
perl-base
perl-modules
procps
python

1.0.1t-1+deb8u5
2.11.0-mytestbed2
1:4.2-3+deb8u1
5.20.2-3+deb8u6
5.20.2-3+deb8u6
5.20.2-3+deb8u6
2:3.3.9-9
2.7.9-1

amd64
amd64
amd64
amd64
amd64
all
amd64
amd64

python-chardet
python-colorama
python-distlib
python-html5lib

2.3.0-1
0.3.2-1
0.1.9-1
0.999-3

all
all
all
all

python-meld3
python-minimal
python-netifaces
python-pip
python-pkg-resources
python-requests

1.0.0-1
2.7.9-1
0.10.4-0.1
1.5.6-5
5.5.1-1
2.4.3-6

amd64
amd64
amd64
all
all
all

python-rrdtool
python-scapy
python-setuptools
python-six
python-urllib3
python-zmq
python2.7
python2.7-minimal
python3

1.4.8-1.2
2.2.0-1
5.5.1-1
1.8.0-1
1.9.1-3
14.4.0-1
2.7.9-2+deb8u1
2.7.9-2+deb8u1
3.4.2-2

amd64
all
all
all
all
amd64
amd64
amd64
amd64

python3-minimal

3.4.2-2

amd64

python3-netifaces
python3-six
python3-zmq
python3.4
python3.4-minimal
readline-common
ripwavemon-oml2
rsync
ruby

0.10.4-0.1
1.8.0-1
14.4.0-1
3.4.2-1
3.4.2-1
6.3-8
2.11.0-mytestbed2
3.1.1-3
1:2.1.5+deb8u2

amd64
all
amd64
amd64
amd64
all
amd64
amd64
all

61 of 67

secure shell (SSH) client, for secure access to remote machines
secure shell (SSH) server, for secure access from remote machines
secure shell (SSH) sftp server module, for SFTP access from remote machines
Secure Sockets Layer toolkit - cryptographic utility
Orbit Traffic Generator
change and administer password and group data
Larry Wall’s Practical Extraction and Report Language
minimal Perl system
Core Perl modules
/proc file system utilities
interactive high-level object-oriented language (default version)
universal character encoding detector for Python2
Cross-platform colored terminal text in Python - Python 2.x
low-level components of python distutils2/packaging
HTML parser/tokenizer based on the WHATWG HTML5 specification (Python 2)
HTML/XML templating system for Python
minimal subset of the Python language (default version)
portable network interface information - Python 2.x
alternative Python package installer
Package Discovery and Resource Access using pkg_resources
elegant and simple HTTP library for Python2, built for human
beings
time-series data storage and display system (Python interface)
Packet generator/sniffer and network scanner/discovery
Python Distutils Enhancements
Python 2 and 3 compatibility library (Python 2 interface)
HTTP library with thread-safe connection pooling for Python
Python bindings for 0MQ library
Interactive high-level object-oriented language (version 2.7)
Minimal subset of the Python language (version 2.7)
interactive high-level object-oriented language (default
python3 version)
minimal subset of the Python language (default python3 version)
portable network interface information - Python 3.x
Python 2 and 3 compatibility library (Python 3 interface)
Python3 bindings for 0MQ library
Interactive high-level object-oriented language (version 3.4)
Minimal subset of the Python language (version 3.4)
GNU readline and history libraries, common files
Report statistics from a Navini RipWave modem
fast, versatile, remote (and local) file-copying tool
Interpreter of object-oriented scripting language Ruby (default
version)

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 10: List of packages installed in monroe/base. (Continued)
Name

Version

Architecture

Description

ruby2.1
rubygems-integration
sed
sensible-utils
shared-mime-info
smokeping
speedtest-cli

2.1.5-2+deb8u3
1.8
4.2.2-4+b1
0.0.9
1.3-1
2.6.9-1+deb8u1
1.0.0

amd64
all
amd64
all
amd64
all
all

startpar
supervisor
systemd
systemd-sysv
sysv-rc
sysvinit-utils
tar
tcpdump
trace-oml2
traceroute
tzdata
tzdata-java

0.59-3
3.0r1-1
215-17+deb8u5
215-17+deb8u5
2.88dsf-59
2.88dsf-59
1.27.1-2+deb8u1
4.6.2-5+deb8u1
2.11.0-mytestbed2
1:2.0.20-2+b1
2016j-0+deb8u1
2016j-0+deb8u1

amd64
all
amd64
amd64
all
amd64
amd64
amd64
amd64
amd64
all
all

ucf

3.0030

all

udev
util-linux
wget
x11-common
x11-xkb-utils
xauth
xkb-data
xserver-common
xvfb
xz-utils
zlib1g:amd64

215-17+deb8u5
2.25.2-6
1.16-1+deb8u1
1:7.7+7
7.7+1
1:1.0.9-1
2.12-1
2:1.16.4-1
2:1.16.4-1
5.1.1alpha+20120614-2+b3
1:1.2.8.dfsg-2+b1

amd64
amd64
amd64
all
amd64
amd64
all
all
amd64
amd64
amd64

Interpreter of object-oriented scripting language Ruby
integration of Debian Ruby packages with Rubygems
The GNU sed stream editor
Utilities for sensible alternative selection
FreeDesktop.org shared MIME database and spec
latency logging and graphing system
Command line interface for testing internet bandwidth using
speedtest.net
run processes in parallel and multiplex their output
A system for controlling process state
system and service manager
system and service manager - SysV links
System-V-like runlevel change mechanism
System-V-like utilities
GNU version of the tar archiving utility
command-line network traffic analyzer
Measure and record libtrace data using OML
Traces the route taken by packets over an IPv4/IPv6 network
time zone and daylight-saving time data
time zone and daylight-saving time data for use by java runtimes
Update Configuration File(s): preserve user changes to config
files
/dev/ and hotplug management daemon
Miscellaneous system utilities
retrieves files from the web
X Window System (X.Org) infrastructure
X11 XKB utilities
X authentication utility
X Keyboard Extension (XKB) configuration data
common files used by various X servers
Virtual Framebuffer ’fake’ X server
XZ-format compression utilities
compression library - runtime

B

Description of metadata fields

The following description of metadata topics and fields is included here for convenience only. The updated version is kept at: https:
//github.com/MONROE-PROJECT/Experiments/wiki/Metadata
DataVersion was set to 1 until beginning of March, 2017. It was then set to 2 to signal the transition in timestamps that most tables
underwent, when their precision was changed from an integer number of seconds (e.g., 1 400 475 869) to the number of seconds and
microseconds since the (UNIX) Epoch (e.g., 1 400 475 869.123 456).

Table 11: Field description for metadata topic “MONROE.META.DEVICE.MODEM”.
Name

Description

NodeId
Timestamp
DataId
DataVersion
SequenceNumber
InterfaceName
InternalInterface

Node numerical ID.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Metadata topic.
Last version is 2.
Monotonically increasing message counter.
Name of the interface in the MONROE node, e.g., “usb0”, “usb1”, “usb2”, “eth0”, . . .
Name of the interface inside the containers, e.g., “op0”, “op1”, “op2”, “eth0”, “wlan0”, . . . Experiments in containers have to bind to these interface names.
Cell ID.
Connection mode of the modem (e.g., 2G, 3G, LTE) indicating the radio access technology the modem uses.
Connection submode for 3G connections (e.g., CDMA, WCDMA, UMTS).

Cid
DeviceMode
DeviceSubmode

62 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 11: Field description for metadata topic “MONROE.META.DEVICE.MODEM”. (Continued)
Name

Description

DeviceState

State of the device reported to the network: UNKNOWN (0) - Device state is unknwon; REGISTERED (1) Device is registered to the network; UNREGISTERED (2) - Device is unregistered from the network; CONNECTED (3) - Device is connected to the network; DISCONNECTED (4) - Device is disconnected from the
network.
EC/IO, quality/cleanliness of signal from the tower to the modem (dB).
Evolved base station ID.
Internationally defined integrated circuit card identifier of the SIM card.
Internation Mobile Subscriber Identity.
Mobile Country Code (MCC) and Mobile Network Code (MNC).
International Mobile Station Equipment Identity.
IP address assigned to the modem by the operator.
Internal IP address of the modem in the MONROE node.
Mobile Country Code (MCC) and Mobile Network Code (MNC).
Operator name as reported by the network for the interface in which the experiment was run.
Local Area Code for the current cell (hex).
Reference Signal Received Power (LTE).
Frequency in MHz (e.g., 700, 800, 900, 1800 or 2600 in Europe).
Reference Signal Received Quality (valid only for LTE networks). The RSRQ measurement provides additional
information when Reference Signal Received Power (RSRP) is not sufficient to make a reliable handover or
cell reselection decision. RSRQ considers both the Received Signal Strength Indicator (RSSI) and the number
of used Resource Blocks (N) RSRQ = (N ∗ RSRP)/RSSI measured over the same bandwidth.
Band corresponding to the frequency used (e.g., 3, 7 or 20 in Europe).
Physical Cell ID.
Mobile Country Code (MCC) and Mobile Network Code (MNC) from network (read from the network). The
tuple uniquely identifies a mobile network operator (carrier) that is using the GSM (including GSM-R), UMTS,
and LTE public land mobile networks.
Received Signal Code Power (UMTS).
Received Signal Strength Indicator.

Ecio
ENodebId
Iccid
Imsi
ImsiMccMnc
Imei
IpAddress
InternalIpAddress
MccMnc
Operator
Lac
Rsrp
Frequency
Rsrq

Band
Pci
NwMccMnc

Rscp
Rssi

Table 12: Field description for metadata topic “MONROE.META.DEVICE.GPS”.
Name

Description

NodeId
Timestamp
DataId
DataVersion
SequenceNumber
Longitude
Latitude
Altitude
Speed
SatelliteCount
Nmea

Node numerical ID.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Metadata topic.
Last version is 2.
Monotonically increasing message counter.
Decimal degrees (WGS84).
Decimal degrees (WGS84).
Meters AMSL.
Speed over ground (knots; multiply by 1.852 to get km h−1 ).
Number of satellites being tracked.
Raw NMEA string from the GPS receiver.

Table 13: Field description for metadata topic “MONROE.META.NODE.SENSOR”.
Name

Description

NodeId
Timestamp
DataId
DataVersion
SequenceNumber
Running
Cpu
Id
Start
Current
Total
Percent

Node numerical ID.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Metadata topic.
Last version is 2.
Monotonically increasing message counter.
Comma separated list of experiment GUIDs.
CPU temperature (◦C).
Session number (boot counter).
Start time (Unix timestamp).
Uptime (seconds since start of the session).
Uptime (cumulative uptime of the node over all sessions).
Uptime (percent of uptime vs. total lifetime of the node).

63 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 13: Field description for metadata topic “MONROE.META.NODE.SENSOR”. (Continued)
Name

Description

System
Steal
Guest
IoWait
Irq
Nice
Idle
User
SoftIrq
Apps
Free
Swap
usb0
usb0charging
usb1
usb1charging
usb2
usb2charging

CPU time spent by the kernel in system activities.
The time that a virtual CPU had runnable tasks, but the virtual CPU itself was not running.
The time spent running a virtual CPU for guest operating systems under the control of the Linux kernel.
CPU time spent waiting for I/O operations to finish when there is nothing else to do.
CPU time spent handling interrupts.
CPU time spent by nice(1)d programs.
Idle CPU time.
CPU time spent by normal programs and daemons.
CPU time spent handling “batched” interrupts.
Memory used by user-space applications.
Unused memory.
Swap space used.
Battery level for MiFi at USB0 (0-100, -1 for inactive).
1 if USB0 battery is charging, 0 otherwise.
Battery level for MiFi at USB1 (0-100, -1 for inactive).
1 if USB1 battery is charging, 0 otherwise.
Battery level for MiFi at USB2 (0-100, -1 for inactive).
1 if USB2 battery is charging, 0 otherwise.

Table 14: Field description for metadata topic “MONROE.META.NODE.EVENT”.
Name

Description

NodeId
Timestamp
DataId
DataVersion
SequenceNumber
EventType

Node numerical ID.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Metadata topic.
Last version is 2.
Monotonically increasing message counter.
Watchdog.Failed: The system watchdog detected an error symptom.
Watchdog.Repaired: The system watchdog resolved the issue.
Watchdog.Status: Periodic status messages from the watchdog.
Maintenance.Start: An interactive login on the node is registered.
Maintenance.Stop: The interactive login session is closed.
System.Halt: System halt is requested.
Scheduling.Started: The node starts to query the scheduling server.
Scheduling.Task.Deploying: A scheduling task passed checks and is being deployed.
Scheduling.Task.Deployed: A scheduling task has been deployed.
Scheduling.Task.Started: A scheduling task has started.
Scheduling.Task.Stopped: A scheduling task has stopped and is being cleaned up.
Extra key for some event types.
Extra key for some event types.
Extra key for Scheduling.Task.* events.

Message
User
id

Table 15: Field description for metadata topic “MONROE.EXP.PING”.
Name

Description

NodeId
Guid
Timestamp
SequenceNumber
DataId
DataVersion
Operator
Iccid
Bytes
Host
Rtt

Node numerical ID.
Unique experiment identifier.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Monotonically increasing message counter.
Metadata topic.
Last version is 2.
Operator name as reported by the network for the interface in which the experiment was run.
Internationally defined integrated circuit card identifier of the SIM card.
Size of the ping message payload.
IP of the destination host of the ping probe.
Round-Trip-Time of the ping probe.

64 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Table 16: Field description for metadata topic “MONROE.EXP.HTTP.DOWNLOAD”.

C

Name

Description

NodeId
Guid
Timestamp
SequenceNumber
DataId
DataVersion
Operator
Iccid
TotalTime
Bytes
SetupTime
DownloadTime
Host
Speed
Port

Node numerical ID.
Unique experiment identifier.
Entry timestamp (in seconds since UNIX epoch with microsecond precision).
Monotonically increasing message counter.
Metadata topic.
Last version is 2.
Operator name as reported by the network for the interface in which the experiment was run.
Internationally defined integrated circuit card identifier of the SIM card.
Total experiment execution time (in fractional seconds).
Total number of bytes downloaded.
Time required to set up the HTTP connection.
Time spent doing the actual download.
IP address of the remote host from which data was downloaded.
Download speed in bytes/s as measured by the experiment.
TCP port of the remote host from which data was downloaded.

How to map container folders to Windows paths

Before being able to access Windows (host) folders from a container, the drive has to be made available to the containers following these
steps:5,6
1. Access the Docker settings dialog from its taskbar icon:

2. From the tab “Shared Drives”, select the drive you want to make available to the containers, e.g., “C”:

3. You will be prompted for login credentials to access the files:
5 https://rominirani.com/docker-on-windows-mounting-host-directories-d96f3f056a2c#.pdeuy0c4o
6 Thanks to Lena for pointing to the solution.

65 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

4. Start the container mounting the desired folder:
$ docker run -v c:/Users/Dell/myresults:/data container_name ls /data

This command executes ls /data inside an instance of the container “container_name,” after mounting “C:/Users/Dell/myresults/”
into that path.
5. The folder can be accessed normally from Windows and will reflect changes to any files automatically.

References
[1] Te-Yuan Huang, Ramesh Johari, Nick McKeown, Matthew Trunnell, and Mark Watson. A buffer-based approach to rate adaptation:
Evidence from a large video streaming service. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM’14), pages
187–198, New York, NY, USA, 2014. ACM Press.
[2] Juluri P., Tamarapalli V., and D. Medhi. SARA: Segment aware rate adaptation algorithm for dynamic adaptive streaming over HTTP.
In ICC QoE-FI Workshop, June 2015.

66 of 67

Project no. 644399

User manual
MONROE Platform User manual

Public
Rev. 1.0/ March 14, 2019

Disclaimer
The views expressed in this document are solely those of the author(s). The European Commission is not responsible for any use that may be made of the information it contains.
All information in this document is provided “as is”, and no guarantee or warranty is given that
the information is fit for any particular purpose. The user thereof uses the information at its sole
risk and liability.

67 of 67

Project no. 644399



---

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.5
Linearized                      : No
Page Count                      : 67
Page Mode                       : UseOutlines
Author                          : 
Title                           : 
Subject                         : 
Creator                         : LaTeX with hyperref package
Producer                        : pdfTeX-1.40.18
Create Date                     : 2019:03:14 17:21:38+01:00
Modify Date                     : 2019:03:14 17:21:38+01:00
Trapped                         : False
PTEX Fullbanner                 : This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) kpathsea version 6.2.3

EXIF Metadata provided by EXIF.tools