A Framework Supporting Replicable Performance Evaluations in Networking

This repository contains an implementation of TriScale, a framework supporting reproducible performance evaluations in networking.

TriScale is described in details in the following paper

Designing Replicable Networking Experiments with TriScale
Romain Jacob, Marco Zimmerling, Carlo Alberto Boano, Laurent Vanbever, Lothar Thiele
Journal of Systems Research (JSys), 2021.
doi.org/10.5070/SR31155408

Check out the project webpage for more info:
http://triscale.ethz.ch/

---

        

---

• Source code
• Reproducing the paper
  • Reproduction alternatives
  • Case studies
  • Scalability Evaluation
  • Plots

---

Source code

TriScale source code is split in the following files

• triscale.py
  Implements the main functions of TriScale (the public API).
• helpers.py
  Additional support functions for running the TriScale analysis.
• triplots.py
  Plotting functions used in TriScale. All plots are created using Plotly.

Reproducing the paper

This repository contains all the necessary code and information to reproduce the results analysis and plots presented in the TriScale paper. Jupyter notebooks are used as wrappers for all our data analysis.

Note. If you are not familiar with Jupyter notebooks, you can find some basic information on TriScale's tutorial page.

Reproduction alternatives

We provide three options to reproduce the paper results:

1. Online, using Binder;
2. Online, using nbviewer;
3. Locally.

Keep in mind that running the plotting and case study notebooks require to download the experiment raw data (available in a separate Zenodo repository); in particular, the congestion-control data zip file is about 2.7 GB large. The required download commands are included in the respective notebooks.

Recommended alternative
To quickly try out the API	Binder
If not familiar with Python environments	Binder or nbviewer
For a quick look	nbviewer
With limited Internet connection	nbviewer
For a more in-depth trial	locally

Using MyBinder.org

You can run the TriScale's notebooks directly in your web-browser, thanks to the amazing service provided by MyBinder.org!

1. Click this launch binder button (it may take a few minutes to load);
   
2. Open any notebook in the repository (see the notebooks' description below);
3. Explore the different functions, run the example code snippets, modify them, run them again...

Note. You cannot save the state of the notebooks open using Binder; once your session expires, you will not be able to retrieve changes you made, or outputs you computed.
If you want to do that, you should run the notebooks locally.

MyBinder.org in a nutshell

MyBinder.org is a service that fetches a public repository, builds a Docker image with all required dependencies, then runs and serves an image in a cloud platform, making it accessible to anyone with on the web, without requiring any install!

Best of all, MyBinder.org is a free and open-source service, managed for scientists by scientists. Check it out, it's really cool!

Using nbviewer

You can simply visualize the notebooks using nbviewer.

Note. The code cannot be run, but you can read all descriptions, comments, and see the outputs. Links for visualizing notebooks in nbviewerare provided in the descriptions below.

Running locally

You can run the notebooks locally by cloning this repository. The list of dependencies is included in the environment.yml file.

If you are using conda, simply execute the following commands to get started:

git clone [email protected]:romain-jacob/triscale.git
cd triscale
conda env create -f environment.yml
conda activate myenv
jupyter notebooks

If you prefer pip over conda, you will also find a requirements.txt file, but this one is not guaranteed to be complete.

Case studies

The TriScale paper includes a set of case studies that illustrate the use and benefits of TriScale for concrete networking performance evaluations. This repository contains the notebooks that allows to reproduce the analysis of these case studies. The raw data are available in a separate Zenodo repository, each notebook contains the required wget commands.

casestudy_congestion-control.ipynb

 

This case study presents a comparison between 17 congestion-control schemes. It reproduces some of the analysis performed in the Pantheon paper, illustrates its limitations, and shows how the approach in TriScale helps to overcome them.

casestudy_glossy.ipynb

 

This case study compares the performance of Glossy, a low-power wireless communication protocol, for different parameter values, using the FlockLab testbed as experiment environment. In particular, this case study illustrates the importance of network profiling: this example shows how one may reach wrong conclusions (even with high confidence!) when the environmental conditions are not properly assessed.

casestudy_failure-detection.ipynb

 

This case study revisits the analysis of Blink, an algorithm that detects failures and reroutes traffic directly in the data plane. We only perform the data analysis and show how using TriScale allows to generalize the results.

casestudy_video-streaming.ipynb

 

This case study revisits the analysis of Pensieve, a system that generates adaptive bitrate algorithms for video streaming using reinforcement learning. We show how TriScale can be used to provide confidence intervals not only on single KPIs, but on entire cumulative distribution functions (CDFs).

Scalability Evaluation

The scalability evaluation of TriScale can be reproduced by running the triscale_scalability.ipynb notebook.

 

Keep in mind that, for this evaluation, the results depend on the computing power and load of the machine, therefore you may obtain results slightly different from those reported in the paper. However, the trends and orders of magnitude should remain similar.

Running this notebook on Binder is technically possible but not recommended: it includes time measurement of rather heavy computations, which is unlikely to be meaningful (the current load of the servers hosting MyBinder.org is unknown and unpredictable.)

Plots

All plots presented in the TriScale paper have been produced using the triscale_plots.ipynb notebook. This same notebook produces larger versions of these plots if you want to have a closer look.

 

In order to run this notebook (either in Binder or locally), you will need to download the data from all the case studies. The required download commands are included in the notebook.