Source code of "Mobile-Kube: Mobility Aware and Energy Efficient Service Orchestration on Kubernetes Edge Servers"
In recent years Kubernetes has become the de facto standard in the realm of service orchestration. Despite its great benefits, there is still a long way to make compatible with decentralised cloud computing platforms. There is an ongoing effort to make it available on distributed edge computing nodes/servers. One of the challenges of mobile edge computing is that the location of the users is changing over time. This mobility will constantly move users to a distant proximity of their connected services. One solution to this problem is to regularly move services to computing nodes closer to the most of the users. However, distributing the services in edge nodes only subject to user movements will result in fragmentation of active nodes. This leads to having a large number of active nodes without the use of their full capacity. In this research we have proposed a method to reduce the latency of Kubernetes applications on mobile edge computing devices while maintaining energy consumption at a reasonable level. Latecncy reduction is achieved by moving the services to Kubernetes nodes closer to the users. On the other hand, Energy consumption is reduced by trying to have the minimum number of active nodes in the Kubernetes cluster. A new state of the art reinforcement learning algorithm named IMPALA have been used in order to maintain a balance between theses two objectives. An experimental framework is designed on top of real-world Kubernetes clusters and real world traces of mobile users movements have been used to simulate the users' mobility. Experimental results have shown the feasibility of design and using reinforcement learning in container placement in Kubernetes driven MEC networks.
- Objectives
- latency reduction: having the services closer to the users
- energy comsomption: having the minimum number of active Kuberentes nodes
-
Download source code from GitHub
git clone https://github.com/saeid93/mobile-kube.git -
Download and install miniconda
-
Create conda virtual-environment
conda create --name mobilekube python=3 -
Activate conda environment
conda activate mobilekube -
if you want to use GPUs make sure that you have the correct version of CUDA and cuDNN installed from here
-
Use PyTorch or Tensorflow isntallation manual to install one of them based-on your preference
-
Install the followings
sudo apt install cmake libz-dev -
Install requirements
pip install -r requirements.txt -
setup tensorboard monitoring
If you want to do the real world Kubernetes experiemnts of the paper you should also do the following steps.
There are several options for setting up a Kuberentes cluster. The repo codes can connect to the cluster through the Python client API as long as you have access to the kube config address e.g. ~/.kube/config in your config files specified.
We have used Google Cloud Platform for our experiments. You can find the toturial for creating the cluster on google cloud and locally in the followings:
The code is separated into three modules
- data: This is the folder containing all the configs and results of the project. Could be anywhere in the project.
- mobility-dataset-preprossing: These scripts are used for preprocessing the Cabspotting and California towers stations dataset used in dataset-driven datasets.
- mobile-kube: the core simulation library with Open-AI gym interface
- experiments: experiments of the paper and the reinforcement learning side of codes.
3.1. mobile-kube
- src: The folder containing the mobile-kube simulators. This should be installed for using.
Go to the mobile-kube and install the library in the editable mode with
pip install -e .
3.2. data
Link the data folder (could be placed anywhere in your harddisk) to the project. A sample of the data folder is available at data.
Go to experiments/utils/constants.py and set the path to your data and project folders in the file. For example:
DATA_PATH = "/Users/saeid/Codes/mobile-kube/data"
- Install the requirements:
pip install -r requirements.txt- Run the code:
python main.py- The options are as follows:
Usage: main.py [OPTIONS]
Options:
-d, --dataset TEXT Directory of Cabspotting data set [default:
data/*.txt]
-g, --get BOOLEAN Get data set from the internet [default: False]
-u, --url TEXT The url of Cabspotting data set [default: ]
-i, --interval INTEGER Enter the intervals between two points in seconds
[default: 100]
--help Show this message and exit.
- Enjoy!
formant of the parsed users mobility dataset. Users data attributes:
- Creating a CSV file with the size of: 2 * #NUM_OF_USERS + 1, each 3 column contains the information of one user.
- For instance, in case when there are 3 users in the network, we have a 7-column table as follows:
| timesteps | lan1 | lat1 | lan2 | lat2 |
|---|---|---|---|---|
| 0 | 35.76727803828804 | 51.35991084161459 | 35.76031345070641 | 1.39458643788907 |
| 1 | 35.76727803828856 | 51.35991084161443 | 35.76031345070765 | 1.39458643788633 |
| 2 | 35.76727803828856 | 51.35991084161443 | 35.76031345070765 | 1.39458643788633 |
Copy the results at the data/dataset_metadata in a numbered foldered to use it in experiments generators.
3.4. experiments
3.4.1. Data Generation
The dataset, workloads, networks and traces are generated in the following order:
- Datasets: Nodes, services, their capacities, requested resources and their initial placements.
- Workloads: The workload for each dataset that determines the resource usage at each time step. This is built on top of the datasets built on step 1. Each dataset can have several workloads.
- Networks: The edge network for each dataset that generates the network with users and stations for the simulation. This contains the network object containing the nodes and stations and intitial location of the users. The network can be built both randomly of based-on the Cabspotting and California towers stations dataset. This is built on top of the nodes-services datasets built on step 1. Each dataset can have several networks.
- Traces: The movement traces for each network. This is the location of each user at each timestep. This can be a random or based-on Cabspotting dataset. This is built on top of the networks built in step 3.
To generate the datasets, workloads, networks and traces, first go to your data folder (remember data could be anywhere in your disk just point the data folder as experiments/utils/constants.py).
3.4.1.1. Generating the Datasets
Go to the your dataset generation config data/configs/dataset-generation/ make a folder named after your config and make the config.json in the folder e.g. see the my-dataset in the sample data folder data/configs/generation-configs/dataset-generation/my-dataset/config.json. Then run the experiments/dataset/generate_dataset.py with the following script:
python generate_dataset.py [OPTIONS]
Options:
--dataset-config-folder TEXT config-folder
[default: my-dataset]
For a full list of config.json parameters options see dataset-configs-options. The results will be saved in data/datasets/<dataset_id>.
4.4.1.2. Generating the Workloads
Go to the your workload generation config data/configs/generation-configs/workload-generation make a folder named after your config and make the config.json in the folder e.g. see the my-workload in the sample data folder data/configs/generation-configs/workload-generation/my-workload/config.json. For a full list of config.json see. Then run the experiments/dataset/generate_dataset.py with the following script:
python generate_workload.py [OPTIONS]
Options:
--workload-config-folder TEXT config-folder
[default: my-workload]
For a full list of config.json parameters options see workload-configs-options. The results will be saved in data/datasets/<dataset_id>/<workload_id>.
4.4.1.3. Generating the Networks
Go to the your dataset generation config data/configs/generation-configs/network-generation make a folder named after your config and make the config.json in the folder e.g. see the my-network in the sample data folder data/configs/generation-configs/dataset-generation/my-network/config.json. Then run the experiments/dataset/generate_network.py with the following script:
python generate_network.py [OPTIONS]
Options:
--network-config-folder TEXT config-folder
[default: my-network]
For a full list of config.json parameters options see network-configs-options. The results will be saved in data/datasets/<dataset_id>/<network_id>.
4.4.1.4. Generating the Traces
Go to the your trace generation config data/configs/generation-configs/trace-generation make a folder named after your config and make the config.json in the folder e.g. see the my-trace in the sample data folder data/configs/generation-configs/trace-generation/my-trace/config.json. Then run the experiments/dataset/generate_trace.py with the following script:
python generate_trace.py [OPTIONS]
Options:
--trace-config-folder TEXT config-folder
[default: my-trace]
For a full list of config.json parameters options see trace-configs-options. The results will be saved in data/datasets/<dataset_id>/<network_id>/<trace_id>.
4.4.2.1. Training the agent
- change the training parameters in
<configs-path>/real/<experiment-folder>/config_run.json. For more information about the hyperparamters in this json file see hyperparameter guide - To train the environments go to the parent folder and run the following command.
python experiments/learning/learners.py --mode real --local-mode false --config-folder PPO --type-env 0 --dataset-id 0 --workload-id 0 --use-callback true
4.4.2. Analysis
4.4.2.1 check_env
4.4.2.2 check_learned
4.4.2.3 test_baselines
4.4.2.4 check environment
4.4.2.5 check environment
4.4.2.6 check environment
4.4.4. Kubernetes interface
The Kubernetes interface is designed based-on the Kubernetes api version 1.
The main operations that are currently implemented are:
- creating
- cluster
- utilisation server
- pods
- actions
- moving pods
- deleting pods
- cleaning nemespace
- monitoring
- get nodes resource usage
- get pods resource usage
a sample of using the interface can be found here
Log of a running emulation - moving service 0 from node 1 to node 0 (s0n1 -> s0n1)
Google cloud console of a running emulation - moving service 0 from node 1 to node 0 (s0n1 -> s0n1)
