nephele is a CLI utility that provides capabilities to create, manage, and migrate KVM virtual machines over the wide-area network. Built upon the principles devised and implemented by Kiryong Ha, nephele utilizes deduplication, compression, and bandwidth adaptation to migrate VMs efficiently in WAN environments where the bandwidth and latency characteristics are far below those found in datacenters where traditional VM migration happens. More information, including related publications, can be found on our website.
Copyright (C) 2011-2020 Carnegie Mellon University
All source code and documentation (except modified-QEMU listed below) are under the Apache License, Version 2.0.
A copy of this license is reproduced in the LICENSE file.
To enable on-demand fetching of the virtual machine, we use a modified-QEMU, which is distributed under GPLv2.
We have tested the nephele server components on Ubuntu 16.04 LTS 64-bit. The nephele client has been tested on Ubuntu 16.04 LTS 64-bit and Ubuntu 18.04 LTS 64-bit. Both client and server require Python 2.7.x.
The ansible inventory file contains a [servers] block which lists all the remote servers you wish to run the playbook against. If you are launching the playbook from the server itself, you can just leave 'localhost'. If you want to run against multiple remote servers in parallel, you can specify all of them in the inventory file.
NOTE: If you are running the ansible playbook against remote machines, the client machines's ssh key must reside in root's authorized_keys file on each nephele server node. You can use ssh-keygen to create a public key if you need to and then use ssh-copy-id root@<node> to push it to the server nodes, assuming you can authenticate to them.
sudo apt install git ansible
git clone https://github.com/cmusatyalab/nephele
cd nephele/ansible
ansible-playbook -i inventory server.ymlTo refresh the git repository, install new source code, and restart the servers you can run the playbook with the following tags:
ansible-playbook -i inventory server.yml --tags git,pippackages,services
The is the typical scenario to upgrade the code base.
Your client's ssh key must reside in root's authorized_keys file on each nephele node. You can use ssh-keygen to create a public key if you need to and then use ssh-copy-id root@<node> to push it to the server nodes, assuming you can authenticate to them.
sudo apt install git ansible
git clone https://github.com/cmusatyalab/nephele
cd nephele/ansible
ansible-playbook client.yml --ask-sudo-passvirt-manager is used for display/keyboard/mouse interaction with virtual machine instances. You should run virt-manager and add remote connections to each nephele server node in order to see what instances are running there and interact with them.
While nephele VMs can be viewed/interacted with via virt-manager just like normal KVM/QEMU instances...
...they have to be managed by using nephele-client which has the following command-line interface:
$ nephele-client
usage: nephele [-h] host {handoff,image,images,ps,run,snapshot,snapshots} ...
Manage VMs across cloudlets.
positional arguments:
host Execute commands on specified remote host.
optional arguments:
-h, --help show this help message and exit
command:
{handoff,image,images,ps,run,snapshot,snapshots}
handoff Handoff a running instance to another nephele node
image Management of base VM images
images List base images tracked by nephele
ps List running nephele instances
run Instantiate a VM from an existing snapshot
snapshot Management of VM snapshots
snapshots List snapshots known to nephele
usage: nephele [-h] host {handoff,image,images,ps,run,snapshot,snapshots} ...
nephele: error: too few argumentsTo launch virtual machines using nephele, you first need to have images to work with. To build a nephele compatible image, you use the nephele <host> image build command. This will lauch and instance of the image, boot into the OS, and then allow you to customize it to your liking. When you have reached the point at which you want to generate a nephele image, just pause the VM in virt-manager using the menu or the pause icon. After the VM is paused, nephele will take the precise memory and disk state and copy it to /var/nephele/images.
$ nephele-client localhost image build -h
usage: nephele host image build [-h] [-t TITLE] [-c CPU] [-m MEM] path
positional arguments:
path Path to an existing base image (will be converted to
raw format)
optional arguments:
-h, --help show this help message and exit
-t TITLE, --title TITLE
Title of the VM instance to display in virt-manager
-c CPU, --cpu CPU Override number of vCPUs. By default, the number
specified in the VM_TEMPLATE.xml will be used.
-m MEM, --mem MEM Override amount of memory (in KiB). By default, the
number specified in the VM_TEMPLATE.xml will be used.You can use the -c and -m option to explicitly set the size the of the VM. By default, the values in VM_TEMPLATE.xml will be used. If you do not provide a title with the -t option, the VM will be entitled nephele-(libvirt UUID).
You can list the images nephele is aware of with nephele-client host image ls or nephele-client host images.
./nephele-client localhost images
IMAGE ID NAME (/var/nephele/images/) SOURCE
edbe05049453 win10.qcow2 /root/win10_base.zip
ee4720c0ca33 Win7.img /root/win7horizon.zip
d7d501d19f06 ubuntu1804-12vcpu.img /root/ubuntu1804-12cpu.zipTo migrate an instance from one node to another using VM handoff, you will need to have the base image installed on each nephele server. To do this, you can export the image from the node that it was built on, and import it on the other nodes.
For example:
$ nephele-client nodeA image export edbe05049453 /root/gold_image.zip
(scp /root/gold_image.zip nodeB:/root)
$ nephele-client nodeB image import /root/gold_image.zipYou can remove nephele images by using nephele-client <host> image rm <id>
NOTE: Removing an image will render any snapshots that have been built atop that image useless. You should also remember to remove the image from ALL nephele nodes so that handoffs do not fail because the image exists in some places but not others.
Snapshots are customized memory/disk deltas with respect to a base image. You must build a snapshot before you can launch (and subsequently handoff) a VM instance. Building a snapshot follows a similar process to building an image: you launch a base image using nephele-client <host> snapshot build <image id> <destination path>, customize it until you are satisfied, and then you pause the VM in virt-manager using the menu or the pause icon. You can specify a title with the -t option so it is easier to find in virt-manager.
$ ./nephele-client localhost snapshot build -h
usage: nephele host snapshot build [-h] [-t TITLE] [-d] [-m] [-n] [-z] id dest
positional arguments:
id an existing base image to start with
dest Destination path for snapshot
optional arguments:
-h, --help show this help message and exit
-t TITLE, --title TITLE
Title of the VM instance to display in virt-manager
-d, --disk-only Ignore memory, create delta from disk blocks only.
(default=False)
-m, --free-mem Extract free memory (default=False)
-n, --no-trim Disable TRIM support. (default=False)
-z, --zip Package snapshot files into a single zip. (default=True)You can list the snapshots that a nephele node is aware of with nephele-client host snapshot ls or nephele-client host snapshots.
NOTE: Snapshots only reside on the nephele node in which they were built. If you wish to launch an instance of a snapshot from another host, you will need to scp that snapshot first.
$ ./nephele-client localhost snapshots
SNAPSHOT IMAGE ID CREATED
/root/t2.zip edbe05049453 2020-02-28 15:23:42.956169
/root/rpcbuild.zip edbe05049453 2020-03-03 08:49:12.872102
/root/rpcbuild_1.zip edbe05049453 2020-03-03 14:22:10.750467You can remove nephele snapshots by using nephele-client <host> snapshot rm <path>.
NOTE: This will not affect the base image that the snapshot was built upon.
You can create iterative snapshots by launching an instance of an existing snapshot, interacting with the VM, and then pausing it in virt-manager using the menu or the pause icon. This will create a snapshot in the form original_snapshot_name_# where # is the next sequential number on disk, starting with 1. For example, if you launched an instance with nephele-client localhost run /root/my_snapshot.zip mysnapshot and installed some security updates and then paused the VM, a snapshot would be created called __/root/my_snapshot_1.zip*.
NOTE: Keep in mind that both my_snapshot.zip and my_snapshot1.zip are built upon the same base image; snapshots are flat deltas and are not hierarchical.
Once nephele has some images and snapshots to work with, we can launch virtual machine instances that are restored to the exact state we left them in when those snapshots were created. This is done with nephele-client <host> run <snapshot path> <title>. The -p option can be used to expose host ports to the VM instance. For instance, if you needed to be able to remote into the VM over the RDP protocol, you could pass -p 3389 to forward the host 3389 traffic onto the VM.
NOTE: A port can only be bound to a single VM instance, so any attempts to launch a subsequent VM that needs any port that is already being forwarded to an existing instance will fail. This applies whether the VM is launched initially on that node, or was the result of a handoff from another nephele node.
$ ./nephele-client localhost run -h
usage: nephele host run [-h] [-d] [-p PORTS] snapshot title
positional arguments:
snapshot Disk/memory snapshot to instantiate from
title Title of the VM instance to display in virt-manager
optional arguments:
-h, --help show this help message and exit
-d, --disk-only Ignore memory, create delta from disk blocks only.
(default=False)
-p PORTS, --ports PORTS
Comma separated list of ports to forward from host to
guest. -p 80,443,8080,8443The run command is an asynchronous operation. After a few seconds, barring any errors (such as trying to launch a snapshot that doesn't exist), the client will return some information about the VM that is about to be instantiated. There will be a delay between the time the information is reported and the instance actually appears inside of virt-manager so that it can be interacted with. This is due to the snapshot being decompressed and the delta being applied to the base image which returns the VM instance to the previous memory/disk state.
$ ./nephele-client host run /root/demo.zip test
HOST host
TITLE test
SNAPSHOT /root/demo.zip
FWDPORTS None
VM should be resumed shortly; check the virt-manager window.If you are finished interacting with the VM, you can shut it down from within the OS, or by using virt-manager (either via the menu or by using the power icon).
As mentioned above, if you want to generate a snapshot, simply pause the VM in virt-manager.
If you want to continue working, but wish to migrate the VM to another nephele node, you can use the handoff command...
VM Handoff is similar to classical VM (KVM/QEMU) migration , but it is optimized for migration across WAN connected nodes rather than at typical datacenter bandwidths. While in the datacenter, we can expect 10-40Gpbs, the average WAN bandwidth in the US is on the order of 20Mbps. Thus VM handoff optimizes for shortest total migration time versus shortest downtime. We aggressively employ deduplication and compression to reduce the amount of data that needs to be shipped across the (relatively) poor links.
In order to handoff an instance, we need to supply either the title or the UUID of the running instance, as well as the destination node. If the name you supply doesn't match the title or UUID of any running instance you will get error. If your supplied title is ambigious, because there are more than one running instance with the same title, you will receive an error indicating you try the instances UUID instead. The UUID can be used by running the ps command which is outlined in the next section.
$ ./nephele-client nodeA handoff test nodeB
Handoff initiated for test [e8690455-c9e6-4c7a-9e52-e918da807eac] to the following destination: tcp://nodeB:8022Even though a handoff has been initiated, you can continue interacting with the desktop until it is suspended. The suspension takes place once the amount of new dirty state that needs to be transfered to the destination is below some threshold. At this point, the VM is stopped at the source and the final bytes are sent to the destination where they will be reassembled against the base image at the destination. This period is called the downtime window. Once the VM is resumed at the destination, you can once again interact with it via virt-manager (although it is now running on a different host).
The nephele-client <host> ps command will display details about any running instances on a particular node. The PID, title, UUID (once resumed and assigned by libvirt), start time, and a handoff url (if handoff has been initiated) are displayed for each instance.
$ ./nephele-client localhost ps
PID TITLE PORTS UUID STARTED HANDOFFURL
89268 test2 None 5d2bee02-48c6-4301-9dda-1618b040dcbd 2020-03-19 15:34:21 tcp://nodeB:8022
106642 test0 None c3d52a9a-5bd1-44f8-9763-0059b71a00dc 2020-03-19 15:21:22 None
109359 test1 9000 66ea5fdb-d9ba-43e8-afcc-6ff8d6bb2ed2 2020-03-19 15:31:37 NoneThe main nephele log file can be found at /var/nephele/nephele.log. It contains the actions from any client calls as well as the machinations of the instances running on that particular nephele node.
Individual logs pertaining to handoffs can be found in the /var/nephele/logs directory on the destination node; the filenames follow the format handoff_from_<ip>_at_<timestamp>.[log|stats]. The handoff log file is a rather verbose file that details all of the memory and disk blocks that were transferred during the handoff and how they were applied. The handoff stats file presents a synopsis of migration that includes the total migration time, amount of data sent over the wire, and a breakdown of the disk/memory state that was transferred.
Iterations: 1
Elapsed Time: 129.345255 seconds
VM Disk Size: 25600 MB
VM Memory Size: 9024 MB
Blobs Received: 292
Total MBytes Received over wire: 146.63
VM STATE (MBytes)
DIRTIED BY VALUE BY REF ZEROS
MEMORY 1876.41 514.30 397.12 964.99
DISK 186.96 77.52 101.62 7.82In the above example, the handoff of a 25GB disk/8GB RAM virtual machine took just over 2 minutes. Though there was nearly 2GB of dirty state between memory and disk, about 1/4 of it was shipped by reference as it could already be found at the destination. In addition, about 1/2 of dirty state was zero blocks which can be highly compressed before transfer. As a result, the total amount of data shipped across the wire was roughly 150MB, a small fraction of the amount of total dirty state.
Additionally, you can enable a live 'heatmap' view of the handoff by setting Const.PRODUCE_HEATMAP_IMAGES = True in configuration.py on each nephele server. This will expose a webpage at http://<node>/heatmap/index.html that will show a layout of both memory and disk regions along with some statistics. As the handoff progresses, this page on the destination host will be periodically updated to show the disk/memory regions that are being dirtied. This page uses OpenSeadragon to support zooming into more granular regions within each image (memory and disk). The more times a particular block gets dirtied, the brighter (whiter) it gets.