Notebooks/vnx-benchmark-rtt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmark Round Trip time experiment (point to point)\n",
    "This notebook will show you how to measure the round trip time between two Alveo nodes using the benchmark application with UDP as a transport protocol.\n",
    "We are going to rely on a Dask cluster to configure the local and remote Alveo cards.\n",
    "\n",
    "This notebook assumes:\n",
    "* Direct connection between the Alveo cards\n",
    "* Dask cluster is already created and running. For more information about setting up a Dask cluster visit the [Dask documentation](https://docs.dask.org/en/latest/setup.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://10.1.212.129:8786</li>\n",
       "  <li><b>Dashboard: </b><a href='http://10.1.212.129:8787/status' target='_blank'>http://10.1.212.129:8787/status</a>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>2</li>\n",
       "  <li><b>Cores: </b>32</li>\n",
       "  <li><b>Memory: </b>232.35 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.1.212.129:8786' processes=2 threads=32, memory=232.35 GB>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dask.distributed import Client\n",
    "\n",
    "client = Client(\"tcp://10.1.212.129:8786\")\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "client_info = client.scheduler_info()['workers']\n",
    "workers = []\n",
    "for cli in client_info:\n",
    "    workers.append(client_info[cli]['name'])\n",
    "\n",
    "if len(workers) != 2:\n",
    "    print(\"Configure your Dask cluster with two workers\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic remote functions\n",
    "In this part we are going to schedule a basic function to the workers to verify that we are able to pinpoint tasks to a particular worker, we are also going to grab the Alveo shell name.\n",
    "You should visually check that your xclbin file is built for the Alveo shell available on the workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Worker name: alveo4b | shell version: \"xilinx_u280_xdma_201920_3\"\n",
      "Worker name: alveo4c | shell version: \"xilinx_u280_xdma_201920_3\"\n"
     ]
    }
   ],
   "source": [
    "import platform, os\n",
    "\n",
    "def verify_workers():\n",
    "    node_name = platform.node()\n",
    "    shell_version = os.popen(\"xbutil dump | grep dsa_name\").read()\n",
    "    #match = True\n",
    "    #if 'xilinx_u280_xdma_201920_3' not in shell_version:\n",
    "    #    match = False\n",
    "    return node_name, shell_version[24:-2]\n",
    "\n",
    "worker_0 = client.submit(verify_workers ,workers=workers[0], pure=False)\n",
    "worker_1 = client.submit(verify_workers ,workers=workers[1], pure=False)\n",
    "\n",
    "worker_check = [worker_0.result(),worker_1.result()]\n",
    "\n",
    "for w in worker_check:\n",
    "    print('Worker name: {} | shell version: {}'.format(w[0],w[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Source Dask device and utilities\n",
    "\n",
    "In this section we will import the libraries and dask on pynq class which allow us to:\n",
    "\n",
    "* Download a `xclbin` file to a worker\n",
    "* Peek and poke registers\n",
    "* Allocate buffers\n",
    "* Start kernels\n",
    "\n",
    "All of these capabilities are available for both local and remote workers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "try {\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n",
       "} catch (e) {};\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from vnx_utils import *\n",
    "import pynq\n",
    "%run dask_pynq.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download xclbin to workers\n",
    "1. Create Dask device for each worker\n",
    "2. Create an overlay object for each worker, this step will download the `xclbin` file to the Alveo card"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/tools/external/anaconda/envs/pynq-dask/lib/python3.7/site-packages/distributed/worker.py:3321: UserWarning: Large object of size 60.94 MB detected in task graph: \n",
      "  (b'xclbin2\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff ... ROR_DATA_END',)\n",
      "Consider scattering large objects ahead of time\n",
      "with client.scatter to reduce scheduler burden and \n",
      "keep data on workers\n",
      "\n",
      "    future = client.submit(func, big_data)    # bad\n",
      "\n",
      "    big_future = client.scatter(big_data)     # good\n",
      "    future = client.submit(func, big_future)  # good\n",
      "  % (format_bytes(len(b)), s)\n"
     ]
    }
   ],
   "source": [
    "daskdev_w0 = DaskDevice(client, workers[0])\n",
    "daskdev_w1 = DaskDevice(client, workers[1])\n",
    "\n",
    "xclbin = '../benchmark.intf3.xilinx_u280_xdma_201920_3/vnx_benchmark_if3.xclbin'\n",
    "ol_w0 = pynq.Overlay(xclbin, device=daskdev_w0)\n",
    "ol_w1 = pynq.Overlay(xclbin, device=daskdev_w1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Link \n",
    "\n",
    "We are going to use the function `link_status` that reports if the CMAC is detecting link, which means that the physical connection\n",
    "between the two Alveo cards is established."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Link worker 0 {'cmac_link': True}; link worker 1 {'cmac_link': True}\n"
     ]
    }
   ],
   "source": [
    "print(\"Link worker 0 {}; link worker 1 {}\".format(ol_w0.cmac_1.link_status(),ol_w1.cmac_1.link_status()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure remote Alveo card\n",
    "\n",
    "1. Set up IP address and MAC address\n",
    "2. Set up connection table\n",
    "3. Launch ARP discovery\n",
    "4. Print out ARP Table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'HWaddr': '00:0a:35:02:9d:0a', 'inet addr': '192.168.0.10', 'gateway addr': '192.168.0.1', 'Mask': '255.255.255.0'}\n",
      "Position   5\tMAC address 00:0a:35:02:9d:e5\tIP address 192.168.0.5\n"
     ]
    }
   ],
   "source": [
    "print(ol_w1.networklayer_1.set_ip_address('192.168.0.10', debug=True))\n",
    "# 2\n",
    "ol_w1.networklayer_1.sockets[0] = ('192.168.0.5', 62177, 60512, True)\n",
    "ol_w1.networklayer_1.populate_socket_table()\n",
    "#3 \n",
    "ol_w1.networklayer_1.arp_discovery()\n",
    "#4\n",
    "ol_w1.networklayer_1.get_arp_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure local Alveo card\n",
    "\n",
    "1. Print out IP and MAC address\n",
    "2. Set up connection table\n",
    "3. Launch ARP discovery\n",
    "4. Print out ARP Table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'HWaddr': '00:0a:35:02:9d:e5', 'inet addr': '192.168.0.5', 'gateway addr': '192.168.0.1', 'Mask': '255.255.255.0'}\n",
      "Position  10\tMAC address 00:0a:35:02:9d:ea\tIP address 192.168.0.10\n"
     ]
    }
   ],
   "source": [
    "print(ol_w0.networklayer_1.get_network_info())\n",
    "#2\n",
    "ol_w0.networklayer_1.sockets[2] = ('192.168.0.10', 60512, 62177, True)\n",
    "ol_w0.networklayer_1.populate_socket_table()\n",
    "#3 \n",
    "ol_w0.networklayer_1.arp_discovery()\n",
    "#4\n",
    "ol_w0.networklayer_1.get_arp_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure application\n",
    "\n",
    "* Configure remote benchmark `traffic_generator_1_0` application in `LOOPBACK` mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ol_w1_tg = ol_w1.traffic_generator_1_0\n",
    "ol_w1_tg.start(TgMode.LOOPBACK, 0)# Use connection in position 0 to reflect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure local benchmark application\n",
    "This part configures the collector, in particular\n",
    "* Allocate buffers\n",
    "* Start collector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "send_packets   = 2 ** 20\n",
    "shape          = (send_packets,1)\n",
    "rtt_cycles     = pynq.allocate(shape, dtype=np.uint32, target=ol_w0.HBM0)\n",
    "pkt            = pynq.allocate(1,     dtype=np.uint32, target=ol_w0.HBM0)\n",
    "\n",
    "collector_h = ol_w0.collector_1_2.start(rtt_cycles,pkt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**This part configures the traffic generator** `traffic_generator_1_2`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "send_pkts = send_packets\n",
    "ol_w0_tg = ol_w0.traffic_generator_1_2\n",
    "ol_w0_tg.reset_stats()\n",
    "ol_w0.networklayer_1.reset_debug_stats()\n",
    "\n",
    "ol_w0_tg.start(TgMode.LATENCY, 2, send_pkts, 1, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read latency result\n",
    "* Call the dask method to synchronize the Alveo buffer with the dask buffer\n",
    "\n",
    "Note that this buffer contains the round trip time in clock cycles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PynqBuffer([[323],\n",
       "            [323],\n",
       "            [321],\n",
       "            ...,\n",
       "            [321],\n",
       "            [321],\n",
       "            [322]], dtype=uint32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rtt_cycles.sync_from_device()\n",
    "rtt_cycles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute some statistics on the results\n",
    "1. Convert the rtt from cycles to microseconds, get clock frequency by querying `.clock_dict['clock0']['frequency']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = int(ol_w1.clock_dict['clock0']['frequency'])\n",
    "rtt_usec = np.array(shape, dtype=float)\n",
    "rtt_usec= rtt_cycles / freq  # convert to microseconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Use `scipy` to compute statistical values\n",
    "    * Mean\n",
    "    * Standard deviation\n",
    "    * Mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Round trip time at application level using 1,048,576 packets\n",
      "\tmean    = 1.071 us\n",
      "\tstd_dev = 0.003780 us\n",
      "\tmode    = 1.070 us, which appears 370,039 times\n",
      "\tmax     = 1.140 us\n",
      "\tmin     = 1.060 us\n"
     ]
    }
   ],
   "source": [
    "from scipy import stats\n",
    "mean, std_dev, mode = np.mean(rtt_usec), np.std(rtt_usec), stats.mode(rtt_usec)\n",
    "print(\"Round trip time at application level using {:,} packets\".format(len(rtt_usec)))\n",
    "print(\"\\tmean    = {:.3f} us\\n\\tstd_dev = {:.6f} us\".format(mean,std_dev))\n",
    "print(\"\\tmode    = {:.3f} us, which appears {:,} times\".format(mode[0][0][0],mode[1][0][0]))\n",
    "print(\"\\tmax     = {:.3f} us\".format(np.max(rtt_usec)))\n",
    "print(\"\\tmin     = {:.3f} us\".format(np.min(rtt_usec)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Box and whisker graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'whiskers': [<matplotlib.lines.Line2D at 0x7f7b6f8cfb50>,\n",
       "  <matplotlib.lines.Line2D at 0x7f7b6f8c4490>],\n",
       " 'caps': [<matplotlib.lines.Line2D at 0x7f7b6f8c45d0>,\n",
       "  <matplotlib.lines.Line2D at 0x7f7b6f8daf90>],\n",
       " 'boxes': [<matplotlib.lines.Line2D at 0x7f7b6f8cfad0>],\n",
       " 'medians': [<matplotlib.lines.Line2D at 0x7f7b6f8eb510>],\n",
       " 'fliers': [<matplotlib.lines.Line2D at 0x7f7b6f8eba10>],\n",
       " 'means': []}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/4AAACqCAYAAAAUa5SxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXYklEQVR4nO3debSlVX3m8e9DFQoICAgCglgaMYoDaANqVBAc2ind2u0Yh2ASWaa1HbpNTLKMYpZDFFuNQzSiWKI2relGo6gRRwQDKjMoGmkaBGUUEUsKBerXf7z7mlPXc6vuUPeee3d9P2vdVeecd9i/9911T9Vz3r3fk6pCkiRJkiT1aZtJFyBJkiRJkhaPwV+SJEmSpI4Z/CVJkiRJ6pjBX5IkSZKkjhn8JUmSJEnqmMFfkiRJkqSOGfwlSdpKJFmTpJKsXuB+Ksm9Z1j23CSnzGIfa5O8YSF1zFeSryf5k0m0LUnSJBj8JUndSHJZkvVJ1iW5uoXLHduyL7TX1yW5NcmvR57/cOTx+iQbRp6vm6GtSvLLts71SU5MssvSHvHyU1Ufr6rHT7qOLWFLfVAiSdKkGfwlSb35/araETgIeDDwlwBV9cSq2rEt+zjw1qnnVbX/yLInAj8ZWbbjJto6sC2/F7ArcMxiHph+m6FckqTNM/hLkrpUVVcDX2T4AGCx27oJ+AxwwNRrSe6W5DNJbkhySZIXjSz7fJL/MfL8E0mOH7fvJIcmOSPJjUmuSvKeJHcYWV5JXtxGLfwsyXuTpC1bleRtbUTCpcCTZzqGJC9M8tmR55ck+eTI8yuSjJ7Lx87Q5lFJTm+Pk+QdSa5N8vMkFyR5wJi2d0rytSTvatvcsdX9oyTXJHl/ku3buo9OcmWSVye5GvjwmP0dleSbSd7d2v1+ksfMcNzbJHlNkstbnSckuXNb/I32541tZMfDZzp/kiQtZwZ/SVKXkuzLcPX+kiVoa1fgqcCZIy+fCFwJ3A14OvCmkfD5R8DzkxyZ5LnAIcDLZ9j97cArgd2BhwOPAf7LtHWe0vZxIPBM4N+311/Ulj0YOLjVMZNTgUe1ILw3sC3wiHZ89wJ2BC6YRZujHg8cBtwH2AV4FvDT0RWS3AX4CvDNqnpZVRXwlrbNQcC9gX2A145sthewG3AP4OgZjuehwKUM5+11wElJdhuz3lHt5wiGkRs7Au9pyw5rf+7SRn+cMUNbkiQtawZ/SVJvPp3kF8AVwLUMoW+xnJPkRuB6YD/gHwCS3B14JPDqqrqlqs4DPgg8H34zGuHFwEeAvwNeUFW/GNdAVZ1dVWdW1W1VdVlr4/Bpq/1tVd1YVT8Cvsa/jXJ4JvDOqrqiqm4A3jzTgVTVpcAv2raHM4yW+HGS+7bnp1XVhlm0OepWYCfgvkCq6uKqumpk+d0YPnD4x6p6TTt3YfjA4pVVdUM7L28Cnj2y3QbgdVX1q6paP8MhXduO/daq+gTwA8aPeHgu8PaqurSq1jFMDXm2UwgkST0x+EuSevPUqtoJeDRD4Nx9Edt6SFXtAmwHvA84Lcl2DIH2hmlh/nKGK9dTTgZWAT+oqtNnaiDJfZKcnOFmhTcxhODpx3T1yOObGa5a0+q4YloNm3Iqw3k7rD3+OkPoP7w9n02bv1FVX2W4ev5e4JokH0iy88gqTwa2B94/8toewA7A2W16w43AP7fXp1xXVbds5lh+3EYPTLmc4XxMdzc2Pi+XA6uBPTezf0mSVgyDvySpS1V1KrAWeNsStHUrwxX9ewIPAH4C7JZkp5HV9gN+PPL8jcDFwN5JnrOJ3b8P+D6wf1XtDPwVkFmWdhVw92k1bMpU8H9Ue3wqMwf/Wamqd1XVvwPuzzB8/89GFh/HEOo/n+RO7bXrgfXA/atql/Zz52k3WRwN9DPZZ+q+A81+DP0y3U8YpgyMrncbcM0s25Ekadkz+EuSevZO4HHTbkq3xSVZBbyQIbBeWlVXAP8CvDnJdkkeBPwxw7cJkOSwtv4L2s+7k+wzdufDUPmbgHVt2P2fzqG0TwIvS7Jvuw/BX2xm/VMZ5rpvX1VXAqcBTwDuApw7h3YBSHJIkocm2Rb4JXALwz0LRr2UYRj+yUm2b9MJjgPekeSubT/7JBl3D4FNuSvDsW+b5BnA/YDPj1nvROCVSe6Z4asf3wR8oqpuA65jmFZwrzm2LUnSsmLwlyR1q6quA04A/nqRmjg/yTrgZ8AfAk9rc+kBngOsYbii/CmGOelfakPdTwBeWlU/bsP8PwR8eNoV6imvAv6AYf79ccAn5lDfcQxz9c8HzgFO2tTKVfWvwDqGwD/1bQWXMtx4b3pgn42dWw0/YxhC/1OmjcBow/GPZpiS8E9tqsSrGW7KeGab3vBl4Hfn2Pa3gP0ZRhC8EXh6Vf10zHrHAx9luIP//2P4cOK/ttpubtt+s007eNgca5AkaVnIxtPfJEmSVrYkRwF/UlWPnHQtkiQtB17xlyRJkiSpYwZ/SZIkSZI65lB/SZIkSZI65hV/SZIkSZI6ZvCXJEmSJKljq+ey8u67715r1qxZpFIkSZIkSdJ8nH322ddX1R7jls0p+K9Zs4azzjpry1QlSZIkSZK2iCSXz7TMof6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLHDP6SJEmSJHXM4C9JkiRJUscM/loZjrkzSZblz2677TbpsyNJkiRJM1o96QKk2aqqSZcwVpJJlyBJkiRJM/KKvyRJkiRJHTP4S5IkSZLUMYO/NuKw9X7Yl5IkSZKg0+C//Qw3Ydt+wkFoudYlLbbl+nffuvqo606rVo2t606rVk20LknLx3J9/7Iu67Ku5VPXmr32GlvXmr32mmhdW0qXN/e7BRh3G7hJx+vlWpe02Jbr333rmpvlWtfNGzaMr2vDhiWvRdLytFzfv6xrbqxrbqxrbi6/5prxdV1zzZLXshi6vOIvSZIkSZIGm73in+Ro4GiA/fbbb9EL0uQtx7nh9bqdJ13CJi3HcyZJkiRJMIsr/lX1gao6uKoO3mOPPZaiJk1YVS27n+Vu0udnJZ4zSZIkSUvDof6SJEmSJHWsy5v7bcf4m0Nst9SFjGl/OdYlLbbl+nffuuZmuda1wzbbjL2R3w7b+Nm2pMFyff+yrrmxrrmxrrm5x557jr2R3z323HMC1Wx5XQb/9ct0mPNyrUtabMv17751zc1yreuXt98+6RIkLXPL9f3LuubGuubGuubmsquvnnQJi8rLIZIkSZIkdczgr414U7h+2JeSJEmSwOAvSZIkSVLXDP6SJEmSJHWsy5v7qU/JuPt/Tt6uu+466RIkSZIkaUYGf60Mx/ycOmbSRUiSJEnSyuNQf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpYwZ/SZIkSZI6ZvCXJEmSJKljBn9JkiRJkjpm8JckSZIkqWMGf0mSJEmSOmbwlyRJkiSpY6mq2a+cXAdcvnjlLIrdgesnXYQWzH7sh33ZB/uxD/ZjH+zHPtiPfbAf+7BS+/EeVbXHuAVzCv4rUZKzqurgSdehhbEf+2Ff9sF+7IP92Af7sQ/2Yx/sxz702I8O9ZckSZIkqWMGf0mSJEmSOrY1BP8PTLoAbRH2Yz/syz7Yj32wH/tgP/bBfuyD/diH7vqx+zn+kiRJkiRtzbaGK/6SJEmSJG21VmzwT3J8kmuTXDTD8iR5V5JLklyQ5CEjy/ZLckqSi5N8L8mapapbG5tvPyY5Isl5Iz+3JHnq0lavUQv8nXxrku+238l3JcnSVa5RC+zHtyS5qP08a+mq1nSz6Mf7Jjkjya+SvGrasick+UHr479Ymoo1zgL7cZPbaunMtx+T3D3J19q/jd9N8vKlq1rTLaAft0vy7STnt358/dJVrekW8r7alq9Kcm6Skxe/2i1rxQZ/YC3whE0sfyKwf/s5GnjfyLITgGOr6n7AocC1i1SjNm8t8+jHqvpaVR1UVQcBRwI3A6csbqnajLXMoy+T/B7wCOBBwAOAQ4DDF7NQbdJa5tePTwYeAhwEPBT4syQ7L2ql2pS1bLofbwBeBrxt9MUkq4D3MvTzAcBzkhywSDVq89Yyj36c5bZaOmuZXz/eBvz39v/VhwEv8fdxotYyv378FXBkVR3I8G/kE5I8bFEq1GysZf7vqwAvBy7ewjUtiRUb/KvqGwwdM5P/CJxQgzOBXZLs3d4wV1fVl9p+1lXVzUtQssaYbz9OW+fpwBfsx8laQF8WsB1wB+COwLbANYtdr8ZbQD8eAJxaVbdV1S+B8zF0TMzm+rGqrq2q7wC3Tlt0KHBJVV1aVb8G/hdDn2sCFtCPs/ld1hKZbz9W1VVVdU57/AuGsLHPYtaqmS2gH6uq1rWn27Yfb7I2IQt5X02yL/Bk4IOLV+HiWbHBfxb2Aa4YeX5le+0+wI1JTmrDNI5tVzi0PM3Uj6OeDZy4ZBVpvsb2ZVWdAXwNuKr9fLGqVuQnqVuJmX4nzweemGSHJLsDRwB3n0B9WpjZvOdKWmJtWuqDgW9NthLNRxsefh7DKOMvVZX9uDK9E/hzYMOkC5mPnoP/uDnCBawGHgW8imFI8b2Ao5auLM3RTP04LByuND4Q+OKSVaT5GtuXSe4N3A/YlyFgHJnksCWtTHMxth+r6hTg88C/MHwQdwbDMFWtLJt8z5W09JLsCPwf4BVVddOk69HcVdXtbXrqvsChSR4w6Zo0N0meAlxbVWdPupb56jn4X8nGV5v2BX7SXj+3DWO8Dfg0w7xULU8z9eOUZwKfqqrfGo6jZWemvnwacGabdrMO+ALDXEYtTzP+TlbVG9u9Nx7HECB/OIH6tDCbe8+VtISSbMsQ+j9eVSdNuh4tTFXdCHwdp8KtRI8A/kOSyximwR2Z5GOTLWlueg7+nwFe0O5A/TDg51V1FfAdYNcke7T1jgS+N6kitVkz9eOU5+Aw/5Vipr78EXB4ktXtPziHs0JvmrKVGNuPbRjjXQCSPIjhZo3ecHPl+Q6wf5J7JrkDw1Sqz0y4JmmrlCTAh4CLq+rtk65H85NkjyS7tMfbA48Fvj/ZqjRXVfWXVbVvVa1h+Lfxq1X1vAmXNSerJ13AfCU5EXg0sHuSK4HXMdwsg6p6P8OQ0ycBlzDc8f2Fbdnt7asZvtLeUM8GjlvyAxAw/35s265huDJ16lLWrPEW0Jf/m+EDuAsZhhT/c1V9dkmL128soB+3BU4b3la5CXheG1WlCdhcPybZCzgL2BnYkOQVwAFVdVOSlzJMn1oFHF9V353EMWjB/fhb21bVhyZwGFu9+fYjwweozwcubPPDAf6qqj6/xIcgFtSPewMfafcU2wb4ZFWtuK+C68VC3lcnVPIWkyqn7kmSJEmS1Kueh/pLkiRJkrTVM/hLkiRJktQxg78kSZIkSR0z+EuSJEmS1DGDvyRJkiRJHTP4S5K6kOT2JOcluSjJZ6e+N3mR27wsye7TXvtWq+NHSa5rj89rX0E6ffsPJ/ndWbb12pF93T7y+CXt57lb5qh+q91PJbnHHNZ/aJJ3LEYtSy3J6iQ3tsd7JfncpGuSJGk+/Do/SVIXkqyrqh3b448A/1pVb1zkNi8DDq6q68csO6ote+kM266qqtvn0eZq4PqqWooPNg4EXlNVz1jENuZ1HpbC9HOd5KPAe6rqW5OtTJKkufGKvySpR2cA+wBkcGwbCXBhkme11x+d5OSpDZK8p4X1qSv5r09yTtvmvu31uyQ5Jcm5Sf4ByGwLmrp6nOQNSb4NHJrk9CQHjSx7R2vzS0nuMod9vyHJK9rj05O8PclpSb6X5OB21f6HSY4Z2eYPk3y7jRr4+yTj/k/wXOCfptV/bKvxi+3q/qlJLk3ypLbeY5N8uj3eKclH2jm8IMlTZzgPj2t1XJjkuCR3aNsf247hgiRvaa/tmeSkJGe1+h82U1vt9ee11y5K8qZpx/K3Sc5PckaSu7Zlv5Nh1MZ3gGPY2KfbOZEkaUUx+EuSupJkFfAY4DPtpf8EHAQcCDwWODbJ3rPY1fVV9RDgfcCr2muvA06vqge3/e83x/LuDJxTVYdW1Rljlp3Z2jwD+Os57nvU+qp6FPAhhrD6YuCBwNFJdknyAOBpwO9V1UHAauDZY/bzCODsaTWe0mr8NUMwfgzwDOBvxmx/DHBdVT2Q4fyfOrKfc6rqUOB84HjgP7f1dmh17gk8Cbh/VT0IeHPb9l3AW6vqYOCZwAdnaivJvsAbgCOABwOPSPKUkRpOraoDGc73H7XX3w38XVUdAlw37XjOAh415jglSVrWDP6SpF5sn+Q84KfAbsCX2uuPBE6sqtur6hqG8HnILPZ3UvvzbGBNe3wY8DGAqvoc8LM51vhr4FMzLLsN+Mf2+GOt7vma+tDjQuDCqrqmqm4BLgP2ZfgA5BDgrHbODgd+Z8x+9mbj8Lu+qqbO64XA16vqtvZ4zZjtHwu8F6AGU+dr9DzcD/hhVf3f9vwEhvN8A7ABOC7J04Bfjuzz/a3uTwO7Jtl+hrYeCny1qq6vqluB/9n2PXUsX2iPR/v44cAn2uOPTjuea4G7jTlOSZKWtdWTLkCSpC1kfVUdlOTOwMnASxiuDs80HP82Nv4AfLtpy3/V/rydjf+9XMjNcdbXzDfXmf76QtqZqn3DyOOp56sZzsnxVbW5UQXr2fi8/HravkbbGfd/ijD+OEbPw9j+qapbkxwMPI5hNMKfAo9v6x9aVaO1kGRcW5uaijG6/Wgf1ww1w3Au1m9in5IkLUte8ZckdaWqfg68DHhVkm2BbwDPSrIqyR4MV3y/DVwOHJDkju3DgsfMYvffoM3xTvJEYNctWPq2DNMSAP4AOH0L7nu6LwPPTPtGgnbvgnHTFi4G7r2Adk4BXtraSJJx5+t7wP5J7tWeP49hmP5OwM5VdTLwSoah+lO1v2Rq4yQHbaKtM4Ej2vFNTWeYmm4wkzMZphDAb8/nvw9w0Wa2lyRp2TH4S5K6U1XnMswdfzbDkPIL2vOvAn9eVVdX1RXAJ9uyjwPnzmLXrwcOS3IOw9XnH23Bsn8OPKTt+5EMc9MXRVVdyHAsX05yAUNo3nPMqp8DHr2Apl4P7JnkIuA8xsyPr6qbgT8GTkpyIcMoguMY5uB/LslUv/23tslLGObqX5Dke8CLZmqrqq4EXgt8vb12ZpuisSkvA17Zbjy447RlRzCcE0mSVhS/zk+SpAnLEn5F31wk2QH4CvDI5fqVe0ulTSU4DXhyG1UiSdKK4RV/SZI0Vrsa/zcMN/nb2t2V4dsEDP2SpBXHK/6SJEmSJHXMK/6SJEmSJHXM4C9JkiRJUscM/pIkSZIkdczgL0mSJElSxwz+kiRJkiR1zOAvSZIkSVLH/j9uOyyRDkS2sQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1296x144 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "red_square = dict(markerfacecolor='r', marker='s')\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_title('RTT Box and whisker plot')\n",
    "ax.set_xlabel('Round Trip Time (microsecond)')\n",
    "ax.set_yticklabels([''])\n",
    "fig.set_size_inches(18, 2)\n",
    "ax.boxplot(rtt_usec, vert=False, flierprops=red_square)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Release Alveo cards\n",
    "* To release the alveo cards the pynq overlay is freed\n",
    "* Delete dask pynq-dask buffers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "del rtt_cycles\n",
    "del pkt\n",
    "pynq.Overlay.free(ol_w0)\n",
    "pynq.Overlay.free(ol_w1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------\n",
    "Copyright (c) 2020-2021, Xilinx, Inc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}