Example - MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook presents a simple use of our algorithm on the MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix seed for reproductibility\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the data from sklearn with the **associated labels**. \n",
    "\n",
    "Labels are not used for clustering but allow to easily create constraints and evaluate the performances of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, labels = load_digits(return_X_y = True)\n",
    "classes = len(np.unique(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset contains 1797 points in 10 classes\n"
     ]
    }
   ],
   "source": [
    "print(\"Dataset contains {} points in {} classes\".format(len(data), classes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the extracted labels we can create pairwise **must link** and **cannot link constraints**.  \n",
    "For real case application, labels are unknown, constraints would be obtained from experts as +1 (must link) and -1 (cannot link) constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.constraint import random_indices, generate_constraint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create *500* random constraints only on the first **1000** points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = np.arange(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices =  random_indices(train, 500)\n",
    "constraints = generate_constraint(labels, indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apply algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ran on 11 September 2020 15:06:51\n",
      "Local\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from utils.metrics import evalSplit\n",
    "from utils.clustering import Clustering\n",
    "from models.kernel_opt import kernel_clustering\n",
    "from kernels.features import produce_kernels, normalize_and_check_kernels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "performances = {} # Dictionary to store test performances for the different methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our algorithm is available in two versions: with and without approximation of kernels. Approximation of kernels allows to reduce the memory usage and complexity when the data dimensionalityis too large"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Without approximation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the different kernels we want to compute\n",
    "kernels_name = ['rbf', 'sigmoid', 'polynomial', \n",
    "                'laplacian', 'linear']\n",
    "\n",
    "# Define options on the kernels computation\n",
    "kernel_args = {\"normalize\": \"expectation\", \n",
    "               \"check_method\": \"trivial\", \n",
    "               \"clip\": True}\n",
    "\n",
    "# Compute and save kernels \n",
    "## As kernels computation is slow, the function saves it in a local folder\n",
    "names, kernels = produce_kernels('mnist', kernels_name, data, n_jobs = 8)\n",
    "names, kernels = normalize_and_check_kernels(names, kernels, classes, n_jobs = 8, **kernel_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the clustering algorithm to use\n",
    "clustering = Clustering.create(\"kmeans\", classes = classes, constraint_matrix = constraints)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "assignment = kernel_clustering(kernels, clustering, constraints)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:732: FutureWarning:The behavior of AMI will change in version 0.22. To match the behavior of 'v_measure_score', AMI will use average_method='arithmetic' by default.\n",
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:844: FutureWarning:The behavior of NMI will change in version 0.22. To match the behavior of 'v_measure_score', NMI will use average_method='arithmetic' by default.\n"
     ]
    }
   ],
   "source": [
    "performances[\"Without Approx\"] = evalSplit(assignment, labels, train)['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With approximation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "kernel_args = {\"normalize\": \"approximation\"}\n",
    "\n",
    "# Compute the kernels approximation\n",
    "names, kernels = produce_kernels('mnist', kernels_name, data, n_jobs = 8, approximation = True)\n",
    "names, kernels = normalize_and_check_kernels(names, kernels, classes, n_jobs = 8, **kernel_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "assignment = kernel_clustering(kernels, clustering, constraints, kernel_approx = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:732: FutureWarning:The behavior of AMI will change in version 0.22. To match the behavior of 'v_measure_score', AMI will use average_method='arithmetic' by default.\n",
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:844: FutureWarning:The behavior of NMI will change in version 0.22. To match the behavior of 'v_measure_score', NMI will use average_method='arithmetic' by default.\n"
     ]
    }
   ],
   "source": [
    "performances[\"With Approx\"] = evalSplit(assignment, labels, train)['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kmeans"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison, we compute a standard kmeans and evaluate it on the same test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "assignment = KMeans(classes).fit_predict(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:732: FutureWarning:The behavior of AMI will change in version 0.22. To match the behavior of 'v_measure_score', AMI will use average_method='arithmetic' by default.\n",
      " /home/vincent/miniconda3/envs/clustering/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:844: FutureWarning:The behavior of NMI will change in version 0.22. To match the behavior of 'v_measure_score', NMI will use average_method='arithmetic' by default.\n"
     ]
    }
   ],
   "source": [
    "performances[\"Kmeans\"] = evalSplit(assignment, labels, train)['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple comparison on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f144ebc3210>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAFwCAYAAABUy2nKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhV5bn38e+PRAQZnKAORAYtgwiiQFER5+qB2qOt2AMWtdoB0Vp9T9VWe/paa8+r9tQOWu1B64it4qyoqNW2ztUiFZkEBEUFRY0Ds2DI/f6xdsgmJLAh7Kyw1+9zXbnc61krK/d+TLj386xnUERgZmZmpaVF2gGYmZnZlucEb2ZmVoKc4M3MzEqQE7yZmVkJcoI3MzMrQeVpB7CpOnToEF27dk07DKqqqigv3+qqryhcF7VcF7VcF7WaQ11Mnjy5MiI6phqENamt7q+va9euvPzyy2mHQWVlJR06dEg7jGbBdVHLdVHLdVGrOdSFpLdSDcCanLvozczMSpATvJmZWQlygjczMytBTvBmZmYlyAnezMysBDnBm5mZlSAneDMzsxJU1AQvaaik2ZLmSrqwnvPbS3pI0quSZkg6vZjxmJmZZUXRErykMuBaYBjQGzhJUu86l30fmBkR/YDDgV9LalmsmMzMzLKimCvZDQLmRsQbAJLGA8cDM/OuCaCdJAFtgY+BqiLGZNagrhc+UvC18684toiRmJk1XjETfCfgnbzjBcABda65BpgAvAu0A0ZERHXdG0kaDYwGqKiooLKysigBb4rFixenHUKzUSp1sVe7KPjahn4HS6UutgTXRS3XhaWhmAle9ZTV/Rf034ApwJHAXsATkp6NiCXrfFPE9cD1AAMHDoy013Su0VziaIwt1WothbqYt7S+X9n6dbhmr/pPtOlJh+Wz1y27JLv/uJfC78WW4rqwplbMQXYLgD3yjitIWur5Tgfui8Rc4E2gVxFjMjMzy4RituAnAd0ldQMWAiOBb9a55m3gKOBZSbsAPYE3ihiTNcYl29df3qYnuNVqZtasFC3BR0SVpLOBx4Ey4KaImCFpTO78WOAXwC2SppF06f84ItJ/wG5mZraVK+p+8BExEZhYp2xs3ut3gWOKGYOZmVkWFTXBm5lt7Tx90rZWTvBmRdb31r4FXzvtW9OKGEnhnNTMtn5ei97MzKwEuQVvZraleKaJNSNO8FYUW2O3tG0mJzWzZsld9GZmZiXICd7MzKwEOcGbmZmVoEw+g/cUIDMzK3VuwZuZmZUgJ3gzM7MSlMku+k3iKUBmZrYVcgvezMysBLkFb2aWAi8GZcXmFryZmVkJcgvezJqMW61mTccteDMzsxJU1AQvaaik2ZLmSrqwnvMXSJqS+5ouaY2knYoZk5mZWRYULcFLKgOuBYYBvYGTJPXOvyYifhUR+0XEfsBFwNMR8XGxYjIzM8uKYrbgBwFzI+KNiFgNjAeO38D1JwF3FDEeMzOzzCjmILtOwDt5xwuAA+q7UNJ2wFDg7AbOjwZGA1RUVFBZWdmowPZqFwVfW9myZ73li1vvUc/FjYsrDcWqi65luxR+32ZSb66LWq6LWq4L21oVM8GrnrKG/lL+HXi+oe75iLgeuB5g4MCB0aFDh0YFNm9pfaHVr0Or2Q2fq7uSXSPjSkOx6mL+mpWF37eZ1JvropbropbrwrZWxeyiXwDkf2ytAN5t4NqRuHvezMxsiylmC34S0F1SN2AhSRL/Zt2LJG0PHAacXMRYmoTn+JqZWXNRtAQfEVWSzgYeB8qAmyJihqQxufNjc5d+HfhLRCwvVixmZmZZU9SV7CJiIjCxTtnYOse3ALcUMw4zM7Os8Up2ZmZmJcgJ3szMrAQ5wZuZmZUgJ3gzM7MS5ARvZmZWgpzgzczMSpATvJmZWQlygjczMytBTvBmZmYlyAnezMysBDnBm5mZlSAneDMzsxLkBG9mZlaCnODNzMxKkBO8mZlZCXKCNzMzK0FFTfCShkqaLWmupAsbuOZwSVMkzZD0dDHjMTMzy4ryYt1YUhlwLXA0sACYJGlCRMzMu2YH4A/A0Ih4W9IXihWPmZlZlhSzBT8ImBsRb0TEamA8cHyda74J3BcRbwNExAdFjMfMzCwzitaCBzoB7+QdLwAOqHNND2AbSU8B7YCrImJc3RtJGg2MBqioqKCysrJRge3VLgq+trJlz3rLF7feY72yrmW7FH7fRr6HLcV1Uct1Uct1Uct1YVurYiZ41VNW9y+lHBgAHAW0Bv4h6cWImLPON0VcD1wPMHDgwOjQoUOjApu3tL7Q6teh1eyGzy1f99z8NSsLv28j38OW4rqo5bqo5bqo5bqwrVUxE/wCIP9jawXwbj3XVEbEcmC5pGeAfsAczMzMbLMV8xn8JKC7pG6SWgIjgQl1rnkQOERSuaTtSLrwXytiTGZmZplQtBZ8RFRJOht4HCgDboqIGZLG5M6PjYjXJD0GTAWqgRsiYnqxYjIzM8uKYnbRExETgYl1ysbWOf4V8KtixmFmZpY1XsnOzMysBDnBm5mZlSAneDMzsxLkBG9mZlaCnODNzMxKkBO8mZlZCXKCNzMzK0FO8GZmZiXICd7MzKwEOcGbmZmVICd4MzOzEuQEb2ZmVoKc4M3MzEqQE7yZmVkJcoI3MzMrQU7wZmZmJag87QDMzCwdkydP/kJ5efkNQB/c4NvaVAPTq6qqvjtgwIAP6rugqAle0lDgKqAMuCEirqhz/nDgQeDNXNF9EXFpMWMyM7NEeXn5DbvuuuveHTt2/KRFixaRdjxWuOrqan344Ye9Fy1adANwXH3XFC3BSyoDrgWOBhYAkyRNiIiZdS59NiK+Wqw4zMysQX2c3LdOLVq0iI4dOy5etGhRnwavKeLPHwTMjYg3ImI1MB44vog/z8zMNk0LJ/etV+7/XYN5vJhd9J2Ad/KOFwAH1HPdQZJeBd4Fzo+IGXUvkDQaGA1QUVFBZWVlowLbq13hv8+VLXvWW7649R7rlXUt26Xw+zbyPWwprotarotarotargvbWhUzwauesrp/Kf8CukTEMklfAR4Auq/3TRHXA9cDDBw4MDp06NCowOYtrS+0+nVoNbvhc8vXPTd/zcrC79vI97CluC5quS5quS5qZakuul74yIAteb/5Vxw7eUPnv/Od7+zRpUuXVRdffPEHAEOGDOneqVOn1XfeeedbAN/73vcqOnXq9Hn37t0/mzFjRuvLLrts0W233bZD7969PxswYMBnAIMGDep55ZVXvnPooYeuaGy8F1544a5XXHHFoobOP//8862HDBnS+5577nl9+PDhSxr784qtmF30C4D8j60VJK30tSJiSUQsy72eCGwjqXn8VZuZWVEdfPDBy1588cW2AGvWrOGTTz4pnz17duua85MmTWp76KGHLhs1atTiyy67bBHAAw88sMPUqVNbN3TPxrj66qt329D52267bef+/fsvu/3223faUj/z888/31K3Wk8xE/wkoLukbpJaAiOBCfkXSNpVknKvB+Xi+aiIMZmZWTNx5JFHLps8eXJbgMmTJ7fu2bPnyjZt2qz58MMPy1auXKl58+a1Gjx48Iqrr75651NPPbXzE0880ebJJ5/c4ac//WlFr169es+YMWNbgDvuuGPHvn377t21a9c+jz32WFuAFStW6MQTT+zao0eP3nvvvXfvhx56qB1Azb1qYjjiiCO++PDDD7c766yzOq1atapFr169eh933HHd6sZaXV3Nww8/vOO4cePmP/vss+1XrFghgNmzZ7fs1q3bPieccELXHj169B46dOieS5cubQHQqVOnvmeeeWanvn377t23b9+9p0+fvi3A8OHDu373u9+tOOCAA3qcddZZFS+88ELrfv369erRo0fvo48+eq8PP/ywbM6cOS27dOnS57333itfs2YNAwYM6Hnfffe135T6LVqCj4gq4GzgceA14K6ImCFpjKQxuctOBKbnnsFfDYyMCA/4MDPLgK5du35eXl4er7/+esunn366zYEHHrh84MCBy//2t7+1ffbZZ7fr2bPnylatWq3NCUcfffTyL3/5y5/+93//94JZs2bN3GeffVYBVFVVadq0aa/98pe/fOfSSy/dHeCXv/zlFwDmzJkz8/bbb39j9OjRXWuScn3+8Ic/LNx2222rZ82aNXPChAlv1j3/xBNPtN1jjz1W7bPPPqsOOOCApXfffff2Nefmz5/fasyYMR/OmTNnZrt27ap/9atfdaw51759+zXTpk177YwzzvjgBz/4wdpe7Xnz5rV6/vnn5/zxj39ccNppp3W77LLLFsyZM2fmPvvss/LHP/7x7j169Fh97rnnLjr99NM7X3LJJbv07NnzsxNOOGGTHgsUdWGDiJgYET0iYq+I+H+5srERMTb3+pqI2Cci+kXEgRHxQjHjMTOz5mXAgAHL/v73v7f5xz/+0faQQw5ZNnjw4OXPP/98m2effbbtoEGDlhVyj2984xufAAwePHj5ggULWgK88MILbU899dSPAPbff//Pdt9999XTpk1rtblx/ulPf9rpxBNP/Bhg5MiRH48fP35tN/2uu+66+phjjlkOcMopp3z0wgsvtK05961vfetjgO9973sfv/LKK2vLTzjhhE/Ky8v56KOPypYuXVp27LHHLstd91HNY4sf/vCHlcuWLSu75ZZbOl577bX5g9YL4pXszMwsNQcddNCyF154oe2sWbNaf+lLX1q55557rv7d7363S9u2bdecfvrpBU0fqGnll5eXs2bNGgE01BlcXl4e1dXVa49XrVq10YZuVVUVjz766I5PPPHEDr/5zW92iwg+/fTT8k8++aQFQO5J81r5xy1atMgvXxtU27Ztq9mIpUuXtli0aFFLgCVLlpTtuOOOG/2efF6a0MzMUnPYYYcte/LJJ3fYYYcd1pSXl7PLLrusWbJkSdkrr7zS9ogjjlhe9/q2bduuWbJkyUZz15AhQ5b96U9/2glg6tSp27733nst991338/22muv1TNmzNhuzZo1zJ07d5upU6e2qfme8vLyWLVq1Xrd+A8++GD7Xr16rVi0aNHUhQsXTnv33XenDR069JPbb799B4D33nuv5ZNPPtkG4Pbbb99p8ODBa3sexo0btxPAjTfeuOP++++/3vvZeeed17Rv335NzdiBG2+8ceeDDjpoGcDZZ5/d6cQTT/zoJz/5ybunnXZal43X5rrcgjczM2Dj09qKYdCgQSs//fTT8hNOOGHtAOtevXqtXL58edluu+1WVff6UaNGfXzmmWd2HTt27C733HPPvIbu+6Mf/eiDU045pUuPHj16l5WVcd11181v3bp1HH300cuuvfbaVT179tynZ8+eK3v37r0i794f7r333r379OmzIv85/O23377Tcccd92n+/YcPH/7Jdddd94Uvf/nLy/bcc8/Pbrrppp3POuusLt26dVt1/vnnf1hz3apVq7Tvvvv2qq6u1vjx49+oL9abb775zTPPPLPLOeec06Jz586r7rjjjvmPPPJI2ylTprS58cYbZ5WXl3P//ffveNVVV+187rnnFjwQveAEL6lNRKz36cPMzGxzlZeXs2zZslfyy+699975+cfnnHPOR+RmWB1zzDHL582bt3ZBtH/+859rFxjYbbfdqhYuXDgNYLvttou694Gky7y+QXQA//u//7sQWFi3vL77jBo1avGoUaMWz549u2WLFi24/fbb367vnueff/6Hv/71r9/b0P0GDx688tVXX52VX3bssccuO/bYY9eW/eUvf2nww0xDNtrNIWmwpJkkI+GR1E/SHzb1B5mZmVnTKaQF/1vg38jNYY+IVyUdWtSozMzMtgI9e/Zc/frrr6+3xDpATW9CWgoaZBcRdYfnrylCLGZmZraFFNKCf0fSYCByK9KdQ6673szMzJqnQlrwY4Dvk+wOtwDYL3dsZmZmzdRGW/ARUQmMaoJYzMzMbAvZaIKXdDPrb/NKRHy7KBGZmVk6Ltl+i24XyyWLm9V2sT//+c+/cNlll1W8++67r+68884lP5askGfwD+e9bgV8nTrbvpqZmW2qgw8+eNk999yzI/BBzXaxy5YtK6s5P2nSpLYnnXTSO0ceeeRyYDEk28VWVVUtrknwm+Kee+7ZuU+fPsv//Oc/75CbW99oVVVVlJc3zzXjNvoMPiLuzfv6M/AfQJ/ih2ZmZqWsmNvF1jVjxoxtV6xY0eLSSy9deNddd63dKObqq6/e+aijjtrrkEMO6d61a9c+55133m6w8W1gzz///N0GDBjQ86abbtrxuuuu26lHjx69u3fvvs+ZZ57ZCWDcuHE7DB48uEd1dTVvvfXWNl27du3z9ttvN+kngc1Zi7470HmjV5mZmW1AMbeLrevWW2/d6YQTTvh46NChy958881WCxcuXJtsp06d2ubuu+9+Y/r06TMmTJiw0zPPPLMdbHgb2FatWlVPnjx59jHHHLPskksu6fTUU0/NmTlz5oxXXnmlzW233bbDqaee+mnHjh0/v+KKKzqedtppXS666KJ3O3fuvN7Su8VUyEp2SyUtqfkv8BDw4+KHZmZmpa5Y28XWdf/99+906qmnflxWVsawYcM+GTdu3I4154YMGbJk1113XdO2bds49thjP3nqqafawoa3gT311FM/AXjuuefaHHjggUt33333qm222YYRI0Z8/PTTT7cFuOGGG97+3e9+t1vLli3jjDPO+Hhz62hzFTKKvl1TBGJmZtlTrO1i87300kut33rrrW2HDh3aA+Dzzz/XHnvsseqiiy76EBre7nVD28C2a9euGhrelhZg/vz527Ro0YLKysryNWvWUFZW1uC1xdBgC15S/w19NWWQZmZWmoq1XWy+cePG7XTeeee9u3DhwmkLFy6c9sEHH0xdtGhRyzlz5rQEeO6559q///77ZcuWLdPEiRN3OOyww5bBhreBrXHooYcuf+mll9q999575VVVVdx99907HX744cs+//xzTj/99G633HLLG927d//s5z//+S6bV0Obb0Mt+F9v4FwAR27s5pKGAlcBZcANEXFFA9d9CXgRGBER92zsvmZmVgQbmdZWDMXaLjbfAw88sNPDDz/8en7ZsGHDPrn11lt32mWXXT4fOHDgshEjRnSbP39+q+HDh3906KGHrpg9e3bLDW0DW6NLly6fX3zxxQsPO+ywHhGho446avHJJ5/86fnnn7/bgQceuHTo0KHLDjjggBX9+/ff+2tf+9ri/v37b/Lo/83VYIKPiCMac2NJZcC1wNEkK+BNkjQhImbWc90vgccb8/PMzGzrU6ztYvPVV3bDDTcsgGQUfYcOHarGjRu33navDW0DW/d+Y8aM+XjMmDHrPGO/8sor124Ru+OOO1a/+eab9W5IU0wFDdmX1AfoTTIPHoCIGLeRbxsEzI2IN3L3GA8cD8ysc90PgHuBLxUYs5mZmW1EISvZ/Qw4nCTBTwSGAc8BG0vwnYD8XegWAAfUuXcnkoVzjmQDCV7SaGA0QEVFBZWVBY27aNBe7RoeFFFXZcue9ZYvbr3HemVdywp/xNLY97CluC5quS5quS5quS5KW37vQL4NbQO7tSikBX8i0A94JSJOl7QLcEMB37feSEbWX/L2d8CPI2JN3dGK63xTxPXA9QADBw6MDh06FPDjGzZvacM/q64OrWY3fG75uufmr1lZ+H0b+R62FNdFLddFLddFLdeFba0KSfCfRUS1pCpJ7YEPgD0L+L4FQP7H1grWX+J2IDA+l9w7AF+RVBURDxRwfzMzM2tAgwle0jXAHcA/Je0A/BGYDCwD/lnAvScB3SV1AxYCI4Fv5l8QEd3yft4twMNO7mZmZo23oRb868CVwO4kSf0OkhHx7SNi6sZuHBFVks4mGR1fBtwUETMkjcmdH9vY4M3MzKx+G5omdxVwlaQuJK3vm0lG0d8haWVEvN7Q9+bdYyLJwLz8snoTe0Sctglxm5nZFtb31r5bdLvYad+attF59dttt93+K1aseAXgzjvv3P5HP/rRHk8++eSc7t27r96SsWRRIbvJvRURv4yI/Um62L8OzCp6ZGZmlhkPPvhgu/PPP3+PiRMnvu7kvmUUstnMNpL+XdKfgUeBOcDwokdmZmaZ8Nhjj7X9/ve/33XChAlza3aIGz58eNdRo0Z1PuCAA3pUVFT0feSRR9p+4xvf6LrnnnvuM3z48K4133vfffe132+//Xr17t1772HDhu25ePHiFgDnn3/+bn369Nm7e/fu+5x00kldqqurARg0aFDPM888s1Pd7WVffvnlVn379t27V69evXv06NF72rRp2zZ9TWxZG1qL/mhJN5GMhh9N0tW+V0SM8EA4MzPbElavXq0RI0Z88d577527//77r7OM6+LFi8v/8Y9/zLniiiveGTFiRPcLLrjg/ddff33GrFmzWr/wwgut33vvvfLLLrtst2eeeWbOzJkzX+vfv/+KX/ziF7sAXHDBBR9Mnz79tddff33GypUrW4wfP377mvvWt73s73//+45nnXXW+7NmzZo5derU17p167bV9yJsaJDdT4DbgfMjosm3uTMzs9K3zTbbRP/+/ZeNHTu2wwEHHJC/OBrHHnvspy1atKB///4rdt55588HDRq0EqBHjx4r582bt+1bb73Vct68ea0GDRrUC5Jd4gYMGLAM4NFHH233m9/8ZtfPPvusxaefflreu3fvlcBiWHd72QsuuKAlwEEHHbT8yiuv3G3BggUtR44c+Unfvn1XNWE1FEWDLfiIOCIi/ujkbmZmxSKJCRMmvDFlypQ2F1544a7552q2gS0rK6Nly5ZrF0pr0aIFVVVVigiGDBmyZNasWTNnzZo1c968eTPuuuuut1asWKHzzjuvy3333Tdvzpw5M08++eTKzz77rEXd++ZvLztmzJiPH3zwwbmtW7euHjZsWI8JEyZs9Vulb9KWe2ZmZltau3btqh977LHX77nnnp1/+9vfFrxs3+GHH7785Zdfbjt9+vRtAZYuXdpi6tSp265YsaIFwK677lq1ePHiFg899NCOG7vXzJkzW+69996rfvrTn35wzDHHfDplypTWm/+OmoeCNpsxM7PSV8i0tmLZZZdd1jz22GNzDjvssF4dO3Zcb5vY+uy+++5V11133fyRI0fuuXr1agH87Gc/W7jvvvsuHjVq1Ie9e/fep6KiYnW/fv3W21e+rttuu22nu+++e+fy8vLo2LHj55dffnndlVe3Ok7wZmaWmpo58ABf/OIXP6/ZivXkk0/+tKa87sYv+dvJHnfccUuPO+641+re9+qrr3736quvXi9JN7S97OWXX77o8ssvX7QF3lKz4S56MzOzEuQEb2ZmVoKc4M3Msqu6urq68P1wrVnJ/b+rbui8E7yZWXZN//DDD7d3kt/6VFdX68MPP9wemN7QNR5kZ2aWUVVVVd9dtGjRDYsWLeqDG3xbm2pgelVV1XcbusAJ3swsowYMGPABcFzacVhx+BObmZlZCXKCNzMzK0FFTfCShkqaLWmupAvrOX+8pKmSpkh6WdKQYsZjZmaWFUV7Bi+pDLgWOJpky9lJkiZExMy8y/4KTIiIkLQvcBfQq1gxmZmZZUUxW/CDgLkR8UZErAbGA8fnXxARyyKiZoegNkBgZmZmjVbMBN8JyN/bd0GubB2Svi5pFvAI8O0ixmNmZpYZxZwmV9/CCeu10CPifuB+SYcCvwC+vN6NpNHAaICKigoqKysbFdhe7QrvKKhs2bPe8sWt91ivrGvZLoXft5HvYUtxXdRyXdRyXdRyXdjWqpgJfgGQ/1tdATS4/V5EPCNpL0kdIqKyzrnrgesBBg4cGB06FLxdcL3mLS180aYOrWY3fG75uufmr1lZ+H0b+R62FNdFLddFLddFLdeFba2K2UU/CeguqZuklsBIYEL+BZK+KEm51/2BlsBHRYzJzMwsE4rWgo+IKklnA48DZcBNETFD0pjc+bHAcOBUSZ8DK4EReYPuzMzMbDMVdanaiJgITKxTNjbv9S+BXxYzBjMzsyzySnZmZmYlyAnezMysBDnBm5mZlSAneDMzsxLkBG9mZlaCnODNzMxKkBO8mZlZCXKCNzMzK0FO8GZmZiXICd7MzKwEOcGbmZmVICd4MzOzEuQEb2ZmVoKc4M3MzEqQE7yZmVkJcoI3MzMrQU7wZmZmJaioCV7SUEmzJc2VdGE950dJmpr7ekFSv2LGY2ZmlhVFS/CSyoBrgWFAb+AkSb3rXPYmcFhE7Av8Ari+WPGYmZllSTFb8IOAuRHxRkSsBsYDx+dfEBEvRMQnucMXgYoixmNmZpYZ5UW8dyfgnbzjBcABG7j+O8Cj9Z2QNBoYDVBRUUFlZWWjAturXRR8bWXLnvWWL269x3plXct2Kfy+jXwPW4rropbropbropbrwrZWxUzwqqes3r8USUeQJPgh9Z2PiOvJdd8PHDgwOnTo0KjA5i2tL7T6dWg1u+Fzy9c9N3/NysLv28j3sKW4Lmq5Lmq5Lmq5LmxrVcwEvwDI/9haAbxb9yJJ+wI3AMMi4qMixmNmZpYZxXwGPwnoLqmbpJbASGBC/gWSOgP3AadExJwixmJmZpYpRWvBR0SVpLOBx4Ey4KaImCFpTO78WOBiYGfgD5IAqiJiYLFiMjMzy4pidtETEROBiXXKxua9/i7w3WLGYGZmlkVeyc7MzKwEOcGbmZmVICd4MzOzEuQEb2ZmVoKc4M3MzEqQE7yZmVkJcoI3MzMrQU7wZmZmJcgJ3szMrAQ5wZuZmZUgJ3gzM7MS5ARvZmZWgpzgzczMSpATvJmZWQlygjczMytBTvBmZmYlyAnezMysBBU1wUsaKmm2pLmSLqznfC9J/5C0StL5xYzFzMwsS8qLdWNJZcC1wNHAAmCSpAkRMTPvso+Bc4CvFSsOMzOzLCpmC34QMDci3oiI1cB44Pj8CyLig4iYBHxexDjMzMwyp2gteKAT8E7e8QLggM25kaTRwGiAiooKKisrGxXYXu2i4GsrW/ast3xx6z3WK+tatkvh923ke9hSXBe1XBe1XBe1XBe2tSpmglc9ZYX/peR/U8T1wPUAAwcOjA4dOjQmLuYtrS+0+nVoNbvhc8vXPTd/zcrC79vI97CluHZApq8AABqbSURBVC5quS5quS5quS5sa1XMLvoFQP7H1grg3SL+PDMzM8spZoKfBHSX1E1SS2AkMKGIP8/MzMxyitZFHxFVks4GHgfKgJsiYoakMbnzYyXtCrwMtAeqJf0foHdELClWXGZmZllQzGfwRMREYGKdsrF5rxeRdN2bmZnZFuSV7MzMzEqQE7yZmVkJcoI3MzMrQU7wZmZmJcgJ3szMrAQ5wZuZmZUgJ3gzM7MS5ARvZmZWgpzgzczMSpATvJmZWQlygjczMytBTvBmZmYlyAnezMysBDnBm5mZlSAneDMzsxLkBG9mZlaCnODNzMxKUFETvKShkmZLmivpwnrOS9LVufNTJfUvZjxmZmZZUbQEL6kMuBYYBvQGTpLUu85lw4Duua/RwP8WKx4zM7MsKWYLfhAwNyLeiIjVwHjg+DrXHA+Mi8SLwA6SditiTGZmZpmgiCjOjaUTgaER8d3c8SnAARFxdt41DwNXRMRzueO/Aj+OiJfr3Gs0SQsfoCcwuyhBb5oOQGXaQTQTrotarotarotazaEuukREx5RjsCZUXsR7q56yup8mCrmGiLgeuH5LBLWlSHo5IgamHUdz4Lqo5bqo5bqo5bqwNBSzi34BsEfecQXw7mZcY2ZmZpuomAl+EtBdUjdJLYGRwIQ610wATs2Npj8QWBwR7xUxJjMzs0woWhd9RFRJOht4HCgDboqIGZLG5M6PBSYCXwHmAiuA04sVTxE0q0cGKXNd1HJd1HJd1HJdWJMr2iA7MzMzS49XsjMzMytBTvBmZmYlyAnezMysBBVzHnxJkPQQ9czNrxERxzVhOM2GpOOAQ3OHT0fEQ2nGkyZJ/YBDcofPRsSracaTFkkHA1MiYrmkk4H+wFUR8VbKoZllklvwG3cl8GvgTWAl8Mfc1zJgeopxpUbS5cC5wMzc1zm5ssyRdC7wZ+ALua8/SfpBulGl5n+BFbkPPD8C3gLGpRtSeiTtIulGSY/mjntL+k7acVl2eBR9gSQ9ExGHbqwsCyRNBfaLiOrccRnwSkTsm25kTS9XFwdFxPLccRvgHxmti39FRH9JFwMLI+LGmrK0Y0tDLrHfDPxXRPSTVE7yd9I35dAsI9yCL1xHSXvWHEjqBmR5Xecd8l5vn1oU6ROwJu94DfUvwZwFSyVdBJwMPJL74LdNyjGlqUNE3AVUQ7I2COv+rpgVlZ/BF+4/gackvZE77gqckV44qboceEXS30mS2aHARemGlJqbgZck3Z87/hpwY4rxpGkE8E3gOxGxSFJn4Fcpx5Sm5ZJ2JjeGp2a1znRDsixxF/0mkLQt0Ct3OCsiVqUZT1OTdHBEPJ+rh52AL5Ek+JciYlG60TUtSd0i4s3c6/7AEJK6eCYiXkk1uJRI+jbJIMPX046lOcj9Xvwe6EMyXqcjcGJETE01MMsMJ/hNIGkwSct9bc9HRGRmEJGkyRExIMvPVWvk1cVfI+KotONpDiRdSvJBpwswGXiW5ANP5mYV5B5PnEOS4HuSfPibHRGfpxqYZYoTfIEk3QbsBUyh9jlaRMQ56UXVtCS9CLxGsn/AnXXPZ6wuXgEeAL4L/Lbu+Yj4TZMH1UxIag18Dzgf6BQRZSmHlApJT0XE4WnHYdnlZ/CFGwj0jmx/Ivoq8GXgSJIWWpaNJHneXg60SzmWZkHST4GDgbbAKyQJ/tlUg0rX85KuIfkwvLymMCL+lV5IliVuwRdI0t3AOd7ONlnYJYvdrvWRNCwiHk07juZA0r+AKuAR4GngxYj4LN2o0pMbhFpXRMSRTR6MZZITfIFyf6z7Af8E1g6uy+JKdpI6knTBdmXd8QjfTiumtOQGHA5n/bq4NK2Y0iSpHclz+CHAfwDvR8SQdKMyyyZ30RfukrQDaEYeJOl6fRLP632QZOrTZPI++GWRpD4kS/YeRvJI6x0y3EUvaXvgZ+Qt6QxcGhGeKmdNwi1422SSpkTEfmnH0RxImh4RfdKOozmQ9AjwDElSn5T1EeOS7iWZHndrrugUoF9EnJBeVJYlTvAFyi1S8Xtgb6AlUAYsj4j2qQaWAkn/DbwQERPTjiVtkq4Hfh8R09KOpTmQ1BLokTvM9LSw+j4I+8OxNSUvVVu4a4CTgNeB1iTTo65JNaL0nAs8LGmlpCWSlkpaknZQKRkCTJY0W9JUSdNy69NnjqTDSP4+rgX+AMyRlLm9GvKslLR2/EFut72VKcZjGeNn8JsgIuZKKouINcDNkl5IO6Y0RISnhdUalnYAzchvgGMiYjaApB7AHcCAVKNKz5nArbln8QCfAKelF45ljRN84Vbkuh+nSPof4D2gTcoxNanc0psNytL8Xkk75V4uTTWQ5mWbmuQOEBFzJGV2s5mImAL0k9Q+d5zVXi5LiZ/BF0hSF+B9kufv/0myg9q1ETEv1cCaUAPzemtkan6vpDdJNhGpb+e4iIg96ykvaZJuIqmT23JFo4DyiDg9vajSI+ky4H8i4tPc8Y7AeRHx03Qjs6xwgt9MuT/WsyLi/6Udi1lzkFsT4PvkbbwD/CFrmzLVkPRKROxfpyzz+zhY03EX/UZI2gP4v8DuJGuP3w78gmTKyx0phmbWrOQS+W9yXwZlkrat+YCTW6N/25Rjsgxxgt+4cSQLVNwLDAVeBGYA+2Zti1Sz+kiaRm7P8/pExL5NGE5z8ifgr5JuJqmfb1M7J96s6NxFvxGSXo2IfnnH7wOds9rtaFZXbnxKgyLiraaKpbmRNJRkgyaAJyLi8TTjsWxxC74AueftNYOpFgHbSWoDEBEfpxZYE/Mo+lp5o+jrlaXfiywn8I2JiMckTSJZrrYy7XgsW9yC3whJ84FqPFrao+jzeBR9LUlLqb+LXiR1kanVHiU9DFwYEdMl7Qb8C3gZ2Au4PiJ+l2qAlhlO8GZmW5CkGRGxT+71T4BeEXFqbqe95zM8JsGamLvobbPkdg7rDbSqKYuIcelFlJ7cI5zurFsXz6QXUbokfYF16+LtFMNJQ/76+0cBfwSIiKWSqtMJybLICd42maSfAYeTJPiJJMu1Pkcy4yBTJH2XZG3+CmAKcCDwDyAzjytqSDoO+DXJlNIPgC7Aa8A+acaVgnck/QBYAPQHHoO10+Qyu7KfNT1vNmOb40SSlsmi3Cpl/cju/N5zgS8Bb0XEEcD+wIfphpSaX5B8wJkTEd1IfkeeTzekVHyH5EPNacCImpXsSOrm5rSCsuxxC34jPFq6XisjolpSVW6d7Q+AzAwqq+OziPhMErlFTWZJ6pl2UCn5PCI+ktRCUouI+LukX6YdVApaRcSYuoUR8XdgQwNVzbYoJ/iNm0ztaOnOJDtCCdgBeBvoll5oqXlZ0g4kzxYnA8uAf6YbUmoW5OriAeAJSZ8A76YcU1o+ldSWZInaP0v6AKhKOaY0PEDSNY+keyNieMrxWEZ5FH2BJI0FJkTExNzxMODLEXFeupGlS1JXoH1EZHIP9Hy5/dC3Bx6LiNVpx9PUcmtDfEbyAXgUSV38OSI+SjWwJpa/Bn1969GbNRUn+AJJmhwRA+qUvRwRA9OKKS2SDq2vPIsjxyV1rq88gyPHLSd/QxlvLmNpcoIvkKTHgWdJ1pcO4GTg0Ij4t1QDS4Gkh/IOWwGDgMlZWuimRt467CKpi27A7Jp50FlQz0I3orZOsrjQzRpgOcn7bw2sqDlFBuvD0uNn8IU7CfgZcD/JP17P5MoyJyL+Pf84t+Pe/6QUTqoiom/+cW453zNSCictfwV2Be4Dxme99yIiytKOwQzcgt9kktpGxLK042hOJAmYWjfZZVUWu2UlbQ+cAIwk6cm4kyTZZ3GWiVmz4BZ8gSQNBm4A2gKdJfUDzoiIs9KNrOlJ+j21XbItgP2AV9OLKD2Sfph32IJk9HTm5sFHxGLgZkm3AiOA35Mkeu8Nb5YSJ/jC/Rb4N2ACQES82tBgswx4Oe91FXBHRGRxQROAdnmvq4BHgHtTiiU1uQ/AJwGHkKxq+PWIeDbdqMyyzQl+E0TEO0lv9Fpr0oolZTtExFX5BZLOrVuWETMj4u78AknfAO5u4PqSk9tx8VNgPDCa3Nz3mu2Fs7SNsFlz4mfwBZJ0D0l34zUkS06eAwyMiJGpBpaC+p4xZ3W+bwN1kaln8JKeovaRTd0tdDO1jbBZc+IWfOHGAFcBnUg2kfgLkKnn75JOAr4JdJM0Ie9UOyBri5kMA74CdJJ0dd6p9mRs9baIODztGMxsfU7whesZEaPyCyQdTLY203gBeA/oQLJrWI2lQNZWsnuXZCzCcSTL9dZYCvxnKhGZmeVxF32B3BVr9ZFUHhGZarGb2dbBLfiNkHQQMBjoWGdKVHsgkwta1Fm5rCXJHtfLM7pC1+uS1vuUHBFZ3V3PzJoJJ/iNa0ky972cdadELSHZFz1zIiK/HpD0NZLlarMofy+CVsA3gA1uMVyqco+spkTEckknk6wJcFVEvJVyaGaZ5C76AknqUvMPlaQWQNuIWJJyWM2GpBcj4sC042gOJD0XEUPSjqOpSZoK9AP2BW4DbgROiIjDUg3MLKPcgi/c5ZLGkMx9nwxsL+k3EfGrlONqcpJOyDtsQdKKzeQnxZq53jk1ddGugctLXVVEhKTjSVruN0r6VtpBmWWVE3zhekfEEkmjgInAj0kSfeYSPJC/2UwVMB84Pp1QUpc/m6CmLv4jnVBSt1TSRcApwCGSykjGZ5hZCpzgC7eNpG2ArwHXRMTn9Q2uyoKIOD3tGJqLiDgi7RiakREk6yR8OyIWSepMNj8AmzULTvCFu46kdfYq8IykLiQD7TKjzoIu64mIc5oqlrTVmVGxnojI3CYruaR+L9A9V1RJsr2ymaXAg+waIWtzoCWtBqYDd5Es9LLOwvwRcWsacaVBUjUwBXgUWMX6dfHzNOJKk6TvkaxFv1NE7CWpOzA2Io5KOTSzTHILvkCSLm7g1KVNGki6diOZBjaC5HnzncC9EfFJqlGloz/J3ufHkozFuAP4a2T7E/P3SaZLvgQQEa9L+kK6IZllV4u0A9iKLM/7WgMMA7qmGVBTi4iPImJs7rnzacAOwAxJp6QbWdOLiCkRcWFE7EcyHex4YKak41IOLU2rImJ1zYGkcjI6u8KsOXALvkARkT9aGklXktsbPmtyU8NOAo4m6aKevOHvKF2SOgL7A31JNiH6IN2IUvW0pJ8ArSUdTbIZ00Mpx2SWWX4Gv5kk7Qj8MyK6b/TiEiHp58BXgddI9v5+LEtjEPJJOp3kUUUr4B7grojIcnKvWQDqO8AxJGMSHo+IP6YblVl2OcEXSNI0arsby4COwKURcU16UTWt3MCyN4CVuaKa+hDJvt/7phJYCnJ1MQ14O1e0zh9SRGSuq17SpRFxcd5xGTCu7i6MZtY03EVfuK/mva4C3s9g67Vb2gE0I57/vr7Oki6KiMsltQTuBl5JOyizrHILfiMktc+tYFffBiIBLImINU0dl1lzI0nAn0l6No4AHo2I36YblVl2OcFvhKSHI+Krkt4kSeiqc0lb4I8R8ZOmj84sfXXW49+GZFGo50lmFxAR/0ojLrOsc4JvpNxzxukRsXfasZilQdLfN3A6IuLIJgvGzNZygt+IOq2T9WStdZL7QHNrRJycdixmZtYwD7LbuJr5761ItgJ9laSbfl+SFbsyte93RKyR1FFSy/xFTbJG0kNsYBGXLI2i97r8Zs2TE/xG1OwWJmk8MDoipuWO+wDnpxlbiuYDz0uaQLKyH5C5f8ivzP33BGBX4E+545NI6idL2qUdgJmtz130BZI0Jbcs6QbLskDSz+orz+gGK89ExKEbK8sCSa0i4rO04zCzhFvwhXtN0g0kLbUATiZZ0S1zahK5pDYRsXxj15e4jpL2jIg3ACR1I1kEKYumS3ofeBZ4Bng+IhanHJNZZrkFXyBJrYAzgZqW2TPAHyJiVXpRpUPSQSRToNpGRGdJ/YAzIuKslENrcpKGAteTrPAHyQZEZ0TE46kFlSJJnYFDgIOBrwCfZrGXy6w5cILfTJKGACdFxPfTjqWpSXoJOBGYEBH758qmR0SfdCNLh6RtgV65w1lZ/NAHIKmCJLkfBvQDPgaei4jLUw3MLKPcRb8JJO1HMohqBPAmcF+6EaUnIt5JFi5bK5Or+UnaDvgh0CUiviepu6SeEfFw2rGl4G1gEnBZRIxJOxizrHOC3whJPYCRJIn9I+BOkp6PLK9F/o6kwUDk1hw/h4yORwBuJtku96Dc8QKSNdizmOD3J5k2+k1JFwKvA09HxI3phmWWTe6i34jcrmHPAt+JiLm5sjciYs90I0uPpA7AVcCXSdYE+AtwTkR8nGpgKZD0ckQMlPRK3uOKVyOiX9qxpUFSW5IkfwjJQNSIiK6pBmWWUW7Bb9xwkhb83yU9RrIPet316LOmZ90tQCUdTLL+eNasltSa3KI3kvYCsvoM/mVgW+AF4Dng0Ih4K92ozLLLLfgCSWoDfI2kq/5I4Fbg/oj4S6qBpUDSvyKi/8bKskDSMcB/Ab1JejIOBk6LiKfSjCsNkjpGxIdpx2FmCSf4zZDbOvYbwIgsbaSRmx43GPg/QP42oO2Br2e4W3pn4ECSnp0XI6Iy5ZBSIWl74GfUTiV9GrjUc+HN0tEi7QC2RhHxcURcl6XkntOSZHvccpLlSWu+lpBMm8scSX8FDoiIRyLi4YiolHR92nGl5CZgKfAfua8lJIMQzSwFbsHbJpPUpebZqqQWJAveLEk5rFRIegN4B/hb3gp/WX1c4eWczZoRt+Btc1wuqX1uXMJMYLakC9IOKiWfAkcBu0h6KNdNnVUrcwtAAWsHXq5MMR6zTHOCt83RO9di/xowEegMnJJuSKlRRFTllum9l2T0+BdSjiktY4BrJc2XNB+4Bjgj3ZDMssvT5GxzbCNpG5IEf01EfC4pq896xta8iIhbJE0DMrV8saTOEfF2RLwK9JPUHiCrj23Mmgu34G1zXEey53kb4BlJXUgGVGVGTRID7pa0U80XyRLG56cYWhoeqHkh6d6IWOLkbpY+D7KzLUJSeURUpR1HU5H0cER8VdKbJIvc5C9+FFla6bDOKn5rX5tZutxFb5tM0sUNnLq0SQNJUUR8NfffbmnH0gxEA6/NLEVO8LY5lue9bgV8lYxtNiNpg9PgIuJfTRVLM9BP0hKSXozWudfkjiMi2jf8rWZWLO6it0bL7Yc+ISL+Le1Ymoqkv2/gdGRwESQza2bcgrctYTsgM8+cATK+XbCZbQWc4G2T5aaC1XT9lAEdydDz97ok9SHZbKZVTVlEjEsvIjMzd9HbZshNi6tRBbyfpRH0+ST9DDicJMFPBIYBz0VEJtfmN7Pmw/PgrWB5c7+X5n2tBNpL2lFSWWrBpedEkqVqF0XE6UA/kj3RzcxS5S562xS3k4yYn8z6c78B2kr6Y0T8pMkjS8/KiKiWVJX7APQBGRuPYGbNkxO8FWxjc79zLfjpQJYS/MuSdgD+SPLBZxnwz3RDMjPzM3jbBJ77vWGSugLtI2JqyqGYmTnBW+Hy5n63AgYCr5J00+8LvBQRQxr63lImaV+gK3k9YhFxX2oBmZnhLnrbBDVzvyWNB0ZHxLTccR+yt8EKAJJuIvmAMwOozhUH4ARvZqlygrfN0asmuQNExHRJ+6UZUIoOjIjeaQdhZlaXE7xtjtck3QD8iaS1ejIZW4s+zz8k9Y6ImWkHYmaWz8/gbZNJagWcCRyaK3oG+ENErEovqnRIOhR4CFgErKJ2g5V9Uw3MzDLPCd4aTdIQ4KSI+H7asTQ1SXOBHwLTqH0GT0S8lVpQZma4i942U+6Z+0nACOBNsjuo7O2ImJB2EGZmdTnBW8Ek9QBGkiT2j4A7SXqBsryz2ixJt5N00699ROFpcmaWNnfRW8EkVQPPAt+JiLm5sjciIrNLs0q6uZ7iiIhvN3kwZmZ53IK3TTGcpAX/d0mPAeNZfz36zMgtzVsZERekHYuZWV1uwdsmk9QG+BpJV/2RwK3A/RHxl1QDS4Gkv0bEUWnHYWZWlxO8NYqknYBvACMi4si042lqkn4NdAfuBpbXlPsZvJmlzQnerBH8DN7MmisneDMzsxLUIu0AzLZmkiok3S/pA0nvS7pXUkXacZmZOcGbNc7NwARgd6ATyXz4+rrtzcyalLvozRpB0pSI2G9jZWZmTc0teLPGqZR0sqSy3NfJJKv8mZmlyi14s0aQ1Bm4BjiIZOvcF4BzvdmMmaXNCd7MzKwEealas80g6eINnI6I+EWTBWNmVg+34M02g6Tz6iluA3wH2Dki2jZxSGZm63CCN2skSe2Ac0mS+13AryPig3SjMrOscxe92WbKrcP/Q2AUyYY7/SPik3SjMjNLOMGbbQZJvwJOAK4H+kbEspRDMjNbh7vozTaDpGpgFVBFMj1u7SmSQXbtUwnMzCzHCd7MzKwEeSU7MzOzEuQEb2ZmVoKc4M3MzEqQE7wZICkk3ZZ3XC7pQ0kPb+T79pP0lQ2cHyjp6i0Zq5lZIZzgzRLLgT6SWueOjwYWFvB9+wH1JnhJ5RHxckScs4ViNDMrmBO8Wa1HgWNzr08C7qg5IamNpJskTZL0iqTjJbUELgVGSJoiaYSkSyRdL+kvwDhJh9f0AkhqK+lmSdMkTZU0PLfF7C2SpufK/7Op37SZlSYneLNa44GRkloB+wIv5Z37L+BvEfEl4AjgV8A2wMXAnRGxX0Tcmbt2AHB8RHyzzv3/L7A4IvpGxL7A30h6ADpFRJ+I6AvcXKw3Z2bZ4gRvlhMRU4GuJK33iXVOHwNcKGkK8BTQCujcwK0mRMTKesq/DFyb9/M+Ad4A9pT0e0lDgSWNeQ9mZjWc4M3WNQG4krzu+RwBw3Mt9f0ionNEvNbAPZY3UC7WXfWuJsn3I/nQ8H3ghs0N3MwsnxO82bpuAi6NiGl1yh8HfiBJAJL2z5UvBdoVeO+/AGfXHEjaUVIHoEVE3EvShd+/McGbmdVwgjfLExELIuKqek79guSZ+1RJ03PHAH8HetcMstvI7f8b2DE3oO5Vkmf5nYCncl3/twAXbYn3YWbmtejNzMxKkFvwZmZmJcgJ3szMrAQ5wZuZmZUgJ3gzM7MS5ARvZmZWgpzgzczMSpATvJmZWQn6/xW6IItzsOuqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(performances).plot.bar()\n",
    "plt.grid(alpha = 0.3)\n",
    "plt.xlabel('Metrics')\n",
    "plt.ylabel('Value')\n",
    "plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}