Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from deeplearning.classifiers.fc_net import *\n",
    "from deeplearning.data_utils import get_CIFAR10_data\n",
    "from deeplearning.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from deeplearning.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_val:  (1000, 3, 32, 32)\n",
      "X_train:  (49000, 3, 32, 32)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "y_train:  (49000,)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.iteritems():\n",
    "    print '%s: ' % k, v.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `deeplearning/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.3\n",
      "Mean of input:  10.001050801424926\n",
      "Mean of train-time output:  9.977781971066806\n",
      "Mean of test-time output:  10.001050801424926\n",
      "Fraction of train-time output set to zero:  0.30162\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.6\n",
      "Mean of input:  10.001050801424926\n",
      "Mean of train-time output:  9.949213362769823\n",
      "Mean of test-time output:  10.001050801424926\n",
      "Fraction of train-time output set to zero:  0.602144\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.75\n",
      "Mean of input:  10.001050801424926\n",
      "Mean of train-time output:  9.985791035156765\n",
      "Mean of test-time output:  10.001050801424926\n",
      "Fraction of train-time output set to zero:  0.750356\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.3, 0.6, 0.75]:\n",
    "    out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "    out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "    print 'Running tests with p = ', p\n",
    "    print 'Mean of input: ', x.mean()\n",
    "    print 'Mean of train-time output: ', out.mean()\n",
    "    print 'Mean of test-time output: ', out_test.mean()\n",
    "    print 'Fraction of train-time output set to zero: ', (out == 0).mean()\n",
    "    print 'Fraction of test-time output set to zero: ', (out_test == 0).mean()\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `deeplearning/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.8929037388038322e-11\n"
     ]
    }
   ],
   "source": [
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "print 'dx relative error: ', rel_error(dx, dx_num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `deeplearning/classifiers/fc_net.py`, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  0\n",
      "Initial loss:  2.3051948273987857\n",
      "W1 relative error: 2.53e-07\n",
      "W2 relative error: 1.50e-05\n",
      "W3 relative error: 2.75e-07\n",
      "b1 relative error: 2.94e-06\n",
      "b2 relative error: 5.05e-08\n",
      "b3 relative error: 1.17e-10\n",
      "\n",
      "Running check with dropout =  0.25\n",
      "Initial loss:  2.29898614757146\n",
      "W1 relative error: 9.74e-07\n",
      "W2 relative error: 5.07e-08\n",
      "W3 relative error: 3.04e-08\n",
      "b1 relative error: 2.01e-08\n",
      "b2 relative error: 1.90e-09\n",
      "b3 relative error: 1.30e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.302437587710995\n",
      "W1 relative error: 4.55e-08\n",
      "W2 relative error: 2.97e-08\n",
      "W3 relative error: 4.34e-07\n",
      "b1 relative error: 1.87e-08\n",
      "b2 relative error: 5.05e-09\n",
      "b3 relative error: 7.49e-11\n",
      "\n"
     ]
    }
   ],
   "source": [
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "    print 'Running check with dropout = ', dropout\n",
    "    model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                              weight_scale=5e-2, dtype=np.float64,\n",
    "                              dropout=dropout, seed=123)\n",
    "\n",
    "    loss, grads = model.loss(X, y)\n",
    "    print 'Initial loss: ', loss\n",
    "\n",
    "    for name in sorted(grads):\n",
    "        f = lambda _: model.loss(X, y)[0]\n",
    "        grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "        print '%s relative error: %.2e' % (name, rel_error(grad_num, grads[name]))\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "(Iteration 1 / 125) loss: 7.971096\n",
      "(Epoch 0 / 25) train acc: 0.232000; val_acc: 0.151000\n",
      "(Epoch 1 / 25) train acc: 0.350000; val_acc: 0.229000\n",
      "(Epoch 2 / 25) train acc: 0.462000; val_acc: 0.264000\n",
      "(Epoch 3 / 25) train acc: 0.592000; val_acc: 0.254000\n",
      "(Epoch 4 / 25) train acc: 0.604000; val_acc: 0.245000\n",
      "(Epoch 5 / 25) train acc: 0.700000; val_acc: 0.296000\n",
      "(Epoch 6 / 25) train acc: 0.782000; val_acc: 0.255000\n",
      "(Epoch 7 / 25) train acc: 0.786000; val_acc: 0.259000\n",
      "(Epoch 8 / 25) train acc: 0.840000; val_acc: 0.280000\n",
      "(Epoch 9 / 25) train acc: 0.896000; val_acc: 0.256000\n",
      "(Epoch 10 / 25) train acc: 0.884000; val_acc: 0.265000\n",
      "(Epoch 11 / 25) train acc: 0.936000; val_acc: 0.297000\n",
      "(Epoch 12 / 25) train acc: 0.962000; val_acc: 0.296000\n",
      "(Epoch 13 / 25) train acc: 0.966000; val_acc: 0.306000\n",
      "(Epoch 14 / 25) train acc: 0.974000; val_acc: 0.285000\n",
      "(Epoch 15 / 25) train acc: 0.994000; val_acc: 0.296000\n",
      "(Epoch 16 / 25) train acc: 0.982000; val_acc: 0.287000\n",
      "(Epoch 17 / 25) train acc: 0.982000; val_acc: 0.292000\n",
      "(Epoch 18 / 25) train acc: 0.992000; val_acc: 0.304000\n",
      "(Epoch 19 / 25) train acc: 0.988000; val_acc: 0.293000\n",
      "(Epoch 20 / 25) train acc: 0.990000; val_acc: 0.290000\n",
      "(Iteration 101 / 125) loss: 0.006705\n",
      "(Epoch 21 / 25) train acc: 0.990000; val_acc: 0.305000\n",
      "(Epoch 22 / 25) train acc: 0.990000; val_acc: 0.314000\n",
      "(Epoch 23 / 25) train acc: 0.974000; val_acc: 0.297000\n",
      "(Epoch 24 / 25) train acc: 0.998000; val_acc: 0.312000\n",
      "(Epoch 25 / 25) train acc: 0.994000; val_acc: 0.307000\n",
      "0.75\n",
      "(Iteration 1 / 125) loss: 16.301255\n",
      "(Epoch 0 / 25) train acc: 0.228000; val_acc: 0.156000\n",
      "(Epoch 1 / 25) train acc: 0.342000; val_acc: 0.245000\n",
      "(Epoch 2 / 25) train acc: 0.480000; val_acc: 0.298000\n",
      "(Epoch 3 / 25) train acc: 0.546000; val_acc: 0.287000\n",
      "(Epoch 4 / 25) train acc: 0.548000; val_acc: 0.300000\n",
      "(Epoch 5 / 25) train acc: 0.618000; val_acc: 0.311000\n",
      "(Epoch 6 / 25) train acc: 0.628000; val_acc: 0.304000\n",
      "(Epoch 7 / 25) train acc: 0.650000; val_acc: 0.327000\n",
      "(Epoch 8 / 25) train acc: 0.734000; val_acc: 0.319000\n",
      "(Epoch 9 / 25) train acc: 0.730000; val_acc: 0.307000\n",
      "(Epoch 10 / 25) train acc: 0.738000; val_acc: 0.318000\n",
      "(Epoch 11 / 25) train acc: 0.740000; val_acc: 0.314000\n",
      "(Epoch 12 / 25) train acc: 0.754000; val_acc: 0.318000\n",
      "(Epoch 13 / 25) train acc: 0.796000; val_acc: 0.304000\n",
      "(Epoch 14 / 25) train acc: 0.810000; val_acc: 0.335000\n",
      "(Epoch 15 / 25) train acc: 0.826000; val_acc: 0.318000\n",
      "(Epoch 16 / 25) train acc: 0.858000; val_acc: 0.288000\n",
      "(Epoch 17 / 25) train acc: 0.828000; val_acc: 0.287000\n",
      "(Epoch 18 / 25) train acc: 0.840000; val_acc: 0.292000\n",
      "(Epoch 19 / 25) train acc: 0.876000; val_acc: 0.320000\n",
      "(Epoch 20 / 25) train acc: 0.892000; val_acc: 0.283000\n",
      "(Iteration 101 / 125) loss: 6.670227\n",
      "(Epoch 21 / 25) train acc: 0.896000; val_acc: 0.311000\n",
      "(Epoch 22 / 25) train acc: 0.886000; val_acc: 0.309000\n",
      "(Epoch 23 / 25) train acc: 0.878000; val_acc: 0.285000\n",
      "(Epoch 24 / 25) train acc: 0.886000; val_acc: 0.302000\n",
      "(Epoch 25 / 25) train acc: 0.920000; val_acc: 0.314000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "\n",
    "num_train = 500\n",
    "small_data = {\n",
    "    'X_train': data['X_train'][:num_train],\n",
    "    'y_train': data['y_train'][:num_train],\n",
    "    'X_val': data['X_val'],\n",
    "    'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0.75]\n",
    "for dropout in dropout_choices:\n",
    "    model = FullyConnectedNet([500], dropout=dropout)\n",
    "    print dropout\n",
    "\n",
    "    solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "    solver.train()\n",
    "    solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4MAAAJXCAYAAAApT4F3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X2cW3Wd9//3JxSElBYKChaQcjcBt+okBFi9gOlU9iF1\nV4WiXBRlWRjUogs663qJum4ZinoJ128vquu1bruSLje6rKVWcUXBVQZkhZWGZJACZuRmKHcKtkDp\niLRzPr8/cmaamc60M5OZnJPk9Xw88mjO95x880nm9CSffO/M3QUAAAAAaC6JqAMAAAAAANQeySAA\nAAAANCGSQQAAAABoQiSDAAAAANCESAYBAAAAoAmRDAIAAABAEyIZBAA0PDNLmNkWMzss6lgAAIgL\nY51BAEDcmNkWSYMfUDMl/VHSQFi21N3/LarYAABoFCSDAIBYM7PHJF3k7nfs4pg93H2ghmHVTCO/\nNgBAtOgmCgCIOwtvOwrMrjSzm8zs22b2kqQPmdnbzeweM9tsZk+b2VfNbI/w+D3MLDCzw8PtG8L9\nt5rZy2b2X2Y2b9QnL1tjZs+a2SYz+5mZHVexfx8zu8bM+sLn7jazPcN9bWFML4b7PxSW/9zMzq+o\n4yIzu2NErB8zs15JD4fl/2hmG8O6/tvM3lHx+D3M7O/N7Ddm9pKZ/dLM3mhm/2xmXxnxen5oZn9d\nxd8DANAgSAYBAPXqTEk3uvt+kv5d0jZJn5B0gKSTJZ0uaWnF8SO7wpwr6e8kzZG0UdKVu3iuH0g6\nWtIbJT0o6YaKfSskvUXSieFzf15SYGZHSvqhpP8vLM9I+tUunmNkfO+VdIKkt4bb94bPc4CkmyWt\nGUw6JX1G0lmS3hW+Hx+W9Kqk68LXKUkys4MkLZD07V3EAQBoEiSDAIB6dbe73ypJ7v5Hd8+7+31e\n9oSkf1E58RlkIx5/s7sXwi6Y35KUHu1Jwvqud/d+d39N0nJJ2bBFMCHpryRd6u6/C4/9RVjnhyTd\n6u5r3T1w903u/sAEXt+X3P0ld/9jGMe3wu1A5QRztqRjwmMvkvQ5d38sPPYBd3/R3e+R9KqZDb4P\n50r6T3ffPIE4AAANimQQAFCvNlZumNmxZvYfYXfOlyRdIen1u3j8cxX3+yXtO9pB4UykV5vZo2b2\noqRelVvxXi/pYEl7SnpslIe+SdKj4341O3tqRByfMbOHzWyzpE2Sktrx+t40RgxSuRXzvPD+eRre\nqgkAaGIkgwCAejWyW+VKlbthHhV2lbxcO7cGTsb5khZJanf3/VVujRscx/hbSa+p3IV0pI3a0XI3\n0laVk7lBbxzlmKHXZ2btkv5G0mJ3n+Puc8I6Bl/fk2PEIJWTv7PMLC3pKJW7vAIAQDIIAGgYsyS9\n5O5/MLM3a/h4wWrr/aOkzWY2U9KXFSZqYZfNf5W0wswODlsR/0c4cc2Nkk43s8XhBC8HmtnbwjqL\nkt5vZnubWUpSxzhi2CZpk5ntZWZXaHgyea2kL5rZUZJkZq1mtn8YY5+kB1QeP7gm7OoKAADJIAAg\n9sa7BtLfSrrAzF6W9A1JN+2inomsq7Ra0rOSnlG55fHuEfs/pfKMn3lJv5f0JZWXbnpC5UlgPqty\nt868yhPASOUxf1K5ZfGb2rnr5sj4bpX0U5W7qD4m6cUwpkH/R9L3JP007CK7UtLeFfuvC5/7+nG8\nXgBAk4hsnUEzu1bSeyT91t3fNsYxX5P0bpW7wlzg7sUahggAQEMIu5le6+5jdSUFADShKFsGV6s8\n7feozOzdko529xaVu/r8c60CAwCgUZjZXpI+KWlV1LEAAOIlsmTQ3e+WtKuprc9Q2J3F3f9b0n5m\ndnAtYgMAoBGY2VtU7qI6R9I/RhwOACBmZkQdwC4cquHThj8dlv02mnAAAKgv7v6gxlgyAwCAOCeD\n42Zm0Qx8BAAAAICYcPcJLakU59lEn1Z5Ed1Bh4Vlo3J3btxid7v88ssjj4Ebt9FunJvc4nRbv369\nksm1Kk+i6iovEelKJm/W+vXrI4+v2W4DAwNKpy+VNFDxNymXDQwM1DSW++9/UOn0pUom1yqZXKt0\n+lLdf/+DEb4Xl0fyXsThbxKHGOJy2/ma5ZFcs3aOY+KiTgYHF+0dzS0qL/QrM3u7pBfdnS6iAABg\n2gRBoHw+r3w+ryAImjKOQqGgUqldw78mJlQqLVChUKhZHEEQqKNjpYrFFervP0v9/WepWFyhjo6V\nNXtP4vJexCGOOMQQF5lMRqlUt6TK8zBQKnWnMplMNEFNUmTJoJl9W9IvJKXM7Ekzu9DMlprZRyXJ\n3W+V9LiZ/Ubl9ZI+HlWsAABg+sTli1WhsEHZbKfa2vrU1tanbLZThcKGmj1/3OKIGskH4iqRSCiX\nW6p0unOo1bq19ZPK5ZYqkahdejX6tXOCom5mnYpb+WUA8XPHHXdEHQIwKs5NVBoYGPD169f7+vXr\nfWBgIJIY7r//QU+nL/Vk8mZ/3eu6vLX1Er///gdr9vwDAwOeTl/q0oBLHt7KZbV8T+IQRxxicHdf\nv369J5NrK2Io35LJm339+vU1iWHn9+IOzouIz4vKeKK+bsUhhsprZ5gTTSiPimzR+alkZt4IrwMA\ngForFDaoo2Nl2AIjpVLdyuWWKpOZX/NYgiAYavHJZDI1/YU9n8+rra1P/f1nDStPJtfqrruOUDab\nbao4dpwXCyRJLS3dWr364pqeF0EQKJvtVLG4QjtaBwOl053K51fU7PyIw3sRlzjiEMPwONolRXvd\nioPBa+cJJ5wgn+AEMiSDAAA0qbh82Y6DuCRhcYlDijY5HxSX5CMO70Vc4og6Bq5bYzMzkkEAAOoJ\nrWHxEJcvmHGJI06iTj4QL1y3xjaZZLAh1hkEAGCi4vAFc+euTtc1dVenKA1OCNHR0TmsFSqXu7im\n50Zc4oiTRCLR1F/wgelEyyAAoOnEYbxJHFqA4hBD3MThR4I4xQHEDdetsdFNFACA3YjLF4m4dHWK\ny5gsABgvrlujo5soAAC7sbu1y5qtO1omM1/5/IqKVqivNvUv6wDij+vW1CEZBAAgAuXFgq9TsXim\nKlsoywutL65pLIzJAlBvuG5NDVJoAEBTKSdh3ZKCitLBJCxTszgGJwpJpzuVTK5VMrlWra2fVC63\nlF+4AQA1wZhBAEDTidN4EyYKAQBMBSaQAQBgnEjCAACNhGQQAFAXSMQAAJhak0kG+fQFANRUobBB\n2Wyn2tr61NbWp2y2U4XChqjDAgCg6dAyCACombis8QcAQKOhZRAAEGu7W+MPAADUDskgAAAAADQh\nkkEAQM3EZY0/AAAaRRAEyufzk3rsjCmOBQCAMQ0utN7R0Tlsjb9c7mLGCwIAMEGFnoI6lnWoNKs0\nqcczgQwAoOZYWgIAgOoEQaDs4qyK6WK5v2eXJjyBDC2DAICaSyQSymazUYcBAEDdKhQK5RbBKn5P\n5adYAAAAAGhCJIMAAAAAUGcymYxSW1LD52SboMiSQTNbZGaPmFnJzC4bZf/+ZvZdM+sxs3vN7E+i\niBMAAAAA4iaRSCi3PKd0Ma1kb3JSdUQygYyZJSSVJJ0m6RlJ90la4u6PVBxztaQt7n6lmR0r6f+5\n+5+NUR8TyACIPSZNAQBUg88RjGbwvDjhhBMmPIFMVGfQSZJ63b3P3bdJuknSGSOO+RNJP5Mkd/+1\npCPM7A21DRMApkahsEHZbKfa2vrU1tanbLZThcKGqMMCAIzD4Dpu+XxeQVBFn7wqFHoKyi7Oqu2a\nNrVd06bs4qwKPYVIYkG8VDMpW1Qtg++XdLq7fzTcPk/SSe7+iYpjviRpb3f/WzM7SdLdkv7U3Xc6\n62kZBBBnQRAom+1UsbhCO36DC5ROdyqfX8EvuwAQYyPXcUttSSm3PKdMa6ZmMey0hIAkBVK6mFZ+\nXZ7PEUiSzKyhlpb4iqSvmtn9kn4lqSBpYKyDu7q6hu63t7ervb19msMDgPEpFAoqldo1vDNGQqXS\nAhUKBZZYAICYCoJAHcs6hiVhxaCojmUdNU3CRl1CICGVZpVq/jlCV9Udon4vuru71d3dXVUdUSWD\nT0s6vGL7sLBsiLtvkdQxuG1mj0t6bKwKK5NBAMDoov7gAoB6EqckLA7i0EoaF3F4L0Y2gF1xxRUT\nriOqbwH3STrGzOaZ2V6Slki6pfIAM9vPzPYM739E0p3u/krtQwWA6mQyGaVS3Ro+93OgVOpOZTK1\n+9Bg3CIA1KdRlxAIyglIrT5HKltJ+1v61d/Sr2K63Eoa1TjKqDTSexFJMujuA5IukXS7pA2SbnL3\nh81sqZl9NDzszZIeNLOHJZ0u6ZNRxAoA1UokEsrlliqd7lQyuVbJ5Fq1tn5SudzSmrXMBUGgjo6V\nKhZXqL//LPX3n6VicYU6OlbW3QcXgOYS5eQtcUjCpJ2XEEj2JtVaaFVueS42XVVrLcrzIm7vRTUi\nGzPo7j+WdOyIspUV9+8duR8A6lUmM1/5/IqKLppfrWkXTcYtAqhHUXfFG0zCKmNoeblFuStrl4QN\nyrRmlF+Xp6u/oj8vGkkks4lONWYTBYBdy+fzamvrU3//WcPKk8m1uuuuI0gGAcROnGbQZLx1fP4e\ncYgjDjGMZjKziTbfmQwATSgu4xYBYLzi1BVvcB23bDbblImgFI+uqlI8zou4vBdTIc5LSwAApsjg\nuMWOjk6VSgskSS0t3crlLq67Dy4AQDToqrpDo7wXdBMFgCZCVycA9SKuXfEQLc6LsU2mmyjJIAAA\nQIgfTOJl5EQhLS+3aPWVq5kopMlxXoyOZBAAAGCSmKEwnkjQMRrOi52RDAIAAEwCXc8A1DtmEwUA\nAJiEOMxQCAC1RjIIAAAAAE2IZBBAwwuCQPl8Xvl8XkEQ7P4BAJpOJpNRaktq5FKcSm1JNe1anFw7\ngcZHMgigoRUKG5TNdqqtrU9tbX3KZjtVKGyIOiwAMdNIi0hPhUJPQdnFWbVd06a2a9qUXZxVoYfu\nskCjYQIZAA0rCAJls50qFleockaIdLpT+fyKpvyCB2DXmKGQyXSAesUEMgBQoVAoqFRq18gZIUql\nBUwIAWBUiURC2WxW2Wy2aZMeJtMBmseMqAMAAADAcLRQAqgFriwAGlYmk1Eq1a2RM0KkUnc27YQQ\nQJwxYUlZ1OP1mEwHaB6MGQTQ0AqFDeroWKlSaYEkqaWlW6tXX6xMZn7EkQGoVOgpqGNZR7l7osqJ\nR255TpnW5ko+4jJeb+Tfo+XlFq2+cnXT/T2AejKZMYMkgwAaHt2tgHiLSwIUB/l8Xm3XtKm/pX9Y\nebI3qbv+5i5ls9maxcK1E6gvk0kGGTMIoOENTggBIJ52N2EJ/3+jwbUTaHz8xAMAABATjNcDUEsk\ngwAANLmoJ24hAdohkUgotzyndDGtZG9Syd6kWgutyi3P0U0TwJRjzCAAAE0sLhO3MGHJcIzXAzBR\nTCADAADGLW4Tt5AAAcDkTSYZjOwqa2aLzOwRMyuZ2WWj7D/QzH5kZkUz+5WZXRBBmAAANKzdTdxS\na4MTlmSzWRJBAKiBSK60ZpaQ9HVJp0uaL+lcMztuxGGXSCq6e1rSQkn/YGbMfgoAaChRj9cDADSv\nqH52O0lSr7v3ufs2STdJOmPEMc9JmhXenyXp9+6+vYYxAgAwrQo9BWUXZ9V2TZvarmlTdnFWhZ7a\ntcgxcQsANLeoksFDJW2s2H4qLKv0L5Lmm9kzknokfbJGsQEAMO2CIFDHsg4V00X1t/Srv6VfxXRR\nHcs6atZCyMyVANDc4tzt8nOSetx9oZkdLeknZvY2d39ltIO7urqG7re3t6u9vb0mQQLYNSaEAEYX\nl4XWM60Z5dfl+X8KAHWmu7tb3d3dVdURVTL4tKTDK7YPC8sqnSzpS5Lk7o+a2eOSjpO0frQKK5NB\nAPFIwgqFDeroWKlSqV2SlEpdp1xuqTKZ+TWPBcDYBiduAQDUj5ENYFdcccWE64jqp7/7JB1jZvPM\nbC9JSyTdMuKYhyX9mSSZ2cGSUpIeq2mUQJ0qFDYom+1UW1uf2tr6lM12qlDYUNMYgiBQR8dKFYsr\n1N9/lvr7z1KxuEIdHSuZJAMQ4/UAANGLbJ1BM1sk6asqJ6TXuvtXzGypJHf3VWb2ekmrVW5BNEn/\n293/bYy6WGcQCAVBoGy2U8XiClUuHJZOdyqfX1GzFsJ8Pq+2tj719581rDyZXKu77jqCVghALLQO\nAJg6k1lnMLIxg+7+Y0nHjihbWXH/BUnvrXVcQL0rFApht8zhA5FKpQU1HYcEYPcYrwcAiBKfOACm\nRSaTUSrVrZF94FKpO+kCB1RgoXUAQFT41AEaTFySsEQioVxuqdLpTiWTa5VMrlVr6yeVyy3lCy8A\nAEAMRDZmcCoxZhAYbscsngskSS0t3Vq9+uJIZvGMw6ymAAAAjW4yYwZJBoEGRRIGAADQPEgGAQAA\nAKAJ1dVsogAARInWcwBAsyMZBAA0nZHr+6W2pJRbnmN9PwBAU6GbKACgqQRBoOzirIrp4o45tQMp\nXUwrvy5PCyEAoC5Nppson3gAgKZSKBTKLYKVn4AJqTSrNNRtFACAZkA3UQBAzTFeDwCA6PHpCwBN\nJAgC5fN55fN5BUEQSQyFnoKyi7Nqu6ZNbde0Kbs4q0JP7VrkMpmMUltSUuXLD8rjBjMZxgwCAJoH\nYwYBNDxaocriMGlKXMbrjXwvWl5u0eorVzOBDACgbrHOIACMEIcEaFCUSWlckrB8Pq+2a9rU39I/\nrDzZm9Rdf3OXstlsTeKQ+JEAANBYmEAGACoEQaCOZR0qpovqb+lXf0u/iumiOpZ11LyLZNRdI5k0\nZWeJRELZbFbZbJZEEADQlPj0A9Cw4pIAxSkpjRrj9QAAiA+SQQCYZnFISuOShCUSCeWW55QuppXs\nTSrZm1RroVW55Tla5wAAqDGWlgDQsAYToGIwfJxcM7ZCDSZhIydNyV1Z+yQs05pRfl2e8XoAAESM\nCWQANLQ4zBoZl8lbBmMhCQMAoPEwmygAjCIOCVAcklIAANC4SAYBIMbikJQCAIDGRDIIAAAAAE2I\ndQYBAAAAAOMSWTJoZovM7BEzK5nZZaPs/7SZFczsfjP7lZltN7P9o4gVmKggCJTP55XP55tuHTkA\nAADUh0i6iZpZQlJJ0mmSnpF0n6Ql7v7IGMe/R1Knu//ZGPvpJorYKBQ2qKNjpUqldklSKtWtXG6p\nMpn5kcYFAACAxlU3YwbN7O2SLnf3d4fbn5Xk7n7VGMd/S9LP3P3aMfaTDCIWgiBQNtupYnGFKtcQ\nSKc7lc+vYMIQAAAATIt6GjN4qKSNFdtPhWU7MbN9JC2StLYGcQFVKRQKYYtg5X+thEqlBUOzSAIA\nAABxMCPqAMbhvZLudvcXd3VQV1fX0P329na1t7dPb1QAAAAAEJHu7m51d3dXVUeU3US73H1RuD1m\nN1Ez+66k77j7Tbuoj26iiAW6iQIAACAK9TRmcA9Jv1Z5AplnJf1S0rnu/vCI4/aT9Jikw9z9D7uo\nj2QQsbFjApkFkqSWlm6tXn0xE8gAAABg2tRNMiiVl5aQ9FWVm0+udfevmNlSlVsIV4XH/JWk0939\ng7upi2QQsRIEwdAYwUwmQ4sgAAAAplVdJYNTiWQQAAAAQDOrp9lEAQAAAAARIhkEAAAAgCZEMggA\nAAAATYhkEAAAAACaEMkgAAAAADShqpJBM7vUzOZMVTAAAAAAgNqotmXwYEn3mdl3zGyRmU1oKlMA\njS8IAuXzeeXzeQVBEHU4AAAACFW9zmCYAL5L0oWSTpD0HZUXkX+0+vDGHQPrDAIxVOgpqGNZh0qz\nSpKk1JaUcstzyrRmIo4MAACgsUS26LyZtaqcDC6SdIekt0v6ibt/purKx/f8JINAzARBoOzirIrp\n4o4+CIGULqaVX5dXIsGQZQAAgKlS80XnzeyTZpaXdLWk/5L0Vnf/mKSspPdXUzeA+lYoFMotgpVX\nmYRUmlVSoVCILC4AAACUzajy8QdIOsvd+yoL3T0ws/dUWTcAAAAAYJpU20/rR5I2DW6Y2Wwz+1NJ\ncveHq6wbQB3LZDJKbUlJlXPGBOVxg5kMYwYBAACiVm0y+A1Jr1RsvxKWAWhyiURCueU5pYtpJXuT\nSvYm1VpoVW55jvGCAAAAMVDVBDJmVnT39IiyB9z9bVVHNrE4mEAGGCEIgqGxeZlMJrIELC5xAAAA\nNLKazyZqZt+V1K0drYEfl7TQ3c+cdKWTi4NkEKjAkg4AAADNJYpk8CBJX5P0Tkku6aeSOt39d5Ou\ndHJxkAwCIZZ0AAAAaD6TSQarmk00TPqWVFMH0Iii7Bq5uyUdstlszWIBAABAfFWVDJrZ3pIukjRf\n0t6D5e7eUWVcQN2iiyYAAADqQbXNFTdIeqOk0yXdKekwSVuqDQqoV0EQqGNZh4rpovpb+tXf0q9i\nuqiOZR0KgmD3FUwBlnQAAADAeFSbDB7j7n8vaau7XyfpLyT9afVhAfVpd100a4ElHQAAADAeVXUT\nlbQt/PdFM3uLpOckHVRlncCksYxBWaY1o/y6PO8FAAAAxlTtt8NVZjZH0hck3SLpIUlXjeeBZrbI\nzB4xs5KZXTbGMe1mVjCzB83sjipjRYMrFDYom+1UW1uf2tr6lM12qlDYUNMY4tRFM5FIKJvNKpvN\nkggCAABgJ5NeWsLMEpI+4O7fmeRjS5JOk/SMpPskLXH3RyqO2U/SLyS9y92fNrPXu/sLY9TH0hJN\nLggCZbOdKhZXqHI9hXS6U/n8itrO5jliApmWl1u0+srVTCADAACAaRPFOoPr3f2ESTzu7ZIud/d3\nh9ufleTuflXFMR+TNNfdl42jPpLBJpfP59XW1qf+/rOGlSeTa3XXXUfUfDkFuqsCAACglmq+zqCk\n/zSzT0v6d0lbBwvdfdNuHneopI0V209JOmnEMSlJe4bdQ/eV9DV3v6HKeDGNSIB2GOyiCQAAAMRV\ntcngOeG/f11R5pKOqrJeqRzb8ZLeKWmmpHvM7B53/81oB3d1dQ3db29vV3t7+xSEgPEqFDaoo2Ol\nSqV2SVIqdZ1yuaXKZObX5PkzmYxSqetULJ6pym6iqdSdymQW1yQGAAAAoFa6u7vV3d1dVR1VdROd\n9JOWu4l2ufuicHu0bqKXSdrb3a8It78p6UfuvnaU+ugmGqG4jNfbkZAukCS1tHRr9eqLa5aQAgAA\nAFGJYszg+aOVu/v1u3ncHpJ+rfIEMs9K+qWkc9394YpjjpP0j5IWSXqdpP+WdI67PzRKfSSDEYrT\neD26qgIAAKAZRTFm8MSK+3urnNzdL2mXyaC7D5jZJZJuV7kp6Vp3f9jMlpZ3+yp3f8TMbpP0gKQB\nSatGSwSBSozVAwAAAMZnSruJmtn+km4a7P5ZK7QMRisu3UQBAACAZhVFy+BIWyUdOcV1IuYSiYRy\nuaXq6OgcNl4vl7uYRBAAAACIqWrHDP5A5dlDpXKT0J9I+o67f3YKYptIHLQMxgDj9QAAAIBoRDGB\nzIKKze2S+tz9qUlXOPk4SAYBAAAANK0ouok+KelZd381DGAfMzvC3Z+osl4AAAAAwDSqth/fGklB\nxfZAWAYAAAAAiLFqk8EZ7v7a4EZ4f68q6wQAAAAATLNqk8Hnzex9gxtmdoakF6qsEwAAAAAwzaqd\nQOZoSd+SdEhY9JSk8939N1MQ20TiYAIZSGJGUwAAADSnms8mWvHE+0qSu79SdWWTe36SwRiIOhEr\n9BTUsaxDpVklSVJqS0q55TllWjM1jQMAAACotSiWlviypKvd/cVwe46kv3X3L0y60snFQTIYsagT\nsSAIlF2cVTFd3NH5OZDSxbTy6/K0EAIAAKChTSYZrPYb8rsHE0FJcvfNkv68yjpRZ4IgUMeyDhXT\nRfW39Ku/pV/FdFEdyzoUBMHuK5gChUKhnIhWntEJqTSrNNRaCQAAAGCHapPBPczsdYMbZraPpNft\n4ng0IBIxAAAAoP5Umwx+S9JPzewiM/uwpJ9Iuq76sICJyWQySm1JDV/1Mih3V81kGDMIAAAAjFRV\nMujuV0n6oqQ3SzpW0m2S5k1BXKgjcUjEEomEcstzShfTSvYmlexNqrXQqtzyHOMFAQAAgFFUPZuo\nmWUkfVDS2ZIel7TW3b8+BbFNJAYmkInYyAlkWl5u0eorV9d8Js+oZzQFAAAAolCz2UTNLCXpXElL\nJP1O0hpJ/8vdI2kVJBmMBxIxAAAAIBq1TAYDSf8h6a/dfWNY9pi7HzXhyqYAySAAAACAZlbLpSXO\nktQv6S4z+2cze6ekCT0xAAAAACA61S46P1PSGSp3GX2npOslrXP326cmvHHHQcsgAAAAgKZVs26i\nYzz5HJUnkTnH3U+bkkrH/9wkgwAAAACaVqTJYJRIBgEAAAA0s1qOGQQAAAAA1LHIkkEzW2Rmj5hZ\nycwuG2X/AjN70czuD29fiCJOAAAAAGhEM6J4UjNLSPq6pNMkPSPpPjP7vrs/MuLQu9z9fTUPEAAA\nAAAaXFQtgydJ6nX3PnffJukmlWclHYnlKgAAAABgGkSVDB4qaWPF9lNh2UjvMLOimf3QzP6kNqEB\nAAAAQOOLpJvoOOUlHe7u/Wb2bknfk5Qa6+Curq6h++3t7Wpvb5/u+AAAAAAgEt3d3eru7q6qjkiW\nljCzt0vqcvdF4fZnJbm7X7WLxzwuKevum0bZx9ISAAAAAJpWPS0tcZ+kY8xsnpntJWmJpFsqDzCz\ngyvun6Ry4rpTIoiyIAiUz+eVz+cVBEHU4QAAAACIuUi6ibr7gJldIul2lRPSa939YTNbWt7tqyR9\nwMw+JmmbpD9IOieKWOtBoaegjmUdKs0qSZJSW1LKLc8p05qJODIAAAAAcRVJN9Gp1szdRIMgUHZx\nVsV0cUc7byCli2nl1+WVSES2lCQAAACAGqmnbqKYIoVCodwiWPmXTEilWSUVCoXI4gIAAAAQbySD\nAAAAANCESAbrXCaTUWpLSqqcMyYojxvMZBgzCAAAAGB0JIN1LpFIKLc8p9Ziq/b+9d7a+9d7q7XQ\nqtzyHOMFAQAAAIwpzovOY7yCvaS+U6Vff6i8feyT5TIAAAAAGAOzida5IAiUzXaqWFyhyulE0+lO\n5fMraB1LQeI1AAAgAElEQVQEAAAAmgCziTahQqGgUqldI6cTLZUWMJsoAAAAgDGRDAIAAABAEyIZ\nrHOZTEapVLdGTieaSt3JbKIAAAAAxsSYwQZQKGxQR8dKlUoLJEktLd1avfpiZTLzI44MAAAAQC1M\nZswgyWCDCIJgaIxgJpNh4hgAAACgiZAMAgAAAEATYjZRAAAAAMC4kAwCAAAAQBMiGQQAAACAJkQy\nCAAAAABNiGQQAAAAAJoQySAAAAAANCGSQQAAAABoQiSDAAAAANCESAYBAAAAoAmRDAIAAABAE4os\nGTSzRWb2iJmVzOyyXRx3opltM7OzahkfAAAAADSySJJBM0tI+rqk0yXNl3SumR03xnFfkXRbbSME\nAAAAgMYWVcvgSZJ63b3P3bdJuknSGaMcd6mkmyX9rpbBAQAAAECjiyoZPFTSxortp8KyIWZ2iKQz\n3f0bkqyGsQEAAABAw5sRdQC7sEJS5VjCXSaEXV1dQ/fb29vV3t4+LUEBAAAAQNS6u7vV3d1dVR3m\n7lMTzUSe1OztkrrcfVG4/VlJ7u5XVRzz2OBdSa+XtFXSR939llHq8yheBwAAAADEgZnJ3SfUozKq\nZHAPSb+WdJqkZyX9UtK57v7wGMevlvQDd//uGPtJBgEAAAA0rckkg5F0E3X3ATO7RNLtKo9bvNbd\nHzazpeXdvmrkQ2oeJAAAAAA0sEhaBqcaLYMAAAAAmtlkWgYjW3QeAAAAABAdkkEAAAAAaEIkgwAA\nAADQhEgGAQAAAKAJkQwCAAAAQBMiGQQAAACAJhTJOoONJAgCFQoFSVImk1EiQX4NAAAAIP7IXKpQ\nKGxQNtuptrY+tbX1KZvtVKGwIeqwAAAAAGC3WHR+koIgUDbbqWJxhXbk1IHS6U7l8ytoIQQAAABQ\nMyw6X0OFQkGlUnu4lQ9vUqm0YKjbKAAAAADEFWMGqzCgx6TDs1K2VC7IpzTwwgclHRFlWAAAAACw\nW3QTnaTt27dr9lvm6g/nvFDZS1T7/Pvr9fKDz2rGDPJsAAAAALVBN9Ea6unpkR//yvB3MCH58a+o\np6cnsrgAAAAAYDxIBqsw2iQxTBwDAAAAoB6QuUxSJpNRaktKCioKAym1JaVMJhNZXAAAAAAwHiSD\nk5RIJJRbnlO6mFayN6lkb1KthVblludoHQQAAAAQe0wgU6UgCIaWkshkMiSCAAAAAGpuMhPIkAwC\nAAAAQJ1jNlEAAAAAwLiQDAIAAABAEyIZBAAAAIAmRDIIAAAAAE0osmTQzBaZ2SNmVjKzy0bZ/z4z\n6zGzgpmtN7N3RhEnAAAAADSiSGYTNbOEpJKk0yQ9I+k+SUvc/ZGKY5Lu3h/ef6ukde5+zBj1MZso\nAAAAgKZVT7OJniSp19373H2bpJsknVF5wGAiGNpX0gs1jA8AAAAAGlpUyeChkjZWbD8Vlg1jZmea\n2cOSbpX0iRrFBgAAAAANb0bUAeyKu39P0vfM7BRJN0g6dqxju7q6hu63t7ervb19usMDAAAAgEh0\nd3eru7u7qjqiGjP4dkld7r4o3P6sJHf3q3bxmEclneTuvx9lH2MGAQAAADStyYwZjKpl8D5Jx5jZ\nPEnPSloi6dzKA8zsaHd/NLx/vCSNlggCAAA0myOOOEJ9fX1RhwEgAvPmzdMTTzwxJXVFkgy6+4CZ\nXSLpdpXHLV7r7g+b2dLybl8l6f1mdr6k1yRtlXROFLECAADETV9fn+gVBTQnswk1/u26rka4kNBN\nFAAANJOwO1jUYQCIwFj//+tpaQkAAAAAQIRIBgEAAACgCZEMAgAAAEATIhkEAABoEEEQKJ/PK5/P\nKwiCyOqIg76+PiUSibp+DVOJc2MHzo0dSAYBAAAaQKGwQdlsp9ra+tTW1qdstlOFwoaa17F582Yt\nXrxY++67r4488kj927/92y6Pv+aaazR37lztv//++vCHP6xt27ZNuq6RpnLWxamUSCT02GOP1ez5\nCj0FZRdn1XZNm9quaVN2cVaFnkLN65jI3/NjH/uYZs2apdmzZ2v27Nnae++9td9++w3tb29v1z77\n7KPZs2dr1qxZevOb3zyhWDg3yphNFAAAoM6MnE0wCAJls50qFldox2/9gdLpTuXzK5RI7P73/6mo\nQ5LOPbe8dHQul9P999+vv/iLv9A999wz6pf12267TRdccIHuuOMOzZ07V2eeeabe8Y536Mtf/vKE\n6xqpr69PRx11lLZt2zZq7O4eWUKwxx57qLe3V0cdddS0P1cQBMouzqqYLlb+WZUuppVflx//uVFl\nHVJ1f88LL7xQe+yxh775zW9KkhYuXKjzzz9fF1544bieu1K9nxvMJgoAAIAhhUJBpVK7hn+1S6hU\nWqBCYXytN1NRR39/v7773e/qi1/8ovbZZx+dfPLJOuOMM3TDDTeMevz111+viy66SMcdd5z2228/\nLVu2TKtXr55UXUEQ6NOf/rTe8IY36JhjjtEPf/jDYfsXLlyoL3zhCzrllFM0c+ZMPf7443r22Wd1\nxhln6MADD1QqlRpKNCTpiiuu0Nlnn60lS5Zo9uzZOuGEE/TAAw8M7X/kkUe0cOFCzZkzR29961v1\ngx/8YNhz5XK5oe3rrrtOp556qiRpwYIFcne97W1v0+zZs7VmzZpxvbeTVSgUVJpVGvlnVWlWaWLn\nRpV1TPTvWWnr1q1au3atLrjggmHl420M4twYG8kgAAAApkSpVNKee+6po48+eqistbVVGzaM3tV0\nw4YNam1tHXbs7373O23evHnCda1atUq33nqrenp6tH79et188807HXPjjTfqm9/8prZs2aLDDz9c\nS5Ys0eGHH67nnntOa9as0ec//3l1d3cPHX/LLbfonHPO0ebNm3XuuefqzDPP1MDAgLZv3673vve9\nWrRokZ5//nl97Wtf04c+9CH19vaO+d4MtjTdeeedkqRf/epXevnll3X22WeP+ZhGMtG/Z6W1a9fq\noIMO0imnnDKs/HOf+5wOOuggnXrqqUPv62g4N8ZGMggAAFDnMpmMUqluSZUTYgRKpe5UJpOpWR2v\nvPKKZs+ePaxs9uzZ2rJly5jHV44Dmz17ttxdW7ZsmXBda9asUWdnpw455BDtv//++tznPrfTMRdc\ncIGOO+44JRIJPffcc/rFL36hq666SnvuuadaW1v14Q9/WNdff/3Q8dlsVosXL9Yee+yhT33qU/rj\nH/+oe++9V/fee6+2bt2qyy67TDNmzNDChQv1nve8Z0JjGms1xCmTySi1JTXyz6rUltTEzo0q65jo\n37PS9ddfr/PPP39Y2dVXX63HHntMTz/9tD7ykY/ove99rx5//PFRH8+5MTaSQQAAgDqXSCSUyy1V\nOt2pZHKtksm1am39pHK5peMezzUVdey77756+eWXh5W99NJLmjVr1riOf+mll2RmmjVr1oTreuaZ\nZ/SmN71paHvevHk7HVO5/5lnntEBBxygZDI57DFPP/30qMebmQ499FA988wzOz3XaI+Ni0Qiodzy\nnNLFtJK9SSV7k2ottCq3PDexc6PKOib69xz05JNPqru7e6dk8MQTT9TMmTO155576vzzz9fJJ5+s\nW2+9ddQ6ODfGNiPqAAAAAFC9TGa+8vkVQ2O4MpmvjvuL+lTVkUqltH37dj366KND3QF7eno0f/78\nUY+fP3++enp69IEPfECSVCwWdfDBB2vOnDl63eteN6G65s6dq40bNw5t9/X17XRM5aQghxxyiDZt\n2qStW7dq5syZksqJx6GHHjp0TGV97q6nnnpKhxxyiNxdTz755LC6n3zySR177LGSpJkzZ6q/v39o\n33PPPTdqzLWSac0ovy5f8XfNTPzcqLKOiZ4bg2688UadcsopOuKII3Z53FiTqkicG7vk7nV/K78M\nAACA5hDn7z7nnnuuf/CDH/StW7f6z3/+c99///39oYceGvXYH//4xz537lx/6KGHfNOmTd7e3u6f\n//znJ1XXN77xDZ8/f74/9dRTvmnTJj/ttNM8kUj4wMCAu7u3t7f7tddeO+wxbW1tfumll/qrr77q\nPT09fvDBB/vPfvYzd3fv6uryvfbay9etW+fbt2/3f/iHf/AjjzzSt2/f7q+99pofffTRftVVV/m2\nbdv8jjvu8NmzZ3upVHJ397/7u7/zhQsXen9/v/f29npLS4ufeuqpQ887d+5c/8lPfjL5N7lOTeTv\nOejYY4/1f/3Xfx1W9uKLL/ptt93mr776qm/fvt1vvPFG33fffb23t3fUOhrt3Bjr/39YPrE8aqIP\niONN0tAfEwAAoNHFORnctGmTn3nmmT5z5kyfN2+e33TTTUP7nnzySZ81a5Zv3LhxqOyaa67xgw8+\n2Pfbbz+/6KKL/LXXXhtXXSNt377dP/WpT/mBBx7oRx11lP/TP/3TsC/8Cxcu3OkL/9NPP+3vec97\n/IADDvBjjjnGV61aNbSvq6vLzz77bF+yZInPmjXLjz/+eC8Wi0P7H3roIV+wYIHvt99+Pn/+fP/+\n978/tO+FF17wd73rXT579mw/5ZRT/Iorrhj2hX/lypU+d+5cnzNnjq9Zs2Yib29dm+i5cc899/i+\n++7rr7zyyrB6nn/+eT/xxBN99uzZPmfOHH/HO97hP/3pT8d83kY7N6YyGWyYdQZbWy/R6tUXK5PZ\ndVMzAABAvdtVlzhMjSuuuEKPPvrosElDACn6c4N1BkfRs/nnWvKhLyoIgt0fDAAAAABNrmGSQV3Q\no94/3Kt8Ph91JAAAAAAQew3TTVRdkop76Yb3X6vzzjsv6pAAAACmDd1EgeZFN9Ex2B7B0LStAAAA\nAICxNU4yGEgtLx+tbDYbdSQAAAAAEHsNs+h8a6FVq//P6gkvoAkAAFBv5s2bN2yRbADNY968eVNW\nV8OMGRwYGCARBAAAANCU6mrMoJktMrNHzKxkZpeNsv+DZtYT3u42s7fuqj4SQcRRd3d31CEAo+Lc\nRJxxfiKuODfRaCLJoMwsIenrkk6XNF/SuWZ23IjDHpPU5u6tkr4o6V9qGyVQPT40EFecm4gzzk/E\nFecmGk1UzWknSep19z533ybpJklnVB7g7ve6+0vh5r2SDq1xjAAAAADQsKJKBg+VtLFi+yntOtn7\nsKQfTWtEAAAAANBEIplAxszeL+l0d/9ouH2epJPc/ROjHLtQ5S6lp7j75jHqq/9ZcAAAAACgChOd\nQCaqpSWelnR4xfZhYdkwZvY2SaskLRorEZQm/qIBAAAAoNlF1U30PknHmNk8M9tL0hJJt1QeYGaH\nS1or6S/d/dEIYgQAAACAhhVJy6C7D5jZJZJuVzkhvdbdHzazpeXdvkrS30s6QNI/WXlV1W3uflIU\n8QIAAABAo2mIRecBAAAAABNT1yu1727heiBKZvaEmfWYWcHMfhl1PGheZnatmf3WzB6oKJtjZreb\n2a/N7DYz2y/KGNG8xjg/Lzezp8zs/vC2KMoY0ZzM7DAz+5mZbTCzX5nZJ8Jyrp+I1Cjn5qVh+YSv\nnXXbMhguXF+SdJqkZ1Qeh7jE3R+JNDAgZGaPScruavIjoBbM7BRJr0i63t3fFpZdJen37n51+GPa\nHHf/bJRxojmNcX5eLmmLu//fSINDUzOzN0p6o7sXzWxfSXmV18W+UFw/EaFdnJvnaILXznpuGdzt\nwvVAxEz1/X8MDcLd75Y08keJMyRdF96/TtKZNQ0KCI1xfkrlaygQGXd/zt2L4f1XJD2s8gz4XD8R\nqTHOzcE12yd07aznL6oTXbgeqDWX9BMzu8/MPhJ1MMAIB7n7b6Xyh4qkgyKOBxjpEjMrmtk36YaH\nqJnZEZLSku6VdDDXT8RFxbn532HRhK6d9ZwMAnF3srsfL+nPJf112BUKiKv6HDOARvVPko5y97Sk\n5yTRXRSRCbvh3Szpk2ErzMjrJddPRGKUc3PC1856TgbHtXA9EBV3fzb893lJ61Tu2gzExW/N7GBp\naOzB7yKOBxji7s/7jkkN/kXSiVHGg+ZlZjNU/rJ9g7t/Pyzm+onIjXZuTubaWc/J4G4XrgeiYmbJ\n8NcamdlMSe+S9GC0UaHJmYaPI7hF0gXh/b+S9P2RDwBqaNj5GX7BHnSWuH4iOjlJD7n7VyvKuH4i\nDnY6Nydz7azb2USl8tISkr6qHQvXfyXikABJkpkdqXJroEuaIelbnJ+Iipl9W1K7pAMl/VbS5ZK+\nJ2mNpDdJ6pP0P939xahiRPMa4/xcqPIYmEDSE5KWDo7RAmrFzE6WdJekX6n8ee6SPi/pl5K+I66f\niMguzs0PaoLXzrpOBgEAAAAAk1PP3UQBAAAAAJNEMggAAAAATYhkEAAAAACaEMkgAAAAADQhkkEA\nAAAAaEIkgwAAAADQhEgGAQAImdmAmd1vZoXw389MYd3zzOxXU1UfAADVmhF1AAAAxMhWdz9+Gutn\ncV8AQGzQMggAwA42aqHZ42Z2lZk9YGb3mtlRYfk8M/upmRXN7CdmdlhYfpCZfTcsL5jZ28OqZpjZ\nKjN70Mx+bGavq9HrAgBgJySDAADssM+IbqJnV+zb7O5vk/T/JH01LPtHSavdPS3p2+G2JH1NUndY\nfrykDWF5i6R/dPe3SHpJ0vun+fUAADAmc6fHCgAAkmRmL7v77FHKH5e00N2fMLMZkp519zeY2fOS\n3ujuA2H5M+5+kJn9TtKh7r6too55km5392PD7c9ImuHuX67JiwMAYARaBgEAGB8f4/5E/LHi/oAY\nuw8AiBDJIAAAO4w6ZjB0TvjvEkn3hPf/S9K54f3zJP08vP+fkj4uSWaWMLPB1sZd1Q8AQE3xiyQA\nADvsbWb3q5y0uaQfu/vnw31zzKxH0qvakQB+QtJqM/u0pOclXRiWd0paZWYXSdou6WOSnhOziQIA\nYoQxgwAA7EY4ZjDr7puijgUAgKlCN1EAAHaPX04BAA2HlkEAAAAAaEK0DAIAAABAEyIZBAAAAIAm\nRDIIAAAAAE2IZBAAAAAAmhDJIAAAAAA0IZJBAEBTMLN5ZhaYGZ99AACIZBAAUCfM7Edm1jVK+Rlm\n9uw4kzzWUwIAIEQyCACoF9dJOm+U8vMk3eDuQY3jmVJmZlHHAABoLiSDAIB68T1JB5rZKYMFZra/\npPdIuj7c/nMzu9/MXjKzPjO7fLyVm9llZvYbM3vZzB40szNH7P+ImT1UsT8dlh9mZmvN7Hdm9ryZ\nfS0sv9zMbqh4/LBuqmZ2h5l90czuNrOtko40swsqnuM3ZvbRETGcYWaF8PX1mtm7zOwDZrZ+xHGf\nMrN1433tAIDmRDIIAKgL7v6qpDWSzq8oPkfSw+7+YLj9iqS/dPf9JP2FpIvN7H3jfIrfSDrZ3WdL\nukLSjWZ2sCSZ2dmSlkk6L9z/Pkm/DxO7/5D0uKTDJR0q6abKsEe+jBHb50n6sKRZkp6U9FtJfx4+\nx4WSrqlIOk9SuXX0b8PX1ybpCUm3SDrCzI4dUe9143zdAIAmRTIIAKgn10k628z2Crf/UhVJj7vf\n5e4bwvsPqpyYLRhPxe6+1t1/G95fI6lX0knh7oskXe3u94f7H3P3jeH+uZI+4+6vuvtr7v6LCbye\nf3X3R9w9cPft7v4jd38ifI6fS7pd0qnhsR2SrnX3n4X7n3X3kru/Juk7CrvQmtl8SfMk/XACcQAA\nmhDJIACgbrj7f0l6XtKZZnaUpBMlfXtwv5mdZGY/C7tsvihpqaTXj6duMzs/7IK52cw2S5pf8dg3\nSXp0lIe9SVJfFeMVN46I4d1mdo+Z/T6M4d3jiEEqd5P9YHj/PEnfcfdtk4wJANAkSAYBAPXmBkl/\npXLSc5u7P1+x79sqjy081N33l7RS0m4nZjGzwyWtkvRxd5/j7nMkbah47EZJR4/y0I2SDh9jJtOt\nkpIV23NHOWao22jY2nmzpKslvSGM4UfjiEHufq+k18zsVJWTwhtGOw4AgEokgwCAenO9pD9Teazd\nyHFx+0ra7O7bwjF2Hxyxf6zEcKakQNILZpYwswslvaVi/zclfdrMjpckMzvazN4k6ZeSnpX0FTNL\nmtnrzOx/hI8pSmozszeZ2X6SPrub17VXeHvB3QMze7ekd1Xsv1bShWa20MoOGTFO8EZJX5c00a6q\nAIAmRTIIAKgr7t4n6Rcqt7rdMmL3xyVdaWYvSfqCpH8f+fAx6nxY0j9IulfScyp3Eb27Yv/Nkr4k\n6dtm9rKkdZIOCLuHvldSi8oTwGyU9D/Dx/xn+PwPSLpP0g92FYu7vyLpE5LWmNkmSUskfb9i/30q\nTyqzQtJLkrpVnrRm0A0qJ7C0CgIAxsXcp3f9XTNbpPIHV0Llge9Xjdj/PklXqvyL7IDKg/B/Fu57\nQuUPvEDSNnc/SQAAYCdmtrfKs5Ee7+5jjS0EAGDItCaD4RiKkqTTJD2j8i+jS9z9kYpjku7eH95/\nq6R17n5MuP2YpKy7b562IAEAaABm9imVl6X4s6hjAQDUhxnTXP9JknrDLj0ys5sknSFpKBkcTARD\n+0p6oWLbRFdWAAB2ycweD++eGWkgAIC6Mt3J4KEaPm32U9qxZtMQMztT0v+W9EZJp1fsckk/MbMB\nSavc/V+mMVYAAOqSux8ZdQwAgPoz3cnguLj79yR9z8xOUXng++DsaCe7+7Nm9gaVk8KH3f3ukY83\ns+kd+AgAAAAAMefuu11OqdJ0J4NPa/hMZ4eFZaNy97vNbIaZHejuv3f3Z8Py581sncqtijslg+Ex\nUxg2MDW6urrU1dUVdRjATjg3EWecn4grzk3EmdmE8kBJ0z8e7z5Jx5jZvHAx3SUaMQ24mR1dcf94\nSXL334frNe0bls9Uea2lB6c5XgAAAABoCtPaMujuA2Z2iaTbtWNpiYfNbGl5t6+S9H4zO1/Sa5K2\nSjonfPjBktaFXUBnSPqWu98+nfECAAAAQLOY9jGD7v5j7RgDOFi2suL+1ZKuHuVxj0tKT3d8wHRq\nb2+POgRgVJybiDPOT8QV5yYazbQvOl8LZuaN8DoAAAAAYDLMbMITyLCGHwAAAAA0IZJBAAAAAGhC\nJIMAAAAA0IRIBgEAAACgCZEMAgAAAEATIhkEAAAAgCZEMggAAAAATYhkEAAAAACa0IyoAwAAANEK\ngkCFQkGSlMlklEjwWzEANAOSQQAAmlihp6COZR0qzSpJklJbUsotzynTmok4MgDAdDN3jzqGqpmZ\nN8LrAACgloIgUHZxVsV0ccfAkUBKF9PKr8vTQggAdcTM5O42kcdwlQcAoEkVCoVyi2Dlt4GEVJpV\nGuo2CgBoXCSDAAAAANCESAYBAGhSmUxGqS0pKagoDMrjBjMZxgwCQKMjGQQAoEklEgnllueULqaV\n7E0q2ZtUa6FVueU5xgsCQBNgAhkAAJocS0sAQP2bzAQyJIMAAAAAUOeYTRQAAAAAMC4sOg9gWtH9\nDAAAIJ5IBgFMm0JPQR3LOsrrmKk8Q2FueU6ZVmYpBAAAiBpjBgFMiyAIlF2cVTFd3NEhPZDSxbTy\n6/K0EAIA8P+3d/9RcpV1nsc/3zJgrNBNgiMhQUkQugLGobsow+ASOo3s0cysPwgjQ8AZVlrcgL9o\nmTmDgBMC7Jkd2B0TnBU3USrCwAwLZlF2j4MyahMcRUNxqwdDoGKQhhCRKJH86IMkqe/+UdXd1b+S\nqk5X3dt9369z+qTqqXtvPal76977red5vg8wgRgzCCAygiAotQhWnmUSUqGpMNBtFAAAAOEhGAQA\nAACAGCIYBFAX6XRaqT0pqVhRWCyNG0ynGTMIAAAQNoJBAHWRSCSUvTmrtnybkluTSm5NqjVoVfbm\nLOMFAQAAIoAEMgDqiqklAAAA6m88CWQIBgEAAABgkhtPMMg8gwAAAIgsepgA9UMwCGDK40YCACan\noCdQ58rO0lRFKiUhy96cVbqVRGTARKCbKIApjRuJoQiMB/FZRA/7BJWKxaIyyzLKt+UHUx4WpbZ8\nm3IP5jg+gGEYMwgAFbiRGIrAeBCfRfSwTzBcLpdT++p29bX0DSlPbk1q4+c3KpPJhFQzIJrGEwzG\n604IQKwEQVC6saw80yWkQlNhoPUhLorFojpXdirflldfS5/6WvqUb8urc2WnisXi4TcwhfBZRA/7\nBADCQTAIADFAYDyIzyJ62CcYTTqdVmpPSqr8PaBYajVOp2kxBiYCwSCAKYsbCQCYvBKJhLI3Z9WW\nb1Nya1LJrUm1Bq3K3pyNXTd/oF7IJgpgyuq/kagch9Syu0XZW+J3I9EfGOeLQ8dPhhUYh5koJGqf\nBdgnGFu6Na3cgzkSC0UIiZ6ip3Kf1IoEMgCmPC5cJcMTdLTsbtH6W9Y3PEFHFBKFROWzwCD2CRB9\nQbBZnZ1rVSh0SJJSqW5lsyuUTi8MtV5xVnnu7Lu3j2yiAICxhR0YRynDa9ifBUZinwDRVSwWlcl0\nKZ9fo8oTeFtbl3K5NbH8voZ9zhpxTV0lsokCAMaWSCSUyWSUyWRCuXBHKVFI2J8FRmKfANEVBEG5\nRXDoCbxQWBLLRE9BsFmZTJfa23vV3t6rTKZLQbC5wXUY5ZpaI8YMAnUQ9i9FAABg6uH+IhqKxaI6\nO9cOaSXN5y9QZ+fkayWdPDUFJomgJ1BmWUbtq9vVvrpdmWUZBT3x+8UMGA0ZXgFgfMK+v0in00ql\nujX8BJ5KPRq783dUWklHvabWiGAQmEBMnAwcGqniAaB2Ubi/SCQSymZXqLX1ak2ffpumT79NZ5zx\nOWWzKzh/h2T4NXU86CYKTKDDjYfKZDKh1Q2IClLFA0BtInN/kXhDmveY9K5nJUm2b4GU+ERj3jtC\nSq2kdymfv0CVyXRKraTLGluXimvqe+59T83rEwwCABquP1EIAGBy6G+d7GnrGYh/eoo96lzZ2fBs\n0GHrbyXt7OxSobBEktTS0q1s9spQPocjuabGZ68BDcB4KBxKsVhULpdTLpej2zAAoGpRuL+IUjbo\nKEinFyqXW6ONG+dr48b5evLJ20Obb7H//mI8CAaBCcR4KIwl7IH/iC5+JABwONxfRFMUpsOpnOJi\nPFo/nSQAACAASURBVJh0HqgDUj+jUpQmWke0BMFmdXauLWelk1KpbmWzK0L7dRlAtB04cED33Xef\nJGn58uWaNq1xI764lkVPsVhUJtNVMcWFMek8EAVR+KWI1obooGsNRjM4T9WX1Nc3T31985TPf0md\nnWv5ziISuI5ESxBs1qJF12jFiqRWrEhq0aJrGjrJOa2T0TP6FBe1IYEMMAUFPYE6V3aWAhCVxhRk\nb84q3cq4RSAqgiDQlsJJ0kmLpEzpu6pcSlsKl5J9GKEb2Wp9F63WIYrKJOdkg5566CYKTDF044ge\n9glGs2nTJv3Rn/2Z/LLnhxwXdvd8/fT++7Vo0aIwq4cYG9n1TJKKamtrbOCBQblcTu3tverru3BI\neTK5QRs3zufHo5iimyiAEeiSGD10rcGYMttHfFeV2R5WbQBJY3U9S6hQWMJ1BKgQdlfq/iku2tq6\nlExuGNc26CYKAA1A1xoMl0gk9Oajp+l1HRhS/uajp3FsABgiSpOcoyQqQ3L6p7gIgkDvqX3O+fq3\nDJrZUjN7xswKZnbtKK9/2Mx6zCwwsyfM7H3VrgsMF/YvNFEQhbmIMLooJBZCdKTTaZ2277QR39XT\n9p3GdxWhKgUe3Rp+cJYCD47NMAxvAUomN6i19Wplsyu4noSgWCyqc2Wn8m159bX0qa+lT/m2vDpX\ndobWQjjersJ1HTNoZglJBUnnS9ohaZOk5e7+TMUySXfvKz/+Q0kPuvup1axbsQ3GDCIyv9BEwfDP\nomV3i9bfsj6WnwUQZXxXEVWDCWSWSJJaWrq1fv2VJJAJGVNXRUMul1P76nb1tfQNKU9uTWrj5zeG\nNobTrPYxg/UOBs+WdKO7/3H5+RckubvfOsby75W02t3PrmVdgkGQoGMkLhjA5MB3FVHFsQmMbioF\ng/UeM3iipBcrnm+XdNbwhczsAkn/TdIJkj5Qy7ooifsJ+3BJU+KYZetIugwAaBy+q4gqjk1gdP1D\ncvLFoY0Qk3FITiQSyLj7tyR9y8zOlfSPkhbUuo1Vq1YNPO7o6FBHR8dEVS/y6B4JYLKJ+w9YAIAj\nE+Z1pD9L+PBu/tlbGpslvLu7W93d3Ue0jUZ0E13l7kvLzw/ZTbS8zDaVWgBbql03zt1E6R5ZErXP\ngRtdYGwjJ7PuDmUya76n0cM+AVANriOjG0830XrXeJOkU81snpkdLWm5pIcqFzCzUyoenylJ7v7b\natYFc8r1i9I8bkGwWZlMl9rbe9Xe3qtMpktBsLmhdQCiqlgsqrNzrfL5Nerru1B9fRcqn1+jzs61\nDc3Axvc0etgnAKoRleuINDWyhNe1ZVAqTQ8h6XaVwpU73f3vzGyFSq1868zsryVdJukNSfskXePu\nm8Zad4z3iG3LYFQHsIYl7F9oisWiMpku5fNrVNlE2dbWpVxuzaQ9UQATJZfLqb29V319Fw4pTyY3\naOPG+Q05Z/E9jR72CYBqReE6ElVRTCAjd39Yw8YAuvvaise3Sbqt2nUx1FQawDoRwh7sHgRBucvC\n0KbaQmFJbBPZAFHD9zR62CcAEA5+apvkotQ9EgAOh8msAdSqWCwql8spl8uFMqE3ooXryMQiWpgC\n0q1p5R7MaePnN2rj5zfqyW89SSbRkHCCAg4tkUgom12htrYuJZMblExuUGvr1cpmVzTsByy+p9HD\nPhkpCgFQFOrAWFIMF4XryFRS9zGDjRDnMYOInsEMV0skSS0t3Vq//sqGZ7gCoizs8b18T6OHfTIo\nCpkSozBtFWNJcShhX0eiaDxjBgkGgTrgBAVEH9/T6GGfRCMAisp0TVFKFMKxickgilNLALE0FVIN\nA1Md39PoYZ8cPplOw+rAtFUD6KqKqSyeZ1oAAABEWhTGkkZpTjugHggGAQAAysJOmhKFAKh/2qph\nVWj4tFVRSBQShZZaoJ7qPs8g4oP+9ACAySzoCXT5ysv17IxnJUkL9i3Q+pvXNzRpSn8AdPnlV+vZ\nZ98hSUqlXlA2e1XDrqv901ZVJpBp2d2i7C2Nn7YqnV6oXG5Nxf3F7dxfABOIBDKYEFHIOgYAwHgV\ni0WdvvRdKrz32SFJU1I/WaAtDz/d0AAkCkGpxI+8UjQS+gDVIpsoQhGVrGMAAIzXpk2b9Ee3/gf5\nHx4YUm5PTdNPr/2xFi1a1JB6cE2NHqY9wWRBNlGEgqxjADC5hT1OLgqeffZZeXHkbZEfTOjZZ59t\nWD24pkZPf1fVjRvna+PG+XryydsJBCOA89bEIBgEACDGSJtfsmDBAlkwd0TSFMvP1YIFC0KrF6KB\naU+iJegJlFmWUfvqdrWvbldmWUZBDz+WjAfdRHHE6NICAJMT46EGFYtFnf7uj6mwb4t05tZS4ZMt\nSs04XVt+fm/DPguuqdHE+Mno4DsyNrqJIhT9Wcfa8m1Kbk0quTWp1qBV2Zsbn3UMAFA90uYPSiQS\nuu/eL6p11rma/vCNmv7wjTpj5mLdd+8XG3ot45oaPbRCRQtdqScWU0tgQqRb08o9mONXMwDApJVO\nL9STT94e+rWMa2p0FItFda7sHNIKlS/m1bmyM/atUJga6CYKAEBM0U0UOLRcLqf21e3qa+kbUp7c\nmtTGz29UJpMJqWbxRTfRsdFNFABGQcYxYHT9E5y3tXUpmdygZHKDWluvVja7ItY3VACii67UE4uW\nQQBT2uD8UB2SpFSqW9nsCtKCAxVIjgGMjlao6OK8NRKTzgNABbrAAQCOVNATqHNlZylpiaSW3S1a\nf8t6pVvTIdcMGIpgEAAq5HI5tbf3qq/vwiHlyeQGbdw4n7EeAICq0AqFyWA8wSDZRAEAAIBD6J90\nHphq+FkDwJSVTqeVSnVLqkwaU1Qq9ajSabr3AACAeKObKIApbTCBzBJJUktLt9avv5IEMgAAYEph\nzCAAjIKxHgAAYKojGAQAAACAGGLSeQAAAABAVQgGAQAAACCGCAYBAAAAIIYIBgEAAAAghggGAQAA\nACCGCAYBAAAAIIYIBgEAAAAghggGAQAAACCGCAYBAAAAIIamhV0BAEDjFItFBUEgSUqn00ok+E0Q\nAIC44i5giigWi8rlcsrlcioWi2FXB0AEBcFmZTJdam/vVXt7rzKZLgXB5rCrBQAAQmLuHnYdjpiZ\n+VT4f4xXEGxWZ+daFQodkqRUqlvZ7Aql0wtDrReA6CgWi8pkupTPr9Hg74BFtbV1KZdbQwshAACT\nnJnJ3a2mdQ4XRJnZZyXd4+67jqRy9RTnYJAbPADVyOVyam/vVV/fhUPKk8kN2rhxvjKZTEg1AwAA\nE2E8wWA1kcJsSZvM7H4zW2pmNb0B6isIgnKLYOWuTKhQWDIwLggAAAAAhjtsMOjuX5TUIulOSR+X\ntNXM/tbMTqlz3QAAEySdTiuV6pZUOaa4qFTqUaXT6XAqBQAAQlVVH8JyH8yXy38HJM2S9E0zu62O\ndUMVuMEDUI1EIqFsdoXa2rqUTG5QMrlBra1XK5tdQXdyAABiqpoxg1dLukzSbyR9XdK33H2/mSUk\nbXX30FsI4zxmUKpMILNEktTS0q31668kgQyAEZhaAgCAqaleCWRukpR1995RXjvd3bfUVs2JF/dg\nUOIGD9HFsQkAAFB/9QoGz5a02d33lJ83Szrd3X867ppOMIJBIJqY9gQAAKAx6hUMBpLO7I+2yt1D\nn3D3M8dd0wlGMAhED9OeAAAANE69ppYYEmm5e1HStForByBemPZkpGKxqFwup1wup2KxePgVAAAA\n6qiaYPA5M/ucmR1V/rta0nP1rhgATCVBsFmZTJfa23vV3t6rTKZLQbA57GoBAIAYq6ab6PGSvizp\nfZJc0vcldbn7K/WvXnXoJgqMFHbiFrqJDuKzAAAA9VaXbqLu/oq7L3f34919trtfGqVAENFBF7jo\niEIrFPPaDaLLLAAAiKLDjv0zs+mSPiFpoaTp/eXu3lnHemGSGZk18i6yRoakWCyqs3PtkFaofP4C\ndXY2vhUqnV6oXG5NRQvl7bELBAEAAKKqmruyf5R0gqQPSHpU0tsl7alnpTC5VAYffX0Xqq/vQuXz\na9TZuZYWwhBErRUqkUgok8kok8nENhBMp9NKpbolVX4fikqlHlU6nQ6nUgAAIPaquTM71d3/RtI+\nd79L0n+S9Ef1rRYmk6gFH0DU0GUWAABEUTVTROwv//s7M3u3pJclHV+/KgE4EqVWqLuUz1+gymQl\npVaoZWFWLdboMgsAAKKmmjuRdWY2S9IXJT0k6WlJt1b7Bma21MyeMbOCmV07yuuXmllP+e9HZnZG\nxWvPl8sDM/tZte+JxqILXLTQChVddJkFAABRcsipJcwsIemj7n7/uDZeWr8g6XxJOyRtkrTc3Z+p\nWOZsSVvc/TUzWypplbufXX7tOUkZd991mPdhaomQDSaQWSJJamnp1vr1V5JAJkRhTy0BAACAxhnP\n1BLVzDP4hLu/Z5wVOlvSje7+x+XnX5Dk7j5qy6KZzZT0lLu/o/z8l5Le4+6/Pcz7EAxGAMEHAAAA\nEI7xBIPVjBn8VzP7K0n/W9K+/kJ3f7WKdU+U9GLF8+2SzjrE8ldI+peK5y7pETM7KGmdu3+tivdE\nSPq7wAEAAACIvmqCwYvL/366oswlvXMiK2Jm50m6XNLiiuJz3P1XZvY2lYLCLe7+o9HWX7Vq1cDj\njo4OdXR0TGT1AAAAACAyuru71d3dfUTbOGw30SPaeKmb6Cp3X1p+Pmo30XLSmA2Slrr7tjG2daOk\nPe7+pVFeo5soAAAAgNiqSzdRM7tstHJ3v7uK7W+SdKqZzZP0K0nLJV0ybPsnqRQI/kVlIGhmSUkJ\nd99rZjMkvV/STVW8Z0MxTi5a2B8AAABAdarpJrqo4vF0lTKDPinpsMGgux80s89I+p5K01jc6e5b\nzGxF6WVfJ+lvJB0n6Q4zM0n73f0sSbMlPWhmXq7nve7+vRr+b3UX9ATqXNmpQlNBkpTak1L25qzS\nrUynEIbBjKYdkqRU6i5lsyvIaAoAAACMouZuouWMn/f1d/2MgjC6iRaLRWWWZZRvy1fO6622fJty\nD+ZokWqwYrGoTKZL+fwaVe6QtrYu5XJr2B8AAACY0sbTTXQ8d8j7JJ08jvWmlCAISi2ClZ9gQio0\nFQa6KaJxgiAotwgO3SGFwhL2BwAAADCKasYM/l+VsodKpTvtd0ka1yT0AAAAAIBoqGbM4P+oeHxA\nUq+7b69TfSaNdDqt1J6U8sWh3URTe1JKpxkz2GjpdFqp1F3K5y9Q5Q5JpR5VOr0szKoBAAAAkXTY\nMYNmdrKkX7n76+Xnb5E0292fr3/1qhPW1BLDE8i07G7R+lvWk0AmJIMJZJZIklpaurV+/ZUkkAEA\nAMCUN54xg9UEg09I+g/u/kb5+dGS/s3dFx1yxQYKc55BpjKIFvYHAAAA4qhewWDe3duGlfW4e+s4\n6lgXTDoPAAAAIM7qlU10p5l9uOJNPiLpN7VWDgAAAAAQHdW0DJ4i6V5Jc8tF2yVd5u6/qHPdqkbL\nIAAAAIA4q0s30YqNHyNJ7r53HHWrK4JBAAAAAHFWl26iZva3ZjbT3fe6+14zm2Vm/3X81QQAAAAA\nhK2aMYN/7O6/63/i7rsk/Un9qgQAAAAAqLdqgsE3mdmb+5+U5xl88yGWBwAAAABE3LQqlrlX0vfN\nbL0kk/RxSXfVs1IAAAAAgPqqKoGMmS2V9B8luaTdkk5w90/XuW5VI4EMAAAAgDir1zyDkvRrlQLB\niyS9T9KWGusGAAAAAIiQMbuJmllK0iWSlkt6RdIDKrUkntegugEAAAAA6mTMbqJmVpT0/yR92t1f\nLJc95+7vbGD9qkI3UQAAAABxNtHdRC+U1Cdpo5n9LzN7n0oJZFChWCwql8spl8upWCyGXR0AAAAA\nqMqYwaC7f8vdl0t6t6SNkj4v6Xgz+6qZvb9RFYyyINisTKZL7e29am/vVSbTpSDYHHa1AAAAAOCw\nqsomOrCw2SyVkshc7O7n161WNQqjm2ixWFQm06V8fo0GY+qi2tq6lMutUSJRbW4eAAAAADgy9cwm\nKkly913uvi5KgWBYgiBQodChoR9hQoXCEgVBEFKtAAAAAKA6NF8BAAAAQAwRDI5TOp1WKtUtqTJp\nTFGp1KNKp9PhVAoAAAAAqlTTmMGoCmtqiSDYrM7OtSoUlkiSWlq6tX79lUqnFza8LgAAAADiazxj\nBgkGj1CxWBwYI5hOp0kcAwAAAKDhCAYBAAAAIIbqnk0UAAAAADA1EAwCAAAAQAwRDAIAAABADBEM\nAgAAAEAMEQwCAAAAQAwRDAIAAABADBEMAgAAAEAMEQwCAAAAQAwRDAIAAABADBEMAgAAAEAMEQwC\nAAAAQAwRDAIAAABADBEMAgAAAEAMEQwCAAAAQAwRDAIAAABADBEMAgAAAEAMEQwCAAAAQAwRDAIA\nAABADBEMAgAAAEAMEQwCAAAAQAwRDAIAAABADBEMAgAAAEAMEQwCAAAAQAwRDAIAAABADBEMAgAA\nAEAMEQwCAAAAQAzVPRg0s6Vm9oyZFczs2lFev9TMesp/PzKzM6pdFwAAAAAwPubu9du4WUJSQdL5\nknZI2iRpubs/U7HM2ZK2uPtrZrZU0ip3P7uadSu24fX8fwAAAABAlJmZ3N1qWafeLYNnSdrq7r3u\nvl/SfZI+UrmAuz/u7q+Vnz4u6cRq1wUAAAAAjE+9g8ETJb1Y8Xy7BoO90Vwh6V/GuS4AAAAAoErT\nwq5APzM7T9LlkhaPZ/1Vq1YNPO7o6FBHR8eE1AsAAAAAoqa7u1vd3d1HtI16jxk8W6UxgEvLz78g\nyd391mHLnSFpg6Sl7r6tlnXLrzFmEAAAAEBsRXHM4CZJp5rZPDM7WtJySQ9VLmBmJ6kUCP5FfyBY\n7boAAAAAgPGpazdRdz9oZp+R9D2VAs873X2Lma0ovezrJP2NpOMk3WFmJmm/u5811rr1rC8AAAAA\nxEVdu4k2Ct1EAQAAAMRZFLuJAgAAAAAiiGAQAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQAAAAAGKI\nYBAAAAAAYohgEAAAAABiiGAQAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQAAAAAGKIYBAAAAAAYohg\nEAAAAABiiGAQAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQ\nAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQAAAAAGKIYBAAAAAAYohgEAAAAABiiGAQAAAAAGKIYBAA\nAAAAYohgEAAAAABiaFrYFQAAAEBt5s+fr97e3rCrASAE8+bN0/PPPz8h2zJ3n5ANhcnMfCr8PwAA\nAKphZuLeB4insb7/5XKrZVt0EwUAAACAGCIYBAAAAIAYIhgEAAAAgBgiGAQAAACAGCIYBAAAwJTT\n29urRCKhYrEYdlUQMRwbgwgGAQAApohisahcLqdcLjfuG90j3cauXbu0bNkyHXPMMTr55JP1z//8\nz4dcfvXq1ZozZ45mzpypK664Qvv37x/3toYzqymxYsMkEgk999xzDX3PyXZsXHXVVWpqalJzc7Oa\nm5s1ffp0HXvssQOvd3R06C1veYuam5vV1NSk008/vaa6cGyU369h7wQAAIC6CYLNymS61N7eq/b2\nXmUyXQqCzQ3fxqc+9SlNnz5dO3fu1D333KOrrrpKW7ZsGXXZ7373u7rtttv0wx/+UL29vdq2bZtu\nvPHGcW2rVmFOzdHoQCToCZRZllH76na1r25XZllGQU/Q8G3Usj+/+tWvas+ePdq9e7d2796tSy65\nRBdddNHA62amO+64Q7t379aePXsm7LiQ4nVsyN0n/V/pvwEAABAPw+99Dh486G1tn3XpoEte/iuV\nHTx4sKptTsQ29u3b50cffbT/4he/GCi77LLL/Lrrrht1+UsvvdRvuOGGgec/+MEP/IQTThjXtg4e\nPOh/+Zd/6X/wB3/gp5xyin/lK1/xRCIxUPeOjg6/4YYb/JxzzvFkMunbtm3zHTt2+Ic//GE/7rjj\nvKWlxb/2ta8NbG/VqlX+0Y9+1C+++GJvamryTCbjPT09A69v2bLFOzo6fObMmf7ud7/bH3rooYHX\nOjo6/M477xx4/o1vfMMXL17s7u7t7e1uZj5jxgxvamry+++///Af7BE4ePCgt324zbVSrlXlv5Xy\ntg+31XZsHOE2at2flfbu3etNTU3+2GOPDZQN/4wPV/+pdGyMFfuUy2uKo2gZBAAAmOSCIFCh0KGh\nnb4SKhSWKAiqa72ZiG0UCgUdddRROuWUUwbKWltbtXnz6K2LmzdvVmtr65BlX3nlFe3atavmba1b\nt07f+c531NPToyeeeELf/OY3Ryxzzz336Otf/7r27Nmjk046ScuXL9dJJ52kl19+WQ888ICuv/56\ndXd3Dyz/0EMP6eKLL9auXbt0ySWX6IILLtDBgwd14MABfehDH9LSpUu1c+dOffnLX9bHPvYxbd26\ndczPpr/F59FHH5UkPfXUU9q9e/eQ1q56CIJAhabC8N2qQlOhtmPjCLdR6/6stGHDBh1//PFavHjx\nkPLrrrtOxx9/vM4999yBz3U0HBtjIxgEAADAhNi7d6+am5uHlDU3N2vPnj1jLl85Dqy5uVnurj17\n9tS8rQceeEBdXV2aO3euZs6cqeuuu27EMh//+Md12mmnKZFI6OWXX9aPf/xj3XrrrTrqqKPU2tqq\nK664QnfffffA8plMRsuWLdOb3vQmXXPNNfr973+vxx9/XI8//rj27duna6+9VtOmTdN5552nD37w\ngzWNafQQuyKGodb9Wenuu+/WZZddNqTstttu03PPPaeXXnpJn/zkJ/WhD31Iv/zlL0ddn2NjbASD\nAAAAk1w6nVYq1S2pMqlHUanUo0qn0w3bxjHHHKPdu3cPKXvttdfU1NRU1fKvvfaazExNTU01b2vH\njh16xzveMfB83rx5I5apfH3Hjh067rjjlEwmh6zz0ksvjbq8menEE0/Ujh07RrzXaOtGRTqdVmpP\navhuVWpPqrZj4wi3Uev+7PfCCy+ou7t7RDC4aNEizZgxQ0cddZQuu+wynXPOOfrOd74z6jY4NsZG\nMAgAADDJJRIJZbMr1NbWpWRyg5LJDWptvVrZ7AolEtXd7k3ENlKplA4cOKBt27YNlPX09GjhwoWj\nLr9w4UL19PQMPM/n85o9e7ZmzZpV87bmzJmjF198ceB5b2/viGUqk3PMnTtXr776qvbt2zdQ9sIL\nL+jEE08ceF65PXfX9u3bNXfuXM2dO1cvvPDCkG1Xrjtjxgz19fUNvPbyyy+PWudGSCQSyt6cVVu+\nTcmtSSW3JtUatCp7c7a2Y+MIt1Hr/ux3zz33aPHixZo/f/4hlzOzMVvUODYOodZBhlH8EwlkAABA\njIx173Pw4EF/4okn/Iknnqg6scdEb+OSSy7xSy+91Pft2+ePPfaYz5w5059++ulRl3344Yd9zpw5\n/vTTT/urr77qHR0dfv31149rW1/96ld94cKFvn37dn/11Vf9/PPPH5EkZHjCkfb2dv/sZz/rr7/+\nuvf09Pjs2bP9Bz/4gbuXkoQcffTR/uCDD/qBAwf87//+7/3kk0/2AwcO+BtvvOGnnHKK33rrrb5/\n/37/4Q9/6M3NzV4oFNzd/YYbbvDzzjvP+/r6fOvWrd7S0uLnnnvuwPvOmTPHH3nkkZo/2yMx2Y6N\nfgsWLPBvfOMbQ8p+97vf+Xe/+11//fXX/cCBA37PPff4Mccc41u3bh11G1Pt2Bjr+69xJJAJPZCb\niD+CQQAAECdRvvd59dVX/YILLvAZM2b4vHnz/L777ht47YUXXvCmpiZ/8cUXB8pWr17ts2fP9mOP\nPdY/8YlP+BtvvFHVtoY7cOCAX3PNNf7Wt77V3/nOd/odd9wx5Ib/vPPOG3HD/9JLL/kHP/hBP+64\n4/zUU0/1devWDby2atUqv+iii3z58uXe1NTkZ555pufz+YHXn376aV+yZIkfe+yxvnDhQv/2t789\n8NpvfvMbf//73+/Nzc2+ePFiv+mmm4bc8K9du9bnzJnjs2bN8gceeKCWj3dSq/XY+MlPfuLHHHOM\n7927d8h2du7c6YsWLfLm5mafNWuWv/e97/Xvf//7Y77vVDs2JjIYtNJ6k5uZ+VT4fwAAAFTjUF3i\nMDFuuukmbdu2bUjSEEAK/9gY6/tfLq9posIpM2awWCwefiEAAAAAgKQpFAxmlmUU9FQ3zwkAAAAA\nxN2U6SaqlVJbvk25B3NVZzUCAACYjOgmCsQX3URHk5AKTQUFAa2DAAAAAHA4UycYBAAAAABUbeoE\ng0UptSeldDoddk0AAAAAIPKmhV2BidIatCp7S5bxggAAYMqbN2+ezGoaGgRgipg3b96EbWvKJJDZ\nv3+/pk2bMrEtAAAAAFQtkglkzGypmT1jZgUzu3aU1xeY2Y/N7HUzu2bYa8+bWY+ZBWb2s0O9z6JF\n1ygINk909YEj0t3dHXYVgFFxbCLKOD4RVRybmGrqGgyaWULS/5T0AUkLJV1iZqcNW+y3kj4r6b+P\nsomipA53T7v7WYd6r3x+jTo71zL5PCKFiwaiimMTUcbxiaji2MRUU++WwbMkbXX3XnffL+k+SR+p\nXMDdf+PuOUkHRlnfqq9jQoXCEqaWAAAAAIAq1DsYPFHSixXPt5fLquWSHjGzTWb2yQmtGQAAAADE\nWF0TyJjZn0r6gLv/l/LzP5d0lrt/bpRlb5S0x92/VFE2x91/ZWZvk/SIpM+4+49GWXfyZ8EBAAAA\ngCNQawKZeqfffEnSSRXP314uq4q7/6r8704ze1ClbqcjgsFa/9MAAAAAEHf17ia6SdKpZjbPzI6W\ntFzSQ4dYfiCoM7OkmR1TfjxD0vsl/byelQUAAACAuKhry6C7HzSzz0j6nkqB553uvsXMVpRe9nVm\nNlvSE5KaJBXN7GpJ75L0NkkPlruATpN0r7t/r571BQAAAIC4mBKTzgMAAAAAalP3Sefr6XATetR8\nmAAABLNJREFU2gNhMrPnzazHzAIz+1nY9UF8mdmdZvZrM/v3irJZZvY9M3vWzL5rZseGWUfE1xjH\n541mtt3Mniz/LQ2zjognM3u7mf3AzDab2VNm9rlyOedPhGqUY/Oz5fKaz52TtmWwPKF9QdL5knao\nND5xubs/E2rFgDIze05Sxt13hV0XxJuZLZa0V9Ld7n5GuexWSb9199vKP6bNcvcvhFlPxNMYx+eI\nDONAo5nZCZJOcPd8OY9FTqX5si8X50+E6BDH5sWq8dw5mVsGDzuhPRAy0+T+jmGKKE/JM/xHiY9I\nuqv8+C5JFzS0UkDZGMenVJFUDgiDu7/s7vny472StqiUGZ/zJ0I1xrHZP5d7TefOyXyjeqQT2gP1\n5pIeMbNNZvbJsCsDDHO8u/9aKl1UJB0fcn2A4T5jZnkz+zrd8BA2M5svqU3S45Jmc/5EVFQcmz8t\nF9V07pzMwSAQdee4+5mS/kTSp8tdoYCompxjBjBV3SHpne7eJullSXQXRWjK3fC+KenqcivM8PMl\n50+EYpRjs+Zz52QOBo9oQnug3tz9V+V/d0p6UKWuzUBU/Lo8tU//2INXQq4PMMDdd/pgUoOvSVoU\nZn0QX2Y2TaWb7X9092+Xizl/InSjHZvjOXdO5mCw1gntgYYxs2T51xqZ2QxJ75f083BrhZgzDR1H\n8JCkj5cf/2dJ3x6+AtBAQ47P8g12vwvF+RPhyUp62t1vryjj/IkoGHFsjufcOWmziUqlqSUk3a7B\nCe3/LuQqAZIkMztZpdZAlzRN0r0cnwiLmf2TpA5Jb5X0a0k3SvqWpAckvUNSr6Q/c/ffhVVHxNcY\nx+d5Ko2BKUp6XtKK/jFaQKOY2TmSNkp6SqXruUu6XtLPJN0vzp8IySGOzUtV47lzUgeDAAAAAIDx\nmczdRAEAAAAA40QwCAAAAAAxRDAIAAAAADFEMAgAAAAAMUQwCAAAAAAxRDAIAAAAADFEMAgAQJmZ\nHTSzJ80sKP/71xO47Xlm9tREbQ8AgCM1LewKAAAQIfvc/cw6bp/JfQEAkUHLIAAAg2zUQrNfmtmt\nZvbvZva4mb2zXD7PzL5vZnkze8TM3l4uP97M/k+5PDCzs8ubmmZm68zs52b2sJm9uUH/LwAARiAY\nBABg0FuGdRO9qOK1Xe5+hqSvSLq9XPYPkta7e5ukfyo/l6QvS+oul58paXO5vEXSP7j7uyW9JulP\n6/z/AQBgTOZOjxUAACTJzHa7e/Mo5b+UdJ67P29m0yT9yt3fZmY7JZ3g7gfL5Tvc/Xgze0XSie6+\nv2Ib8yR9z90XlJ//taRp7v63DfnPAQAwDC2DAABUx8d4XIvfVzw+KMbuAwBCRDAIAMCgUccMll1c\n/ne5pJ+UH/+bpEvKj/9c0mPlx/8q6VOSZGYJM+tvbTzU9gEAaCh+kQQAYNB0M3tSpaDNJT3s7teX\nX5tlZj2SXtdgAPg5SevN7K8k7ZR0ebm8S9I6M/uEpAOSrpL0ssgmCgCIEMYMAgBwGOUxgxl3fzXs\nugAAMFHoJgoAwOHxyykAYMqhZRAAAAAAYoiWQQAAAACIIYJBAAAAAIghgkEAAAAAiCGCQQAAAACI\nIYJBAAAAAIih/w9gLAtJAnFjHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b78a790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "    solver = solvers[dropout]\n",
    "    train_accs.append(solver.train_acc_history[-1])\n",
    "    val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Accuracy Improvement\n",
      "dropout=0.00: 0.3070    dropout=0.75: 0.3140 \n",
      "improvement: 2.28%\n"
     ]
    }
   ],
   "source": [
    "print \"Validation Accuracy Improvement\"\n",
    "print \"dropout=%.2f: %.4f    dropout=%.2f: %.4f \" % (dropout_choices[0], val_accs[0], dropout_choices[1], val_accs[1])\n",
    "print \"improvement: %.2f%%\" % (((val_accs[1] - val_accs[0]) / val_accs[0]) * 100.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question\n",
    "Explain what you see in this experiment. What does it suggest about dropout?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Answer\n",
    "Dropout seems to decrease the training accuracy as expected since it drops the some the neurons' activation during training. However, this leads to less overfitting in the training phase, as seen above in the val accuracy plot. It can be seen that the validation accuracy is greatly increased within the first few epochs, but with more training, the validation accuracies seem to converge to the same value. This seems to suggest that dropout, similar to batch normalization, increases the speed of training and convergence to an optimal solution for the optimization problem."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}