tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Boosting Reinforcement Learning (GBRL)\n",
    "GBRL is a Python-based GBT library designed and optimized for reinforcement learning (RL).\n",
    "GBRL is designed to be integrated in popular RL python libraries as part of the standard RL training loop.  \n",
    "\n",
    "## GBRL Design\n",
    "The standard GBT supervised learning training procedure for K boosting iterations on a given set on inputs x and targets y is as follows:  \n",
    "***For K boosting iterations***   \n",
    "1. Generate predictions using current GBT ensemble.\n",
    "2. Calculate loss L(y, predictions).\n",
    "3. Calculate gradients of the loss function w.r.t predictions  \n",
    "4. Fit a binary decision tree on the gradients and add it to the ensemble.\n",
    "5. Repeat from step 1.\n",
    "\n",
    "The training procedure is typically done end-to-end (E2E) within a GBT framework and is optimized for pre-defined loss functions.  However, E2E learning is not optimal for RL settings, where the training procedure requires interactions with an environment.\n",
    "\n",
    "GBRL draws inspiration from PyTorch and is designed to be easily integrated into RL or other online learning settings. As such, GBRL can be viewed as a functional optimizer that incrementally adjusts a learned function by taking steps in the direction of the gradient.\n",
    "\n",
    "## Get Started with GBRL\n",
    "This is a quick tutorial demonstrating usage examples\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Usage: Training, Saving, Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch as th\n",
    "import gymnasium as gym \n",
    "\n",
    "from sklearn import datasets\n",
    "from torch.nn.functional import mse_loss \n",
    "from torch.distributions import Categorical\n",
    "\n",
    "from gbrl import cuda_available\n",
    "from gbrl.gbt import GBRL \n",
    "from gbrl.ac_gbrl import ActorCritic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# toy dataset\n",
    "X_numpy, y_numpy = datasets.load_diabetes(return_X_y=True, as_frame=False, scaled=False)\n",
    "\n",
    "out_dim = 1 if len(y_numpy.shape) == 1  else y_numpy.shape[1]\n",
    "input_dim = X_numpy.shape[1]\n",
    "\n",
    "X, y = th.tensor(X_numpy, dtype=th.float32), th.tensor(y_numpy, dtype=th.float32)\n",
    "# CUDA is not deterministic\n",
    "device = 'cuda' if cuda_available else 'cpu'\n",
    "\n",
    "# initializing model parameters\n",
    "tree_struct = {'max_depth': 4, # maximum tree depth\n",
    "               'n_bins': 256, # maximum number of candidates to generate per numeric feature\n",
    "               'min_data_in_leaf': 0, # minimum samples in each child when splitting\n",
    "               'grow_policy': 'oblivious' # tree growth policy: [oblivious, greedy]\n",
    "        }\n",
    "\n",
    "optimizer = { 'algo': 'SGD', # optimizer algorithm: [SGD/Adam], Adam is available for cpu only  \n",
    "              'lr': 1.0, # learning rate\n",
    "              'start_idx': 0,\n",
    "              'stop_idx': out_dim\n",
    "            }\n",
    "gbrl_params = {\n",
    "               \"split_score_func\": \"Cosine\", # split score function: [L2, Cosine]\n",
    "               \"generator_type\": \"Quantile\" # candidate generation policy - [Uniform, Quantile]. \n",
    "                }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting up model\n",
    "gbrl_model = GBRL(input_dim=input_dim,\n",
    "                  output_dim=out_dim,\n",
    "                  tree_struct=tree_struct,\n",
    "                  optimizer=optimizer,\n",
    "                  gbrl_params=gbrl_params,\n",
    "                  verbose=0,\n",
    "                  device=device)\n",
    "# set GBRL bias\n",
    "gbrl_model.set_bias_from_targets(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse_incremental_step(model, X, y):\n",
    "    # forward pass - y_pred is a torch tensor with requires_grad = True by default\n",
    "    y_pred = model(X)\n",
    "    # calculate loss - we must scale pytorch's mse loss function by 0.5 to get the correct MSE gradient\n",
    "    loss = 0.5*mse_loss(y_pred, y) \n",
    "    loss.backward()\n",
    "    # perform a boosting step\n",
    "    model.step()\n",
    "    # return RMSE\n",
    "    return loss.sqrt().item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function to plot results\n",
    "def plot_loss(n_iterations, losses, x_label, y_label):\n",
    "    plt.close('all')\n",
    "    _, ax = plt.subplots()\n",
    "\n",
    "    ax.plot(np.arange(n_iterations), losses)\n",
    "    ax.set_xlabel(x_label)\n",
    "    ax.set_ylabel(y_label)\n",
    "    plt.show()\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAALEwAACxMBAJqcGAAAJppJREFUeJzt3Xl8VfWd//HXJwkJyc2+AAkQAoIgIiCgqLigbR2rVv3VVjvaSjtO/XU6rZ3aGVtnOtPlN52pXWztrx3r1lat+rNFccGlWveFRRbZZRcIBEJWsockn98f94SmSEKA3Nzk3vfz8biPe++59577ORx953u/55zv19wdERGJHwnRLkBERPqXgl9EJM4o+EVE4oyCX0Qkzij4RUTiTFK0C+iN/Px8LykpiXYZIiKDyvLlyyvcveDw5YMi+EtKSli2bFm0yxARGVTMbMeRlqurR0Qkzij4RUTijIJfRCTOKPhFROKMgl9EJM4o+EVE4oyCX0QkzsR08C9YWcrvFx/xNFYRkbgV08G/cFUZjyzZGe0yREQGlJgO/lBKEo2tbdEuQ0RkQInx4E+kobU92mWIiAwoMR38aclJNLaoxS8i0lVMB38oJYmG1nY6OjSvsIhIp9gO/uREAJoOqrtHRKRTTAd/Wkp41OkGHeAVETkkpoO/s8Xf0KIWv4hIp9gO/s4Wvw7wiogcEtvBnxwO/kad0ikickhMB39aSmdXj1r8IiKdYjr403VwV0TkQ2I6+NOCg7uNOrgrInJITAd/Zx9/vbp6REQOie3gT+k8uKvgFxHplBTJlZvZB0Ad0A60ufssM8sFHgNKgA+Aa9y9OhLfn5yUwJBE00BtIiJd9EeL/0J3n+7us4Ln3wJedvcJwMvB84hJS07SWT0iIl1Eo6vnSuCB4PEDwFWR/LL0lCRduSsi0kWkg9+BF81suZndFCwb7u5lweO9wPAjfdDMbjKzZWa2bP/+/cddQFpyovr4RUS6iGgfP3Cuu+82s2HAS2b2ftcX3d3N7IhjJrv7PcA9ALNmzTrucZXTUpJ0Vo+ISBcRbfG7++7gvhxYAJwJ7DOzQoDgvjySNaSnJGrIBhGRLiIW/GYWMrOMzsfAxcBa4GlgXvC2ecBTkaoBdHBXRORwkezqGQ4sMLPO73nE3V8ws3eBP5jZjcAO4JoI1kAoOVFDNoiIdBGx4Hf3bcC0IyyvBD4Sqe89XCglSUM2iIh0EdNX7kLnvLtq8YuIdIr54E9LTqT5YAftmnBdRASIg+DvHKhNrX4RkbDYD/7OgdrUzy8iAsRF8AezcKnFLyICxEHwpyVrwnURka5iPvgPtfjV1SMiAsRD8CdrMhYRka5iP/iDFr8GahMRCYuD4O9s8aurR0QE4iD4dXBXROSvxXzwh5J1cFdEpKuYD/6kxARSkhJ0cFdEJBDzwQ8aqE1EpKu4CP605ER19YiIBOIi+NNTNAuXiEinuAj+tGTNuysi0ikugj+UkqQLuEREAvER/MlJOqtHRCQQF8GflqKDuyIineIi+EPJOp1TRKRTXAR/WkqiZuASEQnERfCnJyfR2t5Ba1tHtEsREYm6uAj+tGCEziad0ikiEh/B3zlQW736+UVE4iT4O8fk17n8IiLxEvzB0Mzq6hERiY/g12QsIiJ/ERfBn56i4BcR6RQXwZ8WHNzVQG0iInES/J0HdzVQm4hIPwS/mSWa2UozWxg8H2tmS8xsi5k9ZmbJka7h0Fk9Op1TRKRfWvxfAzZ0eX478DN3Hw9UAzdGuoDUIZpwXUSkU0SD38xGAZcB9wXPDbgImB+85QHgqkjWAJCYYKQOSdTBXRERIt/i/zlwK9A5SE4eUOPunQlcCow80gfN7CYzW2Zmy/bv33/ChYQnXFeLX0QkYsFvZpcD5e6+/Hg+7+73uPssd59VUFBwwvWEUhLVxy8iAiRFcN1zgCvM7FJgKJAJ3Alkm1lS0OofBeyOYA2HpCVrwnUREYhgi9/db3P3Ue5eAnwGeMXdrwdeBT4VvG0e8FSkaugqlKxZuEREIDrn8X8TuMXMthDu87+/P740lKJ5d0VEILJdPYe4+2vAa8HjbcCZ/fG9XYVSEimtVvCLiMTFlbsAo3PS2FnVSG3TwWiXIiISVXET/JdMGcHBduel9fuiXYqISFTFTfBPH53NyOxUFq7eE+1SRESiKm6C38y4fGohb22uoLqhNdrliIhETdwEP8DlU4to63BeXL832qWIiERNXAX/lJGZjMlLY+HqsmiXIiISNUcNfjObY2ah4PFnzewOMxsT+dL6nplx2WmFvLO1ksr6lmiXIyISFb1p8d8FNJrZNOAbwFbgwYhWFUGXTy2ivcN5YZ26e0QkPvUm+Nvc3YErgV+6+6+AjMiWFTmnFGYwriDEM6t0do+IxKfeBH+dmd0GfBZ41swSgCGRLStyzIxPzxzN4m1VvKhWv4jEod4E/7VAC3Cju+8lPKLmjyNaVYTdeO5YJhdmctsTa6hQX7+IxJletfiBO939TTM7GZgOPBrRqiIsOSmBn107nbrmNm57Yg3hniwRkfjQm+B/A0gxs5HAi8DngN9Fsqj+MHFEBv/yNxN5af0+5i8vjXY5IiL9pjfBb+7eCHwS+B93/zQwJbJl9Y8bzx3L7LG5fP+Z9bqaV0TiRq+C38zOBq4Hnj2Gzw14CQnG96+cQl1LG79954NolyMi0i96E+D/BNwGLHD3dWY2jvAsWjFh4ogM/ubU4fzu7e3UNWvIZhGJfUcNfnd/3d2vAH5lZunuvs3db+6H2vrNVy6cwIHmNh5avCPapYiIRFxvhmw4zcxWAuuA9Wa23MxOjXxp/ee0UVlccHIB97+5naZWzcsrIrGtN109dwO3uPsYdy8mPGzDvZEtq/995aLxVDa08ujSndEuRUQkonoT/CF3P9SnH8yfG4pYRVFyRkkus8fmcvcbW3WGj4jEtN4E/zYz+3czKwlu3wa2RbqwaLj1kklUNx7kM/csZn+drugVkdjUm+D/O6AAeCK4FQTLYs7MMTn89vNnsLOqkWvvWcTe2uZolyQi0ud6c1ZPtbvf7O4zgtvX3L26P4qLhjnj83nwxjMpP9DCNXcvYmdlY7RLEhHpU9bdODVm9gzQ7SA2wSme/WLWrFm+bNmy/vo6AFbtqmHeb5eSlJDAA393BqcWZfXr94uInCgzW+7usz60vIfgv6CnFbr7631U21FFI/gBtpTXccP9S6lrbuOeG2Zx9kl5/V6DiMjxOubgH0iiFfwAe2qauOE3S9lZ2cjTX53DpBGZUalDRORYdRf8MTHmTiQVZafy2E1nMSTRuOu1rdEuR0TkhCn4eyEvPYXrZhezcHUZu6p0sFdEBjcFfy/93bljSTC4782YvIRBROJI0tHe0M3ZPbXAMuBud4+Lk90Ls1K5cvpIHlu2i5s/MoG89JRolyQiclx6deUuUE94fJ57gQOEp2M8mRgcs6cnX7pgHM0HO3hgkUbxFJHB66gtfuAcdz+jy/NnzOxddz/DzNZFqrCBaPywDD56ynAeXPQBX7pgHGnJvfnnExEZWHrT4k83s+LOJ8Hj9OBpt6OZmdlQM1tqZqvMbJ2ZfS9YPtbMlpjZFjN7zMyST2gL+tk/zD2JmsaDfOvxNXR0DPxTYUVEDteb4P8G8JaZvWpmrwFvAv9sZiHggR4+1wJc5O7TgOnAJWZ2FnA78DN3Hw9UAzeeQP39buaYHL55ySSeXrWH7y9cz2C4DkJEpKuj9lW4+3NmNgGYFCza2OWA7s97+JwTPjYAMCS4OXARcF2w/AHgu8Bdx1p4NH3pgnFU1Ldw/1vbyU9P5isXTYh2SSIivdbbTuqZQEnw/mlmhrs/eLQPmVkisBwYD/wK2ArUuHtb8JZSYGQ3n70JuAmguLj4SG+JGjPj3y49haqGVn7y4iaW7ajmimlFXHzqCNJT1O8vIgNbb07nfAg4CXgP6JyX0IGjBr+7twPTzSwbWMBffjUclbvfA9wD4SEbevu5/pKQYPzoU1MZnZPK4yt2c8sfVpGStIb/+MRkrp89JtrliYh0qzfN01nAZD+Bzmx3rzGzV4GzgWwzSwpa/aOA3ce73mgbkpjALRdP5OsfO5kVO6v5+Z83828L1hJKTuKq04/4Q0ZEJOp6c3B3LTDiWFdsZgVBSx8zSwU+BmwAXgU+FbxtHvDUsa57oDEzZo7J5d4bZnH2uDy+8cdV/Hn9vmiXJSJyRL0J/nxgvZn9ycye7rz14nOFwKtmthp4F3jJ3RcC3wRuMbMtQB5w//EWP9AMHZLIvfNmMaUoky8/soLXN+2PdkkiIh9y1GGZuxuXPx7G4z9e1Q2t/O29i9lcXs+3LzuFz59TgplFuywRiTPdDcvcm9M5+y3gY0VOKJn5/3AOtzz2Ht97Zj0byg7wf66aQkpSYrRLExHpvqvHzN4K7uvM7ECXW52ZHei/Egen9JQkfv3Zmdx80Xj+sKyU//Wrd1i3pzbaZYmIdB/87n5ucJ/h7pldbhnurmmoeiEhwbjl4once8MsyutauPKXb/PTFzfS0tZ+9A+LiERIr642Ci7EGt71/e6+M1JFxZqPTR7OGSU5fH/hev7vK1v43TsfcPa4PM6bkM/HTyskX0M8i0g/6s3B3a8C3wH2AR3BYnf3qRGu7ZDBdnC3J29vqeCZVXt4c3MFu2uaGJWTynNfO4/MoUOiXZqIxJjjPrgLfA2Y6O6VfV9W/JkzPp854/Nxd97ZWskNv1nKd55ax8+unR7t0kQkTvTmPP5dhGfckj5kZswZn89XLxrPgpW7eeq9QXsBs4gMMr1p8W8DXjOzZwkPtQyAu98RsariyFcuHM8bm/bz7SfXMnNMDqNy0qJdkojEuN60+HcCLwHJQEaXm/SBpMQEfn7t6bjD3z+wjE376qJdkojEuKMe3B0IYungbnde3VjOLY+9R0NLO1+9aDxfmnsSQxJ783dZROTIuju429MFXD8P7p/pOkbPMYzVI8fgwonDeOmWC7j41OH89KVNXH/vEk3tKCIR0VMf/0PB/U/6oxCB/PQUfnndDGaP/YB/f2odC9eUccW0omiXJSIxptvgd/flwb3G6uln188ew8NLdvKzlzZx6ZQRJKnLR0T60FETxcwmmNl8M1tvZts6b/1RXLxKSDC+cfFEtlc08PiK0miXIyIxpjdNyd8Sngy9DbiQ8JSLv49kUQIfPWUY00Znc+efN2tsHxHpU70J/lR3f5nwGUA73P27wGWRLUvMjH+5eCJ7apt5ZImGRRKRvtObC7hazCwB2GxmXyE8R256ZMsSgDnj8zh7XB53vLiJg+0dfO6sElKTNaa/iJyY3rT4vwakATcDM4HPEp4rVyLMzLj96qlML87mv557n/N//Cq/X7yDwXDthYgMXD22+IPhmK91938G6oEv9EtVckhxXhoP3Tibpdur+OmLG/n2k2s50HyQL88dH+3SRGSQ6ukCriR3bwfO7cd6pBtnjs3l0S+exRXTivjRCxtZsFJn+4jI8empxb8UmAGsDK7U/SPQ0Pmiuz8R4drkMAkJxo8/PZX9dS3cOn81wzKGMmd8frTLEpFBpjcHd4cClcBFgAMW3Cv4oyAlKZFff24m1/x6EV/43bucUpjJ+IJ0pozM5LrZxZrQXUSOqqfgH2ZmtwBr+Uvgd9LRxSjKSh3Cgzeeya9f38qmfXW8uXk/j68o5Y1N+7nrszMZOkThLyLd6yn4EwmftmlHeE3BH2XDM4fynU+ceuj5I0t28q8L1vC/H1rO3Z9T+ItI93oK/jJ3/36/VSIn5LrZxSQmwLeeWMMXH1zGnZ85ndxQcrTLEpEBqKfz+I/U0pcB7NozivnR1VN5e0sF593+Cj964X2qGlqjXZaIDDDdTsRiZrnuXtXP9RxRPEzE0pc276vjF69sYeHqPaQNSeSK6UVcPWMUM8fkYKa/5yLxoruJWDQDVwzbvK+OX7++jefXltHY2s6YvDQ+MbWIy6YWMmlEhv4IiMQ4BX8ca2hp4/m1e1mwspRFWyvpcBiXHzr0S2B0riZ4F4lFCn4BoKK+hT+t28uzq8tYtK0Sdzh7XB4fOWUYpxdnc2pRls4IEokRCn75kNLqRhas2M0TK3ezvSJ8UXZigjE2P8S4/BDjCtIZPyydSSMyGD8sXX8QRAaZfg9+MxtNeNKW4YTP+7/H3e80s1zgMaAE+AC4xt2re1qXgj/yyg80s6q0llW7ati0r45tFQ3sqGzgYHv4v4/EBGNGcTY3njuOj00eTmKCjg+IDHTRCP5CoNDdV5hZBrAcuAr4PFDl7j80s28BOe7+zZ7WpeCPjrb2Dj6obGTj3jo2lB3gqVW72VXVxNj8EF+5cDyfnDFSB4hFBrCod/WY2VPAL4PbXHcvC/44vObuE3v6rIJ/YGjvcF5Yu5e739jK6tJa/vbM0XzviikkJ2kyeJGBqLvg75f/Y82sBDgdWAIMd/ey4KW9hLuCjvSZm8xsmZkt279/f3+UKUeRmGBcNrWQBV+ew5fnnsSjS3dx/X2LqahviXZpInIMIt7iN7N04HXgB+7+hJnVuHt2l9er3T2np3WoxT8wPb1qD7fOX4U7FOemMSonlZNHZPCZM4oZmx+Kdnkica+7Fn9vhmU+kS8dAjwOPNxl/P59ZlbYpaunPJI1SORcMa2I8QXpPLGilF3VjZRWN/HWlgrueWMbF04cxufPKeHc8fkk6ECwyIASseC38FG/+4EN7n5Hl5eeJjxn7w+D+6ciVYNE3uSiTCYXTT70vLyumYcX7+ThJTu44TdLGZmdytUzRnLl6SMpyQvpbCCRASCSZ/WcC7wJrAE6gsX/Srif/w9AMbCD8OmcPY4JpK6ewaelrZ0X1u5l/vJS3tpSgTskJRjDM4cyKieViyYN47KphYzK0VXDIpES9bN6ToSCf3Arq23ilffL2V3dRFltM5v21bFuzwEAZhRn89mzxnD51CKdHSTSxxT8MqDsqGxg4eoyFqzczZbyekZkDuXzc0q44ewxpCVH9NCTSNxQ8MuA5O68tmk/9725jbe3VDI2P8TPr53OtNHZ0S5NZNCL6nn8It0xMy6cOIyH//4sHvnibFoOtvPJu97hFy9vpq294+grEJFjpuCXAeOck/J5/p/O5/Kphdzx0iauv28J5Qeao12WSMxR8MuAkpU6hDs/czp3XDON1aW1XPqLt1i0tTLaZYnEFAW/DEifnDGKJ/9xDpmpSVx/32Junb+Kxdsq6egY+MekRAY6HdyVAa2+pY3/fm4DC1buprG1nZHZqXzpgnFcP3uMrggWOQqd1SODWmNrGy+u28cjS3eydHsV503I58efmsaIrKHRLk1kwFLwS0xwdx5espMfPLuB5KQE/vHCkzj/5AImDtfk8SKHU/BLTNle0cCt81fx7gfhydvy01O44OQCLj1tBOdOyCclSdNEiij4JSbtqWni7S0VvLWlglffL+dAcxsZKUmcf3IBpxdnc3pxDqcWZWq+YIlLCn6Jea1tHbyztYLn1pTx9pZKdtc0ARBKTuSSKYV8csZIzhqXpxFCJW5EZTx+kf6UnJTA3InDmDtxGAD7DjSzcmc1r7xfzvNr9vL4ilIyUpIYlplCfnoKw4KRQkfnpDE2P8TMMTkaKE7iglr8EheaD7bz5w37WLKtior6FirqW9h3oIU9NU20BdcGZKQkceGkYVwyZQSXnDpCp4vKoKeuHpEjaGvvYO+BZjaU1fHn9fv484Z9VDa08tFThnHHtdPJHDok2iWKHDcFv0gvtHc4Dy36gP98dgOjc9O4+3MzOXl4RrTLEjku6uMX6YXEBOPzc8YyuSiLLz+8git++RbTRmUzriDEuPx0Lj51OGPyNJG8DG5q8Yt0Y9+BZu58eTMb99axvaKBqoZWAM6bkM91ZxZz5thcckPJunBMBix19YicoD01TfxxWSmPvbuTPbXh4aIzUpIYk5/Gp2eO5nNnafwgGVgU/CJ9pL3DWbytko1769hR2cCq0lre21XDjOJsbr96KhN0TEAGCAW/SIS4OwtW7ub7C9fT2NLOvHPG8MXzxjEsUwPISXQp+EUirKK+hf96bgNPrtxNUkICV88cxZfnnsTo3LRolyZxSsEv0k92VDZw9xvbmL+sFDO4+SMT+OJ543RVsPQ7Bb9IP9tT08T3n1nPC+v2Mn5YOl88byxF2akMzxxKcW6aBo6TiFPwi0TJyxv28R9PrTs0aBxAxtAkrp01mnnnlKgrSCJGwS8SRW3tHeyuaWLfgRbKapt4af0+nl+7F3fnjJJcxuSlUZSdyqQRGXzklOEMSVS3kJw4XbkrEkVJiQmMyQsduur3yukjKatt4qFFO3hnayWvb9pPeV0L7lCYNZTPn1PCZ84oJitNYwVJ31OLX2SAaG3r4M3N+7n/re28s7USgPSUJHJDyeSlJ1OYNZTCrFTG5KVx+dQickPJUa5YBjp19YgMImt31/LaxnIq6lupamilor6FvQeaKatppulgOylJCXxyxihuPLeE8cN0wZgcmbp6RAaRKSOzmDIy60PL3Z3N5fX89u3tPL6ilEeX7uT8kwv4wpwSLphQoCEjpFfU4hcZpCrrW3h4yU5+v3gH5XUtlOSlMXtsHhNHZDBxRAZTR2WRofkE4lq/d/WY2W+Ay4Fyd58SLMsFHgNKgA+Aa9y9+mjrUvCLdK+1rYPn15Yxf3kp6/YcODSKaIKFfznMHptLSX6I/PQUCjJSOHl4Bukp+rEfD6IR/OcD9cCDXYL/R0CVu//QzL4F5Lj7N4+2LgW/SO9V1LewoewA726vYvH2Kt7bVUNrW8eh1xMTjKmjsjh7XPjXQX56eA7iMXm6qCzWROXgrpmVAAu7BP9GYK67l5lZIfCau0882noU/CLH72B7B5X1reyva6G8rpmVO2tYtK2SVbtqDs03DJA5NImrTh/JNbNGH/H4ggw+AyX4a9w9O3hsQHXn8yN89ibgJoDi4uKZO3bsiFidIvGosbWNPTVN7K9rpbyumVffL+e5tXtpbetgcmEmV88cxZXTi8hPT4l2qXKcBlzwB8+r3T3naOtRi1+kf9Q2HuTJ93Yzf3kpa3bXkphgnFGSw+TCLCYVZjBxeAYnDUvXMYJBYqCczrnPzAq7dPWU9/P3i0gPstKGMO+cEuadU8KmfXU8vqKUxVsreWTpDpoP/uU4wYjMoUwqzGD22DzOOSmPKSOzSNSppINGfwf/08A84IfB/VP9/P0i0ksnD8/gto+fAoRnHdtZ1cjGvXVs3V/P1vJ61uyu5faN7wOQOiSRkTmpFGYNZVROKicVpDN+WPhWmJWqPwoDTMSC38weBeYC+WZWCnyHcOD/wcxuBHYA10Tq+0Wk7yQmGGPzQ4zND/3V8vK6ZhZvq2LVrhr21DSxp6aJP+05QFXDrkPvGZJoFGWnMiYvxMenjOCKaUWE1FUUVbqAS0T6XGV9C1vK69m6v4GdVY3sqm5kQ9kBtu1vID0liSunF3HNrNFMHZVF+DwPiYSB0scvInEgLz2FvPQUZo/LO7TM3Vmxs5qHl+xk/vJSHl6yk3EFIa6eMYpPTC2iOE/zEvQXtfhFpN/VNh3k+TVlPLFyN0u3VwFw2sgsLptayPkTCpg4IkPHBfqARucUkQGptLqR59aU8eyavazaVQNAKDmR6cXZzB6bx5zx+UwblUWSJqc5Zgp+ERnwdtc08e72KpbvqGb5jmo27D2Ae3heggnD08kLpZAXSmZyUSaXTy0kTxeX9UjBLyKDTnVDK4u2VfL2lgp2VDZSUd9CRX14foKkBOPCScO4fGohZ43LY3jm0GiXO+Do4K6IDDo5oWQuPa2QS08r/KvlG8oO8MSKUp58bw8vrd8HQEleGrNKcpk2Optpo7KYNCKT5CR1Dx2JWvwiMmi1dzjr9xxgyfZKlmyvYsWOaiqDYamTEoyS/BDjC9KZMDydSSMymVSYQUleKG4OHKurR0Rinruzu6aJVbtqWbenli3l9Wwpr+eDygY6ByJNS07kzLG5nDs+n7PG5TE6J43M1KSYvJ5AwS8icav5YDtbyuvZUHaA1aW1vLO1gq37Gw69PnRIAoVZqRTnpjEmL43i3DSGZQ4lL5RMbiiZ9JQkUpMTSUtOJC158PSQq49fROLW0CGJh+Yx/vSs0QCU1TaxYkcNZbVN7K1tpqy2mZ1VjazYUU1dS1u36yrISGFKUSanFmVRlJ1KdtoQslKHEEpJInVIIqlDEklLSSSUnMTQIQkD8peEgl9E4lJhViqXTU390HJ3p6bxIBX1LVQ2tFLV0EpDSxvNB9upbwn/cli3p5Y3NlfQ3tFzj0ligpGfnszJwzOYNCKDMXkh8tOTyQ2lMConlaLsD39/f1Dwi4h0YWbkhJLJCSUzoYf3tbS1U9XQSm3TQWoaD9LY2kZTa0f4/mA79S1tNLS0UVbTzKbyOh5ctIOWLlNgAozOTeWccfnMKslhbH6I4tw0CjJSIv4rQcEvInIcUpISKcxKpTCrd6329g5nf10LlQ0tVDW0sqW8nkVbK3l+bRmPLfvr0UxDKUmEkpNIS07k3htmUXLYqKgnSsEvItIPEhOMEVlDGZEVvtDsvAkFfGHOWNo7nO0VDeyqbmRXVSNltc00trTR0NpOY2sbacmJfV6Lgl9EJIoSE+zQpDX9RZe1iYjEGQW/iEicUfCLiMQZBb+ISJxR8IuIxBkFv4hInFHwi4jEGQW/iEicGRTDMpvZfmDHcX48H6jow3IGi3jc7njcZojP7dY2984Ydy84fOGgCP4TYWbLjjQedayLx+2Ox22G+NxubfOJUVePiEicUfCLiMSZeAj+e6JdQJTE43bH4zZDfG63tvkExHwfv4iI/LV4aPGLiEgXCn4RkTgT08FvZpeY2UYz22Jm34p2PZFgZqPN7FUzW29m68zsa8HyXDN7ycw2B/c50a61r5lZopmtNLOFwfOxZrYk2N+PmVlytGvsa2aWbWbzzex9M9tgZmfH+r42s68H/22vNbNHzWxoLO5rM/uNmZWb2douy464by3sF8H2rzazGcfyXTEb/GaWCPwK+DgwGfhbM5sc3aoiog34hrtPBs4C/jHYzm8BL7v7BODl4Hms+Rqwocvz24Gfuft4oBq4MSpVRdadwAvuPgmYRnj7Y3Zfm9lI4GZglrtPARKBzxCb+/p3wCWHLetu334cmBDcbgLuOpYvitngB84Etrj7NndvBf4fcGWUa+pz7l7m7iuCx3WEg2Ak4W19IHjbA8BVUSkwQsxsFHAZcF/w3ICLgPnBW2Jxm7OA84H7Ady91d1riPF9TXiK2FQzSwLSgDJicF+7+xtA1WGLu9u3VwIPethiINvMCnv7XbEc/COBXV2elwbLYpaZlQCnA0uA4e5eFry0Fxgerboi5OfArUBH8DwPqHH3tuB5LO7vscB+4LdBF9d9ZhYihve1u+8GfgLsJBz4tcByYn9fd+pu355QvsVy8McVM0sHHgf+yd0PdH3Nw+fsxsx5u2Z2OVDu7sujXUs/SwJmAHe5++lAA4d168Tgvs4h3LodCxQBIT7cHRIX+nLfxnLw7wZGd3k+KlgWc8xsCOHQf9jdnwgW7+v86Rfcl0ervgiYA1xhZh8Q7sK7iHDfd3bQHQCxub9LgVJ3XxI8n0/4D0Es7+uPAtvdfb+7HwSeILz/Y31fd+pu355QvsVy8L8LTAiO/icTPiD0dJRr6nNB3/b9wAZ3v6PLS08D84LH84Cn+ru2SHH329x9lLuXEN6vr7j79cCrwKeCt8XUNgO4+15gl5lNDBZ9BFhPDO9rwl08Z5lZWvDfeuc2x/S+7qK7ffs0cENwds9ZQG2XLqGjc/eYvQGXApuArcC/RbueCG3juYR//q0G3gtulxLu834Z2Az8GciNdq0R2v65wMLg8ThgKbAF+COQEu36IrC904Flwf5+EsiJ9X0NfA94H1gLPASkxOK+Bh4lfBzjIOFfdzd2t28BI3zW4lZgDeGznnr9XRqyQUQkzsRyV4+IiByBgl9EJM4o+EVE4oyCX0Qkzij4RUTijIJfBgQzazez98xslZmtMLNz+nj9/3rY83f6aL1zu4wOOrcv6zazEjO7rsvzWWb2i75av8QvBb8MFE3uPt3dpwG3Af/dx+v/q+B39z79wxKYCxzTertcfXokJcCh4Hf3Ze5+83FVJtKFgl8GokzCQ+12jjv+42As9jVmdu1Rlhea2RvBr4e1Znaemf2Q8OiO75nZw8H76oP7uWb2Wpcx7h8OrhDFzC4Nli0Pxj5f2F3BwQB5XwK+HnzPeWZWYGaPm9m7wW1O8N7vmtlDZvY28FDQsn8z+KXT9dfOD4HzgvV9/bBfF7lm9mQwFvtiM5vaZd2/CbZpm5ndHCwPmdmzwS+qtZ3/XhKfemptiPSnVDN7DxgKFBIefwfgk4SvVp0G5APvmtkbhFvWR1p+HfAnd/+BhedkSHP3N83sK+4+vZvvPh04FdgDvA3MMbNlwN3A+e6+3cwe7al4d//AzH4N1Lv7TwDM7BHCY8a/ZWbFwJ+AU4KPTAbOdfcmM0sDPubuzWY2gfAVnLMID8D2z+5+ebC+uV2+8nvASne/yswuAh4M/j0AJgEXAhnARjO7i/DAZnvc/bJgXVk9bY/ENgW/DBRNncFsZmcDD5rZFMJDUjzq7u2EB6x6HTijh+XvAr+x8MB1T7r7e7347qXuXhp893uEu1jqgW3uvj14z6OEJ7w4Fh8FJgc/IAAyLTyKKsDT7t4UPB4C/NLMpgPtwMm9WPe5wNUA7v6KmeWZWWbw2rPu3gK0mFk54aF81wA/NbPbCQ9x8eYxbovEEHX1yIDj7osIt+ILjuOzbxCerGQ38Dszu6EXH2vp8ridvmsQJQBnBccuprv7SHevD15r6PK+rwP7CP96mQWc6DSCH9oed99EeCTPNcB/mtl/nOB3yCCm4JcBx8wmEZ5irxJ4E7jWwvPrFhAO9aXdLTezMcA+d7+X8OxcnXORHgx+BfTWRmBc0HcP0Js+8TrC3SudXgS+2mW7pnfzuSygzN07gM8R3vYjra+rN4Hrg/XOBSr8sHkYujKzIqDR3X8P/Ji//LtIHFJXjwwUnX38EB55cJ67t5vZAuBsYBXhUUhvdfe9PSyfB/yLmR0k3F3T2eK/B1htZis8PIRzj4K+9y8DL5hZA+EupKN5BphvZlcSDvybgV+Z2WrC/6+9QfgA8OH+B3g8+HXyAn/5NbAaaDezVYTnY13Z5TPfJdyltRpo5C9D93bnNODHZtZBePTHf+jF9kiM0uicIt0ws3R3rw/O8vkVsNndfxbtukROlLp6RLr3xeBXyDrC3TF3R7cckb6hFr+ISJxRi19EJM4o+EVE4oyCX0Qkzij4RUTijIJfRCTO/H+hwYfSTyqonwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# training for 100 iterations\n",
    "n_iterations = 100\n",
    "losses = []\n",
    "for _ in range(n_iterations):\n",
    "    rmse_loss = rmse_incremental_step(gbrl_model, X, y)\n",
    "    losses.append(rmse_loss)\n",
    "    \n",
    "plot_loss(n_iterations, losses, y_label='Training loss', x_label='Boosting Iterations')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GBT work with per-sample gradients but PyTorch losses are typically calculated as the expected loss and derivated w.r.t fixed parameters. GBRL internally multiplies the gradients with the number of samples when calling the step function. This allows to get similar results when using PyTorch losses on 1-dimensional outputs. However, when working with PyTorch losses and multi-output targets or different reductions one should take this into consideration.  \n",
    "For example:\n",
    "1. When using a summation reduction we should use:  \n",
    "    `loss = 0.5*mse_loss(y_pred, y, reduction='sum') / len(y_pred)`\n",
    "2. When working with multiple outputs dimensions we should use:  \n",
    "    `0.5*mse_loss(y_pred, y_multi) * output_dim`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving, loading, and copying a GBRL Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current GBRL Library Version 1.0.9\n",
      "Reading GBRL Library Version 1.0.9\n",
      "######## Loaded GBRL model ########\n",
      "input_dim: 10output_dim: 1 max_depth: 4 min_data_in_leaf: 0\n",
      "generator_type: Quantile n_bins: 256 cv_beta: 0.9 split_score_func: Cosine\n",
      "grow_policy: Oblivous verbose: 0 device: cpu use_cv: 0 batch_size: 5000\n",
      "Loaded: 1016 leaves from 100 trees\n",
      "Model has: 1 optimizers \n"
     ]
    }
   ],
   "source": [
    "# Call the save_model method of a GBRL class\n",
    "# GBRL will automatically save the file with the .gbrl_model ending\n",
    "# The file will be saved in the current working directory\n",
    "# Provide the absolute path to save the file in a different directory.\n",
    "gbrl_model.save_model('gbt_model_tutorial')\n",
    "# Loading a saved model is similar and is done by calling the specific class instance.\n",
    "loaded_gbt_model = GBRL.load_model('gbt_model_tutorial', device=device)\n",
    "# Copying a model is even more straighforward\n",
    "copied_model = gbrl_model.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Manually Calculated Gradients\n",
    "Alternatively GBRL can use manually calculated gradients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def manual_rmse_incremental_step(model, X, y):\n",
    "    # forward pass - y_pred is a torch tensor with requires_grad = True by default\n",
    "    y_pred = model(X)\n",
    "    # calculate loss - we must scale pytorch's mse loss function by 0.5 to get the correct MSE gradient\n",
    "    loss = 0.5*mse_loss(y_pred, y) \n",
    "    grad = y_pred - y\n",
    "    # perform a boosting step\n",
    "    model.step(X, grad)\n",
    "    print(loss.sqrt().item())\n",
    "    # return RMSE\n",
    "    return loss.sqrt().item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple iterations at once (standard supervised learning)\n",
    "GBRL supports training multiple boosting iterations with targets similar to other GBT libraries. This is done using the `fit` method.  \n",
    "***Note: only the RMSE loss function is supported for the `fit` method***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting GBRL device to cuda\n",
      "Setting GBRL feature weights \n",
      "Setting GBRL bias \n",
      "Boosting iteration: 1 - MultiRMSE Loss: 38.5797\n",
      "Boosting iteration: 2 - MultiRMSE Loss: 36.2629\n",
      "Boosting iteration: 3 - MultiRMSE Loss: 34.4074\n",
      "Boosting iteration: 4 - MultiRMSE Loss: 33.2252\n",
      "Boosting iteration: 5 - MultiRMSE Loss: 32.2341\n",
      "Boosting iteration: 6 - MultiRMSE Loss: 31.9472\n",
      "Boosting iteration: 7 - MultiRMSE Loss: 31.2075\n",
      "Boosting iteration: 8 - MultiRMSE Loss: 30.1109\n",
      "Boosting iteration: 9 - MultiRMSE Loss: 29.8426\n",
      "Boosting iteration: 10 - MultiRMSE Loss: 29.0514\n",
      "Boosting iteration: 11 - MultiRMSE Loss: 28.7877\n",
      "Boosting iteration: 12 - MultiRMSE Loss: 28.3994\n",
      "Boosting iteration: 13 - MultiRMSE Loss: 28.1013\n",
      "Boosting iteration: 14 - MultiRMSE Loss: 27.2042\n",
      "Boosting iteration: 15 - MultiRMSE Loss: 26.8625\n",
      "Boosting iteration: 16 - MultiRMSE Loss: 25.9826\n",
      "Boosting iteration: 17 - MultiRMSE Loss: 25.5647\n",
      "Boosting iteration: 18 - MultiRMSE Loss: 24.7986\n",
      "Boosting iteration: 19 - MultiRMSE Loss: 24.1443\n",
      "Boosting iteration: 20 - MultiRMSE Loss: 23.3593\n",
      "Boosting iteration: 21 - MultiRMSE Loss: 22.7118\n",
      "Boosting iteration: 22 - MultiRMSE Loss: 21.8528\n",
      "Boosting iteration: 23 - MultiRMSE Loss: 21.3475\n",
      "Boosting iteration: 24 - MultiRMSE Loss: 20.7077\n",
      "Boosting iteration: 25 - MultiRMSE Loss: 20.4818\n",
      "Boosting iteration: 26 - MultiRMSE Loss: 19.8492\n",
      "Boosting iteration: 27 - MultiRMSE Loss: 19.2729\n",
      "Boosting iteration: 28 - MultiRMSE Loss: 18.7241\n",
      "Boosting iteration: 29 - MultiRMSE Loss: 18.4136\n",
      "Boosting iteration: 30 - MultiRMSE Loss: 18.2943\n",
      "Boosting iteration: 31 - MultiRMSE Loss: 18.1081\n",
      "Boosting iteration: 32 - MultiRMSE Loss: 17.4549\n",
      "Boosting iteration: 33 - MultiRMSE Loss: 17.0828\n",
      "Boosting iteration: 34 - MultiRMSE Loss: 16.4608\n",
      "Boosting iteration: 35 - MultiRMSE Loss: 16.2257\n",
      "Boosting iteration: 36 - MultiRMSE Loss: 16.0034\n",
      "Boosting iteration: 37 - MultiRMSE Loss: 15.7727\n",
      "Boosting iteration: 38 - MultiRMSE Loss: 15.5215\n",
      "Boosting iteration: 39 - MultiRMSE Loss: 15.231\n",
      "Boosting iteration: 40 - MultiRMSE Loss: 14.7094\n",
      "Boosting iteration: 41 - MultiRMSE Loss: 14.6251\n",
      "Boosting iteration: 42 - MultiRMSE Loss: 14.2636\n",
      "Boosting iteration: 43 - MultiRMSE Loss: 14.0607\n",
      "Boosting iteration: 44 - MultiRMSE Loss: 13.9144\n",
      "Boosting iteration: 45 - MultiRMSE Loss: 13.6068\n",
      "Boosting iteration: 46 - MultiRMSE Loss: 13.2338\n",
      "Boosting iteration: 47 - MultiRMSE Loss: 12.825\n",
      "Boosting iteration: 48 - MultiRMSE Loss: 12.6524\n",
      "Boosting iteration: 49 - MultiRMSE Loss: 12.1593\n",
      "Boosting iteration: 50 - MultiRMSE Loss: 12.05\n",
      "Boosting iteration: 51 - MultiRMSE Loss: 11.772\n",
      "Boosting iteration: 52 - MultiRMSE Loss: 11.5681\n",
      "Boosting iteration: 53 - MultiRMSE Loss: 11.5173\n",
      "Boosting iteration: 54 - MultiRMSE Loss: 11.3958\n",
      "Boosting iteration: 55 - MultiRMSE Loss: 11.1758\n",
      "Boosting iteration: 56 - MultiRMSE Loss: 11.0855\n",
      "Boosting iteration: 57 - MultiRMSE Loss: 10.8746\n",
      "Boosting iteration: 58 - MultiRMSE Loss: 10.577\n",
      "Boosting iteration: 59 - MultiRMSE Loss: 10.5139\n",
      "Boosting iteration: 60 - MultiRMSE Loss: 10.3431\n",
      "Boosting iteration: 61 - MultiRMSE Loss: 10.0624\n",
      "Boosting iteration: 62 - MultiRMSE Loss: 9.8757\n",
      "Boosting iteration: 63 - MultiRMSE Loss: 9.62955\n",
      "Boosting iteration: 64 - MultiRMSE Loss: 9.5414\n",
      "Boosting iteration: 65 - MultiRMSE Loss: 9.35272\n",
      "Boosting iteration: 66 - MultiRMSE Loss: 9.16555\n",
      "Boosting iteration: 67 - MultiRMSE Loss: 9.02244\n",
      "Boosting iteration: 68 - MultiRMSE Loss: 8.82241\n",
      "Boosting iteration: 69 - MultiRMSE Loss: 8.66658\n",
      "Boosting iteration: 70 - MultiRMSE Loss: 8.55436\n",
      "Boosting iteration: 71 - MultiRMSE Loss: 8.35623\n",
      "Boosting iteration: 72 - MultiRMSE Loss: 8.26789\n",
      "Boosting iteration: 73 - MultiRMSE Loss: 8.21895\n",
      "Boosting iteration: 74 - MultiRMSE Loss: 8.18072\n",
      "Boosting iteration: 75 - MultiRMSE Loss: 8.0978\n",
      "Boosting iteration: 76 - MultiRMSE Loss: 7.9846\n",
      "Boosting iteration: 77 - MultiRMSE Loss: 7.7927\n",
      "Boosting iteration: 78 - MultiRMSE Loss: 7.37001\n",
      "Boosting iteration: 79 - MultiRMSE Loss: 7.34261\n",
      "Boosting iteration: 80 - MultiRMSE Loss: 7.2136\n",
      "Boosting iteration: 81 - MultiRMSE Loss: 7.06581\n",
      "Boosting iteration: 82 - MultiRMSE Loss: 6.98016\n",
      "Boosting iteration: 83 - MultiRMSE Loss: 6.86069\n",
      "Boosting iteration: 84 - MultiRMSE Loss: 6.75899\n",
      "Boosting iteration: 85 - MultiRMSE Loss: 6.54197\n",
      "Boosting iteration: 86 - MultiRMSE Loss: 6.44827\n",
      "Boosting iteration: 87 - MultiRMSE Loss: 6.31682\n",
      "Boosting iteration: 88 - MultiRMSE Loss: 6.06001\n",
      "Boosting iteration: 89 - MultiRMSE Loss: 5.99263\n",
      "Boosting iteration: 90 - MultiRMSE Loss: 5.86513\n",
      "Boosting iteration: 91 - MultiRMSE Loss: 5.8357\n",
      "Boosting iteration: 92 - MultiRMSE Loss: 5.66526\n",
      "Boosting iteration: 93 - MultiRMSE Loss: 5.49422\n",
      "Boosting iteration: 94 - MultiRMSE Loss: 5.34011\n",
      "Boosting iteration: 95 - MultiRMSE Loss: 5.28356\n",
      "Boosting iteration: 96 - MultiRMSE Loss: 5.219\n",
      "Boosting iteration: 97 - MultiRMSE Loss: 5.06981\n",
      "Boosting iteration: 98 - MultiRMSE Loss: 4.9546\n",
      "Boosting iteration: 99 - MultiRMSE Loss: 4.82375\n",
      "Boosting iteration: 100 - MultiRMSE Loss: 4.78847\n"
     ]
    }
   ],
   "source": [
    "gbrl_model = GBRL(input_dim=input_dim,\n",
    "                  output_dim=1,\n",
    "                  tree_struct=tree_struct,\n",
    "                  optimizer=optimizer,\n",
    "                  gbrl_params=gbrl_params,\n",
    "                  verbose=1,\n",
    "                  device=device)\n",
    "final_loss = gbrl_model.fit(X, y, iterations=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RL using GBRL\n",
    "Now that we have seen how GBRL is trained using incremental learning and PyTorch we can use it within an RL training loop\n",
    "\n",
    "For simplicity, we train an agent using REINFORCE, which might not give the best results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_returns(rewards, gamma):\n",
    "    returns = []\n",
    "    running_g = 0.0\n",
    "    for reward in rewards[::-1]:\n",
    "        running_g = reward + gamma * running_g\n",
    "        returns.insert(0, running_g)\n",
    "    return returns\n",
    "\n",
    "def perform_rollout(rollout_buffer, agent, env):\n",
    "    obs, _ = env.reset(seed=0)\n",
    "    rollout_buffer['rewards'] = []\n",
    "    done = False\n",
    "    while not done:\n",
    "        action_logits, _ = agent(obs, requires_grad=False)\n",
    "        action = Categorical(logits=action_logits).sample()\n",
    "        action_numpy = action.cpu().numpy().squeeze()\n",
    "        \n",
    "        obs, reward, terminated, truncated, _ = env.step(action_numpy)\n",
    "        rollout_buffer['rewards'].append(reward)\n",
    "        rollout_buffer['obs'].append(obs)\n",
    "        rollout_buffer['actions'].append(action)\n",
    "\n",
    "        done = terminated or truncated\n",
    "\n",
    "    \n",
    "def reinforce_step(agent, observations, actions, returns, entropy_coeff=1e-1):\n",
    "     # model update\n",
    "    action_logits, values = agent(observations)\n",
    "    dist = Categorical(logits=action_logits)\n",
    "    log_probs = dist.log_prob(actions)\n",
    "    \n",
    "    # calculate reinforce loss with subtracted baseline\n",
    "    returns = (returns - returns.mean())\n",
    "    value_loss = th.nn.functional.mse_loss(returns, values)\n",
    "    policy_loss = -(log_probs*returns).mean()\n",
    "    loss = policy_loss + value_loss\n",
    "    loss.backward()\n",
    "    agent.step()\n",
    "\n",
    "# Helper function to plot results\n",
    "def plot_returns(iterations, losses, x_label, y_label):\n",
    "    plt.close('all')\n",
    "    _, ax = plt.subplots()\n",
    "\n",
    "    ax.plot(iterations, losses)\n",
    "    ax.set_xlabel(x_label)\n",
    "    ax.set_ylabel(y_label)\n",
    "    plt.show()\n",
    "        \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****************************************\n",
      "Shared GBRL Tree with input dim: 4, output dim: 3, tree_struct: {'max_depth': 4, 'n_bins': 256, 'min_data_in_leaf': 0, 'grow_policy': 'oblivious'} policy_optimizer: {'algo': 'SGD', 'start_idx': 0, 'stop_idx': 2, 'scheduler': 'Const', 'init_lr': 0.1} value_optimizer: {'algo': 'SGD', 'start_idx': 2, 'stop_idx': 3, 'scheduler': 'Const', 'init_lr': 0.0075}\n",
      "****************************************\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAALEwAACxMBAJqcGAAAM9lJREFUeJzt3Xd8leX5+PHPlQ0hgwx22BuRISAoCIooWouj7j2qdVStWlfHV/21X6u1X1ttbZW6cKM4UXGhggORsPcKBAIJGZC9k+v3x3kSA2QckjPD9X698sp5xjnPdfSQ69z3/dzXLaqKMcYYAxDi7wCMMcYEDksKxhhj6llSMMYYU8+SgjHGmHqWFIwxxtQL83cAbZGUlKR9+/b1dxjGGBNUli9fnquqyY0dC+qk0LdvX1JTU/0dhjHGBBURSW/qmHUfGWOMqWdJwRhjTD1LCsYYY+pZUjDGGFPPkoIxxph6lhSMMcbUs6RgjDGmniUFY4wBVJX5q/eyr7Dc36H4lSUFY4wBtmYXc+vrKzn3qe/Yll3s73D8xpKCMcYAGzMLASgoq+KCp79n5a4Dfo7IPywpGGMMsDGziPBQ4f1fTyYmKpxL/7uURVty/B2Wz1lSMMYYYFNWIQO7xDCwSyfm3TSJvknRXPfiMt5ftcffofmUJQVjjMHVfTSsWwwAXWKimPuriRzXpzO/mbuK77bl+jk637GkYIw56u0vqWRfYQVDu8fU74uNCueFa8bTPyma38xdRU5RhR8j9B1LCsaYo96mLNcg87DusQft7xgRxlOXjaWwrIo75q6itlb9EZ5PWVIwxhz1NmUWATC0W+xhx4Z2i+WhWSP4dlsu//56m69D8zlLCsaYo96mrEKSOkWQHBPZ6PGLxqcwa1QPHv98C0vT8o7otUsrq3l7eQaXPfsD//P+Ok+E61VBvfKaMcZ4wsbMokZbCXVEhIfPG8majHxue2MlH982hcROjScQcM2O/nHHfuYtz+DjtZmUVNbQITyUJdvz+PXJA+kSG+WNt+ER1lIwxhzVqmtq2bKviKHdYpo9r1NkGP+6dCwHSqq48vkfWZ7e+OS2rfuKuOzZpVw0+wc+XpvJz47tzpu/msT8WydTq/DB6r3eeBse47WkICLPi0i2iKxrsO8xEdkkImtE5F0RiW9w7H4R2SYim0XkdG/FZYwxDe3MK6WiuvawQebGHNMzjn9eOoacogp+8Z/vueXVFezKKwWguKKahz/eyBlPfMP6vYU8NGsEy/5wKn89fxQT+iUwsEsnRvWK450VgT3vwZvdRy8C/wJearDvc+B+Va0WkUeB+4F7RWQ4cDEwAugBfCEig1W1xovxGWNM/Z1HDW9Hbc7pI7oxeWASsxenMXtxGp9v2Md5Y3vy1eZs9hVWcNG4FO6ZOaTR7qVzxvTkofkb2JxVxJAWWib+4rWWgqouBvYfsu8zVa12Nn8AejmPzwbeUNUKVd0BbAMmeCs2Y4ypszGzkNAQYWCXTm4/JzoyjDtmDOar305j1ugezE3dTXJMJO/cfAKPnn9sk+MNPx/Vg9AQ4Z2VGZ4K3+P8OdB8LTDXedwTV5Kok+HsO4yI3ADcANC7d29vxmeMOQpsyixiQHI0kWGhR/zcbnFR/O2CUTzw8+F0jAgjNESaPT+pUyRTByfz/sq93Hv6UEJaON8f/DLQLCK/B6qBV4/0uao6W1XHqeq45ORkzwdnjDmqbMpq/s4jd8REhbeYEOqcO6YnWYXl/HCEt7b6is+TgohcDZwFXKaqddMD9wApDU7r5ewzxhivKSirYk9+mVuDzJ4yY3hXYiLDeGdlYP6J82lSEJGZwD3ALFUtbXDoA+BiEYkUkX7AIOBHX8ZmjDn6bMo8skFmT4gKD+WMkd34ZF0WZZWBdy+NN29JfR1YAgwRkQwRuQ7X3UgxwOciskpEngZQ1fXAm8AG4BPgFrvzyBjjbZuyXOUthrWx++hInTOmJ8UV1Xy+cZ9Pr+sOrw00q+oljex+rpnz/xf4X2/FY4wxh9qUVUh8x3C6xjY9O9kbJvZLpEdcFO+uyGDWqB4+vXZLbEazMeao5SpvEYOIb+8CCgkRzh7Tk8VbcwOuJLclBWPMUam2VtmcVeTTQeaGzhvTk5pa5eO1mX65flMsKRhjfO6z9Vn88T3/VgxN319KWVWNz8cT6gzqGkPP+A78uHN/yyf7kCUFY4zPvb96Ly//kM6+wnK/xeCPO48ONTolnlW78v12/cZYUjDG+FxdETl/TuDamFVEiMCgLv5NCnvyywJqXMGSgjHGp1SVnXklgH+TwqbMQvomRdMh4sjLW3jKmN7xAKzane+3GA5lScEY41MHSqsoKnfVxVyy3T9JobqmlmU79zOqV7xfrl/nmJ5xhIUIK3c1vjaDP1hSMMb4VLrTSpjUP5GdeaVkFpT5PIalO/ZzoLSK00d09fm1G4oKD2Vo9xhrKRhjjl7pznjCxRNc5c780VpYsC6TDuGhTB3cxefXPtTolHjWZBRQU6stn+wDlhSMMT6VnleKCJw2vBvxHcN9nhRqa5VP1+9j2pBkv44n1Bmd0pniimq25xT7OxTAkoIxxsfS80roHhtFh4hQju+XwBIfDzYv33WAnKIKZh7TzafXbUr9YHOA3JpqScEY41M780rokxgNwMT+iWQcKGP3/tIWnuU5C9ZmEREawilD/d91BNAvMZrYqDBW7g6MwWZLCsYYn9q1v5Q+iR0BmDQgEfDdramqyqfrs5gyKImYqHCfXLMlISHCqJR4VlpLwRhztCkqryK3uLK+pTC4SwwJ0RFudSFlFpTx5w83cMtrKyival1l/TUZBezJLwuYrqM6Y1Li2bKviJKK6pZP9jJ/rtFsjDnK1N151NdpKYSECBP7J/DD9jxUtdFqpWk5xTyzKI13VmZQq1BTq8RGhfGX84494usvWJdFWIgwY7h/b0U91Oje8dQqrN1TwMT+iX6NxVoKxhif2eWMHfR2kgK4xhX2FpTXH6tTUFbFLa+tYPrji3hv1R4umdCbr387jZunDeD1H3czb3nGEV1bVflkXSaTBiQS3zGi7W/Gg0andAYIiC4kaykYY3ymrrxFXfcRuCaxgWu+Qt3+iuoabngplRW7DnDT1AFcc2I/kmNcC+HcOWMwK3fl84f31jKiR6zbpa83ZRWxM6+U60/q78m35BEJ0RH0SezIqgAYbLaWgjHGZ3bllZLUKYJOkT99Hx3YpRNJnSLrB5tra5W73lzN0h37+dsFo7hn5tD6hAAQFhrCk5eMITYqnJtfXUFheZVb116wLqt+fkQgGp0SHxAzmy0pGGN8puHtqHVEXOMKS9Jc4wp/WbCRD9dkcv8ZQzl7dM9GXyc5JpJ/XTqWXftLuXfeGlRbng38ybpMxvdNOCjBBJLRKfHsK6xwq+zHm6m72ZvvnfIglhSMMT6TnvfT7agNTRqQyL7CCh78YD3//WYHV5/Qlxta6OaZ0C+Be2cOYcG6LF7+Ib3Zc7fnFLNlXzFnBNhdRw2NTokHWp7E9tWmbO59ew3PLNrulTgsKRhjfKK8qobMgnL6JEQfdqzujps5S9KZOaIbfzxruFvrJl8/pT/H9enMy0uaTwqfrMsC4PQRgZsUhveIJSI0hJXNdCGl55Vw+xsrGdYtlvvOGOaVOCwpGGN8om7Wct+kw1sK/ZOi6Z3QkfF9O/OPi0cTGtJyQgBX19OZI7uzNbu4fuGexny2YR+jesXRI75D64L3gciwUIb3iG2ypVBWWcOvXl6OiPDMFcd5rW6TJQVjjE/sdP5oHzqmAK4/7vNvncxr108kKvzI/tidOsxVruKLjfsaPZ5VUM7q3fmcFsCthDqjU+JZsyefyurag/arKve/s4bN+4p44uLRpCQcnlg9xZKCMcYn6tZR6NPEH7S4DuGEhx75n6Q+idEM7NKJhZsaTwqfb6jrOgqsCWuNOWFAIuVVtUx97Cse/2xzfetqzvc7eW/VXu48dTDThni3ZpPNUzDG+ER6XimxUWHEd/R8zaFTh3Xl2W/SKCyvIvaQmkafbdhH/6RoBiR38vh1PW3G8K48fflxvP7jLv751Tb++dU2JvZLZNnO/Zw6rCu3nDzQ6zFYS8EY4xPp+0vpkxjt1gDykTp1WBeqa5VFm3MO2l9QVsWS7XnMGNHVK9f1NBFh5jHdmHPtBL699xRunz6I9LwS+iZF8/hFowhxc6ylLaylYIzxifS8Ekb2jPPKa4/p3ZmE6AgWbtzHz0f1qN//9eZsqms1YCesNadnfAd+c+pgbp8+iFrF7cH3trKWgjHG66pqask4UEbfRgaZPSE0RJg2JJmvNudQXfPTIO1n6/eRHBPJGGcOQDASEZ8lBLCkYIzxgb35ZdTU6kGF8Dzt1GFdKSirYnm6q35QeVUNX2/OZsbwrj7pdmkvvJYUROR5EckWkXUN9iWIyOcistX53dnZLyLypIhsE5E1IjLWW3EZY3xvZ33JbO+0FACmDEoiPFRYuCkbgO+351JSWcNpAVYmO9B5s6XwIjDzkH33AQtVdRCw0NkGOAMY5PzcAPzHi3EZY3ys7nbUvl5sKcREhTOxf2L9fIXP1u+jU2RY/epuxj1eSwqquhjYf8jus4E5zuM5wDkN9r+kLj8A8SLS3VuxGWN8Kz2vlA7hoV4vRnfqsK6k5ZSwLbuYLzbuY9qQZCLDvDPzt73y9ZhCV1XNdB5nAXXtup7A7gbnZTj7DiMiN4hIqoik5uTkNHaKMSbApOeV0Cexo9dvC53uzG5+7NNN5BZXBnSto0Dlt4FmddW6bbne7eHPm62q41R1XHJyshciM8Z4WnpeKb29WJqhTq/OHRnaLYZP1+8jPNR1R5I5Mr5OCvvquoWc39nO/j1ASoPzejn7jDFBrrZWSd9fSt8k7w0yN1TXWjhhQBIxUZ6fPd3e+TopfABc5Ty+Cni/wf4rnbuQJgIFDbqZjDFBLKuwnMrq2kbXUfCGuolqMwN47YRA5rUZzSLyOjANSBKRDOAB4BHgTRG5DkgHLnRO/xg4E9gGlALXeCsuY4xv1a/L3Mg6Ct4wKiWed28+gWN7xfvkeu2N15KCql7SxKHpjZyrwC3eisUY4z9pOa6k0C/ZN0kBXGUvTOvYjGZjjFel5ZQQFR5C99gof4di3NBiUhCRaBEJcR4PFpFZImKjN8YYt6TlFtMvqZOVmggS7rQUFgNRItIT+Ay4AtdsZWOMaVFaTgn9fdh1ZNrGnaQgqloKnAf8W1UvAEZ4NyxjTHtQUV1DxoFSBvjodlTTdm4lBRGZBFwGfOTss3njxpgWpeeVUqvQPwhWPTMu7iSF24H7gXdVdb2I9Ae+8m5Yxpj2IC2nGMC6j4JIi7ekOoXtFjfYTgNu82ZQxpj2IS3XuR3Vuo+CRotJQUQGA78F+jY8X1VP8V5Yxpj2IC2nhC4xkVZuIoi4M3ntLeBp4FmgxrvhGGPak7ScYus6CjLuJIVqVbVFb4wxRywtt4QzR9rSKMHEnYHm+SJys4h0d5bTTBCRBK9HZowJavtLKskvraK/jScEFXdaCnVVTe9usE+B/p4PxxjTXtTdeTTAbkcNKs0mBae8xX2qOtdH8Rhj2om6Qng2phBcmu0+UtVaDm4hGGOMW7bnFhMRGkKvzr5ZR8F4hjtjCl+IyG9FJMXGFIwx7krLca3LHGqF8IKKO2MKFzm/G653YGMKxphmpeUUM7CLjScEG3dmNPfzRSDGmPajuqaWXftLOW2ELYkZbNyZ0XxlY/tV9SXPh2OMaQ92HyijqkbtdtQg5E730fgGj6NwLae5ArCkYIxplBXCC17udB/d2nBbROKBN7wVkDEmeJRWVlNQVkX3uA4H7a+/HTXJxhSCTWvWaC4BbJzBmKOcqnLNC8s4/e+LyS+tPOhYWm4xnTuG0zk6wk/RmdZyZ0xhPq67jcCVRIbjKpJnjDmKfbExm6U79gPw9KI07jtjaP2x7TkltrBOkHJnTOFvDR5XA+mqmuGleIwxAUJVEWl8jkF1TS2PLNhI/6RohveI5cXvd3DtiX3pEhsFuLqPTh6S7MtwjYe40310pqoucn6+U9UMEXnU65EZY/xmb34Zkx/9iv83fwOqetjxN1Mz2J5Twr1nDOXu04dQXaP866ttABSWV5FbXGEthSDlTlKY0ci+MzwdiDEmMJRX1XDjK8vJLCjj+e928O+vtx90vKSimsc/38K4Pp05bXhX+iRGc9H4FF7/cRe795dazaMg12RSEJGbRGQtMERE1jT42QGs8V2IxhhfUVX+5/11rMko4N+XHcc5o3vw2KebeWfFTz3G//0mjdziCn73s2H13Uu3njKIEBH+/sWWBtVRLSkEo+bGFF4DFgB/Ae5rsL9IVfd7NSpjjF+89uMu3kzN4NcnD2TmMd04ZWgXsosquGfeGpJjIhnSLYbZi9M4c2Q3xvbuXP+8bnFRXH1CX2Z/k0ZJRTWhIULvBEsKwajJloKqFqjqTlW9BEgBTlHVdCBEROyWVGPamRW7DvDgB+uZOjiZO2YMBiAiLISnrziOgV06cePLy7ln3hoqq2u55/Shhz3/xqkDiI4I49P1+0jp3IGIsNbc8W78rcX/ayLyAHAvcL+zKwJ4pS0XFZE7RGS9iKwTkddFJEpE+onIUhHZJiJzRcRucDbGR7KLyrnpleV0j+vAExePPqiyaWxUOC9eM4G4DuF8vTmHyyf2oW8j5Ss6R0dw/RRXnUwbZA5e7qTyc4FZuCatoap7gZjWXlBEegK3AeNU9RggFLgYeBT4u6oOBA4A17X2GsaYI/O/H22koKyKpy8/jviOh38f6xYXxZxrJ3Dx+BRunz6oyde5bko/usdFMSYl3ovRGm9yZ55CpaqqiCiAiHiiozAM6CAiVUBHIBM4BbjUOT4HeBD4jweuZYxpRnlVDZ9v2McvxvZieI/YJs8b1DWGR35xbLOv1SkyjK/vnkZEqHUdBSt3/s+9KSLPAPEicj2wEHi2tRdU1T24JsTtwpUMCoDlQL6qVjunZQA9G3u+iNwgIqkikpqTk9PaMIwxjkVbciitrGHmMZ4pcx0ZFtrkpDcT+FpMCqr6N2Ae8DYwBPijqj7Z2guKSGfgbFz1k3oA0cBMd5+vqrNVdZyqjktOthmTxrTVp+uyiOsQzsT+if4OxQSAZruPRCQU6KyqnwOfO4O/V4vIRlUd1sprngrsUNUc5xrvACfiaomEOa2FXsCeVr6+McZNldW1fL5xH6eP6Ea4dfkYmp+8djGwH1gjIotE5DQgDdds5svacM1dwEQR6SiuNuZ0YAPwFXC+c85VwPttuIYxxg1L0vIoKq9mpq2QZhzNtRT+ABynqttEZCywBDhfVee35YKqulRE5uFaqKcaWAnMBj4C3hCRPzv7nmvLdYwxLs0VtvtkXSbREaFMHpTk46hMoGouKVSq6jYAVV0hIlvbmhDqqOoDwAOH7E4DJnji9Y0xLrnFFVz4zBKuPbEfl0/sc9Cxmlrls/X7OHloF6LCQ/0UoQk0zSWFLiJyZ4Pt+Ibbqvq498IyxnjCows2kZZTwp8/2sDUwcmkJHSsP7Zs537ySio545jufozQBJrmRpb+i2uSWt3PodvGmAC2PH0/by3P4LyxPQkV4ffvrTuoDPYn67KIDAthmq17YBposqWgqg/5MhBjjOdU19Tyx/fW0z0uij+dfQzH9ozjwfkb+GD1Xs4e3ZPaWuXT9VmcNDiZ6Eh35rCao4Xdg2ZMO/Tq0l1syCzkj2cNJzoyjCsm9WVUSjz/b/4G8ksrWbOngMyCcs7w0IQ1035YUjCmncktruBvn21m8sCk+j/6oSHCX84dSX5ZFQ9/vJEF6zIJCxGmD+3q52hNoLF2ozFBqqZW2ZpdRN/E6IPuHnpkwSbKq2p4cNaIg25FHd4jluun9OfpRduJjQrjhIFJxHUM90foJoC1mBRE5GHgr6qa72x3Bu5S1T94OTZjTDPmLtvN795dS3ioMKx7LKNT4ukR34F5yzO4adoABnY5vHz17dMH8dHavezeX2ZdR6ZR7nQfnVGXEABU9QBwptciMsa45YuN++gRF8Uvp/QnOiKMecszeGTBJnrERXHrKQMbfU6HiFAeO38Ux/aKs1nMplHudB+FikikqlYAiEgHINK7YRljmlNRXcOS7XlcMK4X9850rYJW153UuWMEHSOa/qc9sX8iH/x6sq9CNUHGnaTwKrBQRF5wtq/Btd6BMcZPUnceoKyqhqmDf5pjEBoiDO3W9HoIxrijxaSgqo+KyBpchesA/qSqn3o3LGNMcxZvySE8VKzctfE4t+4+UtUFwAIvx2KMcdOiLTmM75tgE8+MxzVXOvtb53eRiBQ2+CkSkULfhWiMaWhfYTmbsoo4abCVpzCe11yZi8nOb6tzZEwAWbTFtQztVEsKxguaTAoiktDcE1V1v+fDMca0ZPGWHLrERDK0m31fM57XXIfkckABAXoDB5zH8bhWT+vn7eCMMQerqVW+2ZrLqcO6NrlwjjFt0eSYgqr2U9X+wBfAz1U1SVUTgbOAz3wVoDHmJ2sy8ikoq2Kqlbs2XuLOjOaJqvpx3YZzJ9IJ3gvJGNOURVtyEIEpA235TOMd7tzPtldE/gC84mxfBuz1XkjGmKYs3pLDsb3i6Rwd4e9QTDvlTlK4BNd6yu8624udfca0G6pKlnOr5+asIsqrapgyKJnRKfGEhgRG331BaRWrdufz65Mbr2tkjCe4M6N5P3C7iMS4NrXY+2EZ4xv7Csv57VurWb07n8Ly6vr9IvCPL7aSEB3BtMHJnDKsC1MHJxMT1XKp6eqaWjILytm1v5Rd+0sZ1Sue4T3aXn7i22251Co2nmC8yp3S2SOBl4AEZzsXuEpV13k5NmO8SlW5e94aUnce4NyxPRnaLYYhXWMY0i0GQVi0NYevNmXz5eZs3lm5h8iwEGYM78q5Y3py0uBkwkNdQ3J5xRV8uy2XRZtzWLHrABkHyqiu/Wkt5K6xkSy8axqd2jj7eNGWbGKiwhjVK75Nr2NMc9z5lD4D3KmqXwGIyDRgNjbYbILcq0t3sXhLDn86ewRXTOp72PFZo3owa1QPamqVFbsOMH/1Xuav3suHazJdLYghyWzLLmbtngJUoXPHcI7vl8gZI7vTJ6EjvRM7UlZZw3VzUvnnwq3cf+awVseqqizeksuUQUmEhdqCicZ73EkK0XUJAUBVvxaRaC/GZIzXpeeV8PDHG5kyKInLJ/Zp9tzQEGF83wTG903gj2cNZ/GWHN5duYfPN+xjSNcY7jh1MFMHJ3NMz7hGxx8uHNeL577dwQXjejGwS+smnH29OYeswnJbPtN4nTtJIU1E/gi87GxfDqR5LyRjvKumVrnrzdWEhgh/Pf/YI5oEFh4awvRhXZk+zP0/zvfOHMon67L4n/fX8+ovjz/iSWe1tcpjn26md0JHZo3ucUTPNeZIudMOvRZIBt5xfpKcfcYEpWe/SSM1/QAPzRpB97gOXr9eYqdI7p45lO+35/Hhmswjfv7H6zLZkFnInTMG149jGOMt7tx9dAC4DUBEQnF1J1mVVBOUNmcV8X+fbeH0Ea4BY1+5dEJv5i7bxZ8/2sDJQ7u4PehcXVPL459tYUjXGH4+yloJxvta/NohIq+JSKwzjrAW2CAid3s/NGM877531hATFcbD5470ae2g0BDhT2cfw77CCp5cuNXt572zYg9puSXcddrggJkvYdo3d9qiw52WwTm4FtrpB1zhzaCM8YY9+WWs3JXPjVMHkNjJ98uMj+ndmYvHp/D8tzvYnFXU4vkV1TX844stjEqJZ8ZwG2A2vuFOUggXkXBcSeEDVa3CVT211UQkXkTmicgmEdkoIpNEJEFEPheRrc7vzm25hjGHWuysQzDNj5O/7pk5lLgO4fxm7ioqqmuaPfe1pbvYW1DOPacPsYqoxmfcSQrPADuBaGCxiPQB2jqm8ATwiaoOBUYBG4H7gIWqOghY6Gwb4zGLNufQIy6KgV06+S2GhOgIHv3FsWzMLOT/PtvS5HklFdU89dU2ThiQyIlW/M74UItJQVWfVNWeqnqmuqQDJ7f2giISB5wEPOe8fqWq5gNnA3Oc0+bgapkY4xFVNbV8ty2XkwYn+/1b96nDu3L5xN7MXpzGt1tzGz3nmUXbyS2u5LenD/FxdOZo19zKa5er6isicmcTpzzeymv2A3KAF0RkFK7FfG4Huqpq3f16WUCjnagicgNwA0Dv3r1bGYI52qzanU9RRXXALGH5+zOHs2R7Hne9tYpPbj+pvuppWWUND81fzxvLdvOzY7sztrf1ohrfaq6lUDdrOaaJn9YKA8YC/1HVMUAJh3QVqarSxLiFqs5W1XGqOi45OTD+gZvAt2hzDqEhwgkB0hXTISKUJy4ew/6SSu57Zw2qyuasImb961vmpu7mlpMH8MRFo/0dpjkKNdlSUNVnnN8PefiaGUCGqi51tufhSgr7RKS7qmaKSHcg28PXNUexxVtzGJMST1yHlquc+soxPeO4+/QhPPzxJu6Yu4oF67KIiQrn5WuPZ/KgwEhe5ujjzjyF/iIyX0RyRCRbRN4Xkf6tvaCqZgG7RaSus3Q6sAH4ALjK2XcV8H5rr2FMQ3nFFazdUxAwXUcN/XJyf04YkMh7q/YyoV8CC26fYgnB+JU70ypfA54CznW2LwZeB45vw3VvBV4VkQhcdZSuwZWg3hSR64B04MI2vL4x9b7dlosG6DoEISHCvy8by9Id+5kxrCshNkHN+Jk7SaGjqr7cYPuVts5oVtVVwLhGDk1vy+sa05hFm3NIiI7gmB5x/g6lUfEdIzh9RDd/h2EM4F5SWCAi9wFv4Br8vQj4WEQSoH5lNmMCUm2tsnhrDlMGJdm3cGPc4E5SqOvG+dUh+y/GlSRaPb5gjLdtyCwkt7gyIMcTjAlE7lRJ7eeLQIzxhkVOaYspgywpGOOOJu8+EpF7Gjy+4JBjD3szKGM8ZdGWHEb0iCU5xvcF8IwJRs3dknpxg8f3H3JsphdiMcajisqrWJF+wLqOjDkCzSUFaeJxY9vGBJzvt+dRXaucZEnBGLc1lxS0iceNbRsTcBZvySE6ItTqBxlzBJobaB4lIoW4WgUdnMc421Fej8yYNvp+ex4T+ycSEWbrGhvjruZqH4X6MhBjPGlvfhk7cku47HirpGvMkbCvUKZd+n57HoAtUGPMEbKkYNql77fnkhAdwZCubanybszRx5KCaXdUle+35TFpQKKVtjDmCFlSMO1OWm4JWYXlnDjAuo6MOVKWFEy7UzeecMKARD9HYkzwsaRg2p3vt+XSM74DfRI7+jsUY4KOJQXTrtTWKkvSXOMJIjaeYMyRsqRg2pUNmYXkl1Zx4kDrOjKmNSwpmHZlSf14gg0yG9MalhRMu/Ld9lwGJEfTNdYqsRjTGpYUjM/89q3VXPrfH1D1Tj3Fyupaftyx31oJxrSBJQXjEz/u2M+85Rl8vz2PrzfneOUaazLyKa2ssfEEY9rAkoLxutpa5U8fbqBbbBQ94zvwxMKtXmktfLctDxGY2N+SgjGtZUnBeN17q/awdk8B98wcws0nD2DV7nwWb831+HW+357LiB6xxHeM8PhrG3O0sKRgPGLV7nwKy6sO219aWc1fP9nMsb3iOGd0T84/rhc94qJ44ostHm0tlFXWsHJXvpW2MKaNLCmYNvtoTSbnPPUdP3vyG9btKTjo2OzFaWQVlvPHs4YTEiJEhoVy07QBrNiVz3fb8jxy/Zpa5ZEFG6msqbVS2ca0kSUF0yabs4q4e95qjukZS3WNct5/vuf1H3ehqmQVlPPMojR+NrI74/sm1D/nwvEpdIuN4omFbW8tFFdUc/1LqcxZks41J/ZlyiBLCsa0RXPLcRrTrIKyKm58ZTnRkWE8d9V4wkKE38xdxf3vrCV15wEqa2qpqVXuO2PoQc+LDAvlxqn9eXD+Bpak5bX6FtI9+WVc9+IytmYX8+dzjuHyiX088baMOapZS8G0Sm2tcufcVezeX8q/LxtL19goEjtF8uI1E7h9+iDeWZnB/NV7uXZyP1ISDi9Md/GE3nSJieTJhVtbdf3Vu/M5+1/fsedAGS9cPd4SgjEe4rekICKhIrJSRD50tvuJyFIR2SYic0XEbiEJYE9+uZWFm7L541nDD+oaCg0R7pgxmBevmcB5Y3tyy8kDGn1+VHgoN04dwA9p+1m4cd8RXTu7qJzLn11Kh4gQ3rn5BE4anNym92KM+Yk/Wwq3AxsbbD8K/F1VBwIHgOv8EpVp0UdrMvnHF1s5b2xPrpzU+Df0qYOTefzC0cREhTf5Opce35t+SdH88qVUHnh/HcUV1W5d/2+fbqa8uoY510xgkC23aYxH+SUpiEgv4GfAs862AKcA85xT5gDn+CM207R1ewq48vkfueW1FYzsGcfD545sU3nqqPBQ5t86masm9eWlH9KZ8fiiFlsNazLyeWt5Btec2I/+yZ1afW1jTOP81VL4B3APUOtsJwL5qlr3VTED6NnYE0XkBhFJFZHUnBzvlEswB0vPK+G211dy1j+/ZfXufH535lDeunESUeGhbX7tTpFhPDhrBG/fdAIxUWFcNyeVW19f2WirQVV5aP4GEqMj+PUpA9t8bWPM4Xx+95GInAVkq+pyEZl2pM9X1dnAbIBx48Z5p7KaAVwTz55YuJXnv91BaIhw87QB/GrqAOI6NN0l1Fpje3fmw1un8J+vt/Pkl1vJOFDKi1dPIK7jT9f6YPVelqcf4K+/OJbYZrqljDGt549bUk8EZonImUAUEAs8AcSLSJjTWugF7PFDbMbx+YZ9PPjBevbkl3HBcb347elDvF6OOiIshNtPHcSQbjHc9vpKLvnvD7x83QQSO0VSWlnNIws2MbJnHOcf18urcRhzNPN595Gq3q+qvVS1L3Ax8KWqXgZ8BZzvnHYV8L6vYzOQcaCU619K5fqXUomODOWtGyfx2AWjfLo+wcxjuvHfq8aRllvMhc8sIaugnKcXpZFZUM4DP3fNjDbGeEcgTV67F3hDRP4MrASe83M87VptrfLctzv4bnsuB0oqySup5EBJJSWVNXQID+X+M4Zy7eR+hIf6Z9hp6uBk5lwzgevmpHL+09+TU1TBrFE9GNfg9ldjjOeJtxY88YVx48Zpamqqv8MIOgVlVdw5dxULN2UzpGsMXeOiSOgYTkJ0JEkxEcwa1YNenQ+fcOYPq3fnc+XzP1JRXcOXd02jR3wHf4dkTNATkeWqOq6xY4HUUjA+sDmriBtfWc7u/aU8NGsEV07q06bbSr1tVEo8H946mYKyKksIxviAJYWjyIdr9nLPvDVER4bx+g0TD5qJHMhSEjqS4u8gjDlKWFJo56pralm0JYfXlu5i4aZsjuvTub5WkTHGHMqSQju1N7+Muct282bqbjILykmOieT26YO45eSBRIRZHURjTOMsKbQzqsqrS3fx0Pz1VNcqJw1K5oGfj2D6sC5+u5PIGBM8LCm0IxXVNTz4wXpe/3E304Yk86ezj2m0bLUxxjTFkkI7kV1Yzk2vrmB5+gFunjaAu04bQqhN8jLGHCFLCu3A6t353PByKoVl1fzr0jGcdWwPf4dkjAlSlhSC3MbMQi57dilxHcJ5+6YTGN4j1t8hGWOCmCWFIJZZUMY1Lyyrr1Fkk7uMMW1lSSFIFZVXcc0LyyiuqObNX1lCMMZ4hiWFIFRVU8vNr65ga3YxL1w93rqMjDEeYzeuBxlV5ffvruWbrbn85dyRtmi9McajLCkEmb9/voU3UzO4bfogLhxvFYGMMZ5lSSGIPL1oO09+uY2LxqVwx6mD/B2OMaYdsqQQJOZ8v5NHFmxi1qgePHzeyIAud22MCV6WFILAm6m7eeCD9cwY3pX/u3CUzVQ2xniNJYUAN3/1Xu57ew1TBiXxr0vHWFE7Y4xX2V8YDysoreKK55Zy91uryS4qb9NrLU8/wB1zVzGubwKzrxhHZFioh6I0xpjG2TwFDyoqr+LKF35kw94CABasy+K26QO5+oR+R7yGQUlFNXe+uYpucVE8e9U4OkRYQjDGeJ+1FDykpKKaa15Yxvo9Bfz7suP47I6pHN8vgYc/3sTMfyzmq03ZqKrbr/eXBRvZtb+Uv10witiocC9GbowxP7Gk4AFllTVcN2cZK3fn8+QlY5gxvCv9kqJ57urxvHDNeACueXEZZz/1HR+s3ktVTW2zr7doSw6v/LCLX07ux8T+ib54C8YYA1j3UZuVV9Vww8upLN2xn39cNJozR3Y/6PjJQ7pw4oAk3lq+m+e+2cFtr6+kR1wU15zYj4smpBzWCigoreKeeasZ1KUTd502xJdvxRhjkCPp0gg048aN09TU1FY/P6+4gtUZ+azaXcDq3flUVtdyfP8EThyYxKhe8S2OA5RUVHPjK8v5Zmsufz3/WC4c1/wM49pa5ctN2fz3mzSW7thPx4hQzjq2OxdP6M2YlHhEhNvfWMlHazJ59+YTGdkrrtXvzRhjmiIiy1V1XKPHjsak8OWmfTzwwXp27y8DIERgcNcYwkKF9XsLUYUO4aGM75fApRN6c/qIrodNFsstruDaF5exfm8hfzlvZIsJ4VBrMwp45Yd05q/ZS2llDUO6xjC+X2de+WEXd84YzG3TbcayMcY7LCkcYm1GAf/+ehujU+IZnRLPMT3jiI509aQVlFbxw448lmzP46vN2aTnlTJlUBIPzhrBgOROAOzKK+XK55eSVVjOU5eOZfqwrq1+D8UV1cxfvZc3lu1m9e58RqXE8/aNkwiz+QjGGC+xpNBK1TW1vPxDOo9/toXy6hqundyPU4d15aZXVlBVU8vzV4/nuD6dPXa97TnFJHWKJK6D3W1kjPEeSwptlFNUwaOfbGLe8gwAesRF8dJ1ExjYJcbr1zbGGE9rLinY3UduSI6J5G8XjOKSCb15d2UGt5w8kO5xttKZMab98XlSEJEU4CWgK6DAbFV9QkQSgLlAX2AncKGqHvB1fM05rk9nj3YXGWNMoPHHaGY1cJeqDgcmAreIyHDgPmChqg4CFjrbxhhjfMjnSUFVM1V1hfO4CNgI9ATOBuY4p80BzvF1bMYYc7Tz632PItIXGAMsBbqqaqZzKAtX91Jjz7lBRFJFJDUnJ8c3gRpjzFHCb0lBRDoBbwO/UdXChsfUdUtUo7dFqepsVR2nquOSk23RemOM8SS/JAURCceVEF5V1Xec3ftEpLtzvDuQ7Y/YjDHmaObzpCCuehHPARtV9fEGhz4ArnIeXwW87+vYjDHmaOePeQonAlcAa0VklbPvd8AjwJsich2QDlzoh9iMMeao5vOkoKrfAk2tPD/dl7EYY4w5WFCXuRCRHFytCnckAbleDMebgjX2YI0bLHZ/sdh9o4+qNnqnTlAnhSMhIqlN1foIdMEae7DGDRa7v1js/mf1mY0xxtSzpGCMMabe0ZQUZvs7gDYI1tiDNW6w2P3FYvezo2ZMwRhjTMuOppaCMcaYFlhSMMYYU6/dJwURmSkim0Vkm4gE9BoNIvK8iGSLyLoG+xJE5HMR2er8DshVfkQkRUS+EpENIrJeRG539gd8/CISJSI/ishqJ/aHnP39RGSp89mZKyIR/o61MSISKiIrReRDZztY4t4pImtFZJWIpDr7Av7zAiAi8SIyT0Q2ichGEZkULLG3pF0nBREJBZ4CzgCGA5c4C/oEqheBmYfsC5bFh4J58aQK4BRVHQWMBmaKyETgUeDvqjoQOABc578Qm3U7rnVJ6gRL3AAnq+roBvf3B8PnBeAJ4BNVHQqMwvXfP1hib56qttsfYBLwaYPt+4H7/R1XCzH3BdY12N4MdHcedwc2+ztGN9/H+8CMYIsf6AisAI7HNTs1rLHPUqD8AL1w/QE6BfgQVwmZgI/biW0nkHTIvoD/vABxwA6cG3WCKXZ3ftp1SwHXim67G2xnOPuCiVuLDwWS1iye5G9OF8wqXCXbPwe2A/mqWu2cEqifnX8A9wC1znYiwRE3uNZM+UxElovIDc6+YPi89ANygBecbrtnRSSa4Ii9Re09KbQr6voKEtD3ELd28SR/U9UaVR2N65v3BGCofyNqmYicBWSr6nJ/x9JKk1V1LK7u3VtE5KSGBwP48xIGjAX+o6pjgBIO6SoK4Nhb1N6Twh4gpcF2L2dfMAmaxYfaw+JJqpoPfIWr2yVeROoqCQfiZ+dEYJaI7ATewNWF9ASBHzcAqrrH+Z0NvIsrGQfD5yUDyFDVpc72PFxJIhhib1F7TwrLgEHO3RgRwMW4FvMJJkGx+FAwL54kIskiEu887oBrLGQjruRwvnNawMWuqverai9V7Yvrs/2lql5GgMcNICLRIhJT9xg4DVhHEHxeVDUL2C0iQ5xd04ENBEHsbvH3oIa3f4AzgS24+oh/7+94Woj1dSATqML1beQ6XH3EC4GtwBdAgr/jbCL2ybiay2uAVc7PmcEQP3AssNKJfR3wP87+/sCPwDbgLSDS37E28x6mAR8GS9xOjKudn/V1/zaD4fPixDkaSHU+M+8BnYMl9pZ+rMyFMcaYeu29+8gYY8wRsKRgjDGmniUFY4wx9SwpGGOMqWdJwRhjTD1LCiagiUiNU0VztYisEJETPPz6vztk+3sPve60BlVLp3kybhHpKyKXNtgeJyJPeur1zdHNkoIJdGXqqqI5CldBw794+PUPSgqq6tGk45gGHNHrNpiR3Ji+QH1SUNVUVb2tVZEZcwhLCiaYxOIqBY24PCYi65ya/Be1sL+7iCx2Wh3rRGSKiDwCdHD2veqcV+z8niYiXzeomf+qM2sbETnT2bdcRJ6saxE0xikOeCNwh3OdKc4M6rdFZJnzc6Jz7oMi8rKIfAe87LQIvnFaSA1bSY8AU5zXu+OQVkmCiLwnImtE5AcRObbBaz/vvKc0EbnN2R8tIh85LbF1df+9zNGruW8jxgSCDk710ihc5YhPcfafh2tW6SggCVgmIotxfSNvbP+luEpI/6+41tnoqKrfiMiv1VUIrzFjgBHAXuA74ERxLQbzDHCSqu4QkdebC15Vd4rI00Cxqv4NQERew7Xewbci0hv4FBjmPGU4rkJxZSLSEZihquUiMgjXjPdxuIqv/VZVz3Jeb1qDSz4ErFTVc0TkFOAl578HuIr8nQzEAJtF5D+41u/Yq6o/c14rrrn3Y9o/Swom0JXV/dEWkUnASyJyDK6yGq+rag2uQmSLgPHN7F8GPC+uon3vqeoqN679o6pmONdehavbphhIU9UdzjmvAzc0+uymnQoMdxoeALHiqi4L8IGqljmPw4F/ichooAYY7MZrTwZ+AaCqX4pIoojEOsc+UtUKoEJEsnGVdl4L/J+IPIqrTMY3R/heTDtj3UcmaKjqElzf/pNb8dzFwEm4Koa+KCJXuvG0igaPa/Dcl6gQYKIzVjJaVXuqarFzrKTBeXcA+3C1esYBbV1W87D3o6pbcFX4XAv8WUT+p43XMEHOkoIJGiIyFAgF8oBvgIvEtThOMq4/+D82tV9E+gD7VPW/wLO4/hACVDmtB3dtBvo7YwUA7vTBF+HqsqnzGXBrg/c1uonnxQGZqloLXIHrvTf2eg19A1zmvO40IFcPWdeiIRHpAZSq6ivAY/z038Ucpaz7yAS6ujEFcC01eZWq1ojIu7jWPFiNqzrrPaqa1cz+q4C7RaQKVxdQXUthNrBGRFaoq+x0s5y+/puBT0SkBFe3VEvmA/NE5GxcyeA24CkRWYPr3+BiXIPRh/o38LbTqvmEn1oRa4AaEVmNa13vlQ2e8yCubrI1QCk/lXJuykjgMRGpxVWd9yY33o9px6xKqjFHSEQ6qWqxczfSU8BWVf27v+MyxhOs+8iYI3e903pZj6uL5xn/hmOM51hLwRhjTD1rKRhjjKlnScEYY0w9SwrGGGPqWVIwxhhTz5KCMcaYev8fRXFbNPMJaPcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "device = 'cuda'if th.cuda.is_available() else 'cpu'\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "wrapped_env = gym.wrappers.RecordEpisodeStatistics(env, 50)  # Records episode-reward\n",
    "num_episodes = 1000\n",
    "gamma = 0.95\n",
    "optimizer = { 'algo': 'SGD',\n",
    "              'lr': 0.1,\n",
    "              'start_idx': 0,\n",
    "              'stop_idx': env.action_space.n,\n",
    "            }\n",
    "value_optimizer = { 'algo': 'SGD',\n",
    "              'lr': 0.0075,\n",
    "              'start_idx': env.action_space.n,\n",
    "              'stop_idx': env.action_space.n + 1,\n",
    "            }\n",
    "\n",
    "bias = np.zeros(env.action_space.n + 1, dtype=np.single)\n",
    "agent = ActorCritic(input_dim=env.observation_space.shape[0],\n",
    "                    output_dim=env.action_space.n + 1,\n",
    "                    tree_struct=tree_struct,\n",
    "                    policy_optimizer=optimizer,\n",
    "                    value_optimizer=value_optimizer,\n",
    "                    gbrl_params=gbrl_params,\n",
    "                    verbose=0,\n",
    "                    bias=bias, \n",
    "                    device=device)\n",
    "\n",
    "update_every = 15\n",
    "mean_returns = []\n",
    "iterations = []\n",
    "rollout_buffer = {'actions': [], 'obs': [], 'returns': []}\n",
    "\n",
    "for episode in range(1, num_episodes + 1):\n",
    "    perform_rollout(rollout_buffer, agent, wrapped_env)\n",
    "    rollout_buffer['returns'].extend(calculate_returns(rollout_buffer['rewards'], gamma))\n",
    "\n",
    "    if episode % update_every == 0:\n",
    "        returns = th.tensor(rollout_buffer['returns'], device=device)\n",
    "        actions = th.cat(rollout_buffer['actions']).to(device)\n",
    "        # input to model can be either a torch tensor or a numpy ndarray\n",
    "        observations = np.stack(rollout_buffer['obs'])\n",
    "        reinforce_step(agent, observations, actions, returns)\n",
    "        rollout_buffer = {'actions': [], 'obs': [], 'returns': []}\n",
    "\n",
    "        mean_returns.append(np.mean(wrapped_env.return_queue))\n",
    "        iterations.append(agent.get_iteration())\n",
    "\n",
    "plot_returns(iterations, mean_returns, y_label='Episodic Returns', x_label='Boosting Iterations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explainability\n",
    "GBRL implements SHAP value calculation. SHAP values can be calculated over the entire ensemble as well as for individual trees.\n",
    "GBRL returns SHAP values with shap: [n_samples, n_features, n_actions]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs, info = wrapped_env.reset(seed=0)\n",
    "# per tree shap values\n",
    "tree_shap = agent.tree_shap(0, obs)\n",
    "# for the entire ensemble\n",
    "shap_values = agent.shap(obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SHAP values are calculated internally and can be plotted using the [SHAP library](https://github.com/shap/shap).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAEZCAYAAACU8lxmAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAALEwAACxMBAJqcGAAAPdZJREFUeJzt3XlcFdf5P/DPcy/LBS4ooGyyKKJGIMEYjCKgAip7mqbfftsk3yxN08QsTdpf0yZdkqZZ2rRJ2iRN06TZmzZJk9YmEcR9Q9QI7nvEDRVEUFzY4d7z+2OGeMXLIrLo8Hm/XvfF3JkzZ545DPeZc2a4I0opEBERkfGYBjoAIiIi6htM8kRERAbFJE9ERGRQTPJEREQGxSRPRERkUEzyREREBsUkT2RAIvKkiPxjoOPoKRF5XUQeH+g4uiIiBSJyR2+UvZh9FpEVInJ3d+N0WG+GiBy52PUGmoj8QkTeGug4rkRM8kQORCRJRNaIyGkROSkiRSIySV92p4isdrLOQRGZ2W7eDBFRIvJou/kj9fm1+uugiDzWt3vVv/QEVCMi7t0sf0G7KqXmKKWe7oPYYkVkoYhUi8glf0mIUipTKfX+xZbtz32+0jg7EVFK/VYpddEnNcQkT/Q1EfEBkAfgzwD8AIwA8BsATT2o7g4AJwHc3sHyoUopK4CbATwhIhk92MZlR0RGAkgGoADcMLDRONUC4BMA3x/oQIxKRFwGOgY6h0me6JyxAKCU+kgpZVNKNSilFimltl5MJSLiBeB/ADwAYIyIxHdUVim1FsAOALFO6ikQkQfbzdsiIjfp0y+LyGEROSMiG0QkuYN4LugZOY4+iIhJRB4TkX0ickJEPhERP32ZRUT+oc8/JSLFIhLYye7fDmAdgPegneg4bjNMROaKSJVe36siMh7A6wAS9JGNU3rZ90TkGYd1fyAipfroyhciEuKwTInIHBHZq8f4FxERZ8EppfYopd6G1uaXzHHYvK13LiIv6CMZB0Qks33Z7uyziPiKSJ7eVjX6dGg3Y3IXkZdEpFx/vdR+VEUf/q7Wj4NbHeZnichOETkrIkdF5BGHZTkisllv4zUico3DsoMi8qiIbAVQp0//u902XxaRV/Tp74nILn07+0XkXn2+F4ACACFybrQrRNpdfhKRG0Rkhx7LCr1NHWN5RES2ijYi9y8RsXSn7YyISZ7onK8A2ETkfRHJFBHfHtZzE4BaAJ8CWIh2ya6NaBIBxADY5KTIR9B6+m3lowFEAMjXZxUDmABt1OFDAJ/28MPshwBuBDAdQAiAGgB/0ZfdAWAIgDAA/gDmAGjopK7bAfxTf6W3nRCIiBnaKMkhACOhjZJ8rJTapde5VillVUoNbV+hiKQC+B2A/wUQrNfxcbtiOQAmAbhGL5fezX3vbZMB7AEwDMAfALzd/oSjO/sM7bP5XWi/73Bobf5qN2P4JYAp0I6NOADXA/iVw/IgPb4R0H6/fxORcfqytwHcq5TyhnbiuQwARORaAO8AuBfacfAGgC/anTzcDCAbwFBov58sEfHW1zdD+718qJc9Du135gPgewD+JCITlVJ1ADIBlOttY1VKlTvunIiMhfa38SMAwwHMBzBPRNwciv0vgAwAo6AdE3d2q+UMiEmeSKeUOgMgCdpQ85sAqvReo2PPdYree/j6Be1D2NEdAP6llLJB+1D7roi4titTDW04/y0AjymlljoJ6b8AJohIhP7+VgBzlVJNerz/UEqdUEq1KqVeBOAOYJyTeroyB8AvlVJH9LqfBPA/og27tkD7UI/SRzc26O10ARFJgpaUPlFKbQCwD8At+uLroZ1A/FQpVaeUalRKXXB/QwduBfCOUmqjHt/PofWCRzqUeU4pdUopVQZgObQENxAOKaXe1H/370M7Kels5MMp/ff6H6VUvVLqLIBnoZ2EdcetAJ5SSh1XSlVBu+R0W7syjyulmpRSK6GdNP6vPr8FQLSI+CilapRSG/X59wB4Qyn1pX4cvA/tMtYUhzpfUUod1kfADgHYCOCb+rJUAPVKqXX6/uUrpfYpzUoAi6Bd5umO7wDIV0otVkq1AHgBgAeAqe1iKVdKnQQwDwN3PAw4JnkiB0qpXUqpO5VSodB6MiEAXnIosk4pNdTxBaCsbaGIhAFIgdaTBYDPAVig9XAcDVNK+SqlxiulXukglrPQPoC/q8+62aFe6EOSu/QhyVPQetzDerDbEQD+63DSsguADVpy+gDaaMTH+tDvH5ycsLS5A8AipVS1/v5DnBvFCIOWAFt7EF8ItN47AEApVQvgBLSeaJtjDtP1AKw92M559CHttiHj17u52tdxKKXq9cmLjkVEPEXkDRE5JCJnAKwCMFTvEXflvPbSp0Mc3tfoPWZny78FIAvAIRFZKSIJ+vwIAD9pd3Ib1q7ew+3i+BDnRqJuwblePPSRsnWiXX45pW+zu8du++PBrm+7T4+HKxWTPFEHlFK7oV1bvuB6eSdug/Z3NU9EjgHYDy3Jd+vfrJz4CMDN+oetBVovFaJdf/8ZtB6Yr36ycRqAs2vRdQA8297oiWK4w/LDADLbnbxYlFJHlVItSqnfKKWiofWUcuDkZkIR8dBjmS4ix/R9/zGAOBGJ07cRLs5vyurqLvdyaEmmbVte0EYXjnax3iXR7+huGzKe09vVd7H8J9BGZSYrpXwATNPnO73XoJ3z2gvaSJPjkLev3oYXLFdKFSulvgEgAMBn0G5SBLTf37PtjhFPpdRHnezTpwBmiHYvwTehJ3l9iP8/0HrggfqxO99h3y72eBBoJxx9ejxcqZjkiXQicpWI/ET/UGrrld8M7Uay7roD2vDoBIfXt6Bdn/TvQVjzoX2gPQXtEoBdn+8NoBVAFQAXEXkC2vVNZ74CYBGRbL0X/itoQ/ttXgfwbNtlAREZLiLf0KdTRORq/cTgDLThXDsudCO03n+0w36PB1AI7aRgPYAKAM+JiJdoN/Ql6utWAghtd03V0UcAviciE/QE8VsAXyqlDnZQvkP6fRAWAG76e4t081/9ellX++wN7Tr8KdFugvz1RdT9EYBf6b/HYQCeAND+OxN+IyJu+sliDrT7OdxE5FYRGaIPg5/Bud/1mwDmiMhkvQ299OPJu6Mg9EsFK6DdW3BAvxcB0NreHdqx2yrazYmzHVatBOAvIkM6qPoTANkikqYfzz+BdulgTddNM/gwyROdcxbajVNfikgdtOS+HdqHSJdEZAq0hPwXpdQxh9cXAErhcBNdd+nXoOcCmAmH4U5oQ+gLoCXwQwAaceFwaVsdpwHcD+36/1FoPXvHu+1fBvAFgEUichbafk/WlwUB+De0D/xdAFZCG8Jv7w4A7yqlyhz3HdrNYrdC66XlAoiCdnnjCLRrq4B2c9cOAMdEpLp9xUqpJQAeh9b7qwAwGucuYVysCGjJs+3u+gZoN8r1t073GdolIg9o926sg/a77q5nAJQA2ApgG7Rr4884LD8G7ebKcmiXf+boo1aANhJ1UL9EMAfa7w5KqRIAP4D2+6yBdjzf2Y1YPkS7Y1e/DPUQtGRdA20o/wuH5buhnajs1y8NOF4SgFJqD4D/g/avrtXQjqtcpVRzN+IZdESpS/4+CCIiIroMsSdPRERkUEzyREREBsUkT0REZFBM8kRERAbFBwlQX+OdnUR9RETAm6cNqzvfidAl9uSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIrfeEeGU92g0Gwb6CiI+t4XS4oGOgS6zDHJk+E024ARrzPLk/EV3xAx0CHQZY7D9URERAbFJE9ERGRQTPJEREQGxSRPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJEREQGxSRPRERkUEzyREREBsUkT0REZFB8QA31CXmh1Q/A254uwDAP4HfTTLhlvPNzyieLbHj2SwV387l5W+8wI3KonFfu7zvsuKPAjjdnm3D3NTw/JSLqCj8pDSgwMLAxOTn5pUstc4n+AqC58n4z/pltxn2L7dhRrTos/J1xgtqHXb5+tU/wNY0Kv/3Sjhj/PoyYiMhgBn2Snzlz5m1jxoypsFqtdovFooYNG9YcFxe3KTMzM66ndSYmJr7l7+/f3FU5EVFubm7K3d1dWa1W+6hRo6pnzZr17Z5ut01lZaWlsLDwRwCQkZGRJCIqMzMzvqMyvU1eaPUC8C0Aj1vdBEmhghuiBB/stPe4zp8X2vHQtSYM8+i1MImIDG9QJ/mUlJRHV65c+Xer1XooOTl5UmNjo0yaNGmSq6vridra2rt6UmdOTs5FpaHk5OT7m5qaZPr06RFubm4NJSUlH/Vku5eZsQBa1SMuX7XNiBsu2FHd8Qrz9in4vdqKmHdb8dfN558MrK9QKDmmMGeCdLA2ERE5M6iT/LZt254eO3bs3k2bNk0pKCjYAAAFBQVbSkpKZhYWFj4MAMnJyS8HBQU1WCwW5ePjY4uNjd2ZnZ09vK0OPz+/1vj4+KURERE17u7uqrKy8r/r16//fk1Njau7u7tyd3dXqampP+oqlvz8/MPBwcHvnDp1ypyVlRWVnZ3tHxcXt2no0KGtVqvVPmbMmIqMjIwpbeWTk5NfDggIaLJYLMrb29sWHR1d6hhTUlLSawCwcuXKQgBYtmxZsbu7u4qPj1/SvgwApKamPhQaGlrr4eGhhg8f3jR16tR/OCz7kdlsVsnJyS/5+/u3eHh4qHHjxh3Ozs4O7mB3rADOOM4Y4g6cbXY+XP+/40zYdZcZVfeb8eZsM55aa8dHu7REb7Mr3L/EhlfTzDAJkzwR0cUYtEk+PT199okTJ1wDAgLe7Kyc2Wyuvuaaa74za9Ysl4SEhFnHjx8fXVFR8bljmdLS0uljx469b/bs2abAwMCbr7/++rd9fX1bmpqapKmpSZYtW/ZSV/FkZWWNLC8v/76vr2/r/PnzSw8dOlR44sSJMQkJCZOnT58e6Obmdnrz5s3Lc3NzXbOzs/3XrFnzUGxs7M8bGxtl2rRpQcHBwX9yVu/06dOTASA1NXVSU1OTlJSUzGxfJiMjI3n16tUvR0REfJyWlmaJi4u7f8uWLbdMmzbtxbYydrsdJ0+ezJwyZUpoy68ri7/6wY7Q+Smfl8sLrardazWAWgA+AFBWVgYAONMEuNgav95m23wAsNYfRohVYDYJQm2H8fBEE/79lUJZWRle26xwzXBBSOvhr8ufOHnSaT2O00SDTUd/C5y+cqd7w6C9u761tTUSAFxdXXd2Vm7FihVPO7xdlpCQ8HlZWVmWY5moqKiVixcv/lh/W5OYmNjtOIqKil7z9PR8zdXV1T58+PCq+Pj4b+Xm5pr37NkzPikp6bG2EYbs7Ozpe/bsOVZfX3+nxWKZazab0dDQcH1WVlbk/Pnz90O70a1HTpw48avAwMCzRUVFd+uz3p44ceL3jxw5cgeAn7SVi4iI+EZ+fn4lgOsnTJiwvq6uLnTv3r0h7evTr8m7yAutY9Qj4QCALVUK8WHnrmSEh4d3OC0Vdih9eulGG1YeVpi/PwRAK042ApuOD8XhVhtenWnusB6iwaazvylOX5nTvWHQ9uRdXFz2A0BLS0t0Z+VSUlIeDQ8PP912Y97GjRu/3dDQ4O5YxmKxlHa0flcSExPvr6+vl9OnT5tLS0uDFi1a9EVra+tVra2tcHNzK24rl5+fX2m1Wm0tLS3R+fn5J5KSkn5cVVU1Y8WKFaUhISF1ycnJr/Y0hsbGxhFWq/W8K+YWi+Wr2tpa77b3JpMJ8+fP39323sXFpb61tfW8dmijHnGpAzAXwFN1zQpFRxU+L1W4Ldr54fZ5qR01jQpKKayvUHhlkx3fGK0Nzb+XoQ3lb75De8UHAb+easKzyYP20CUi6rZB+0m5cOHCRf7+/i3Hjx+/u6MyOTk5XmvWrHluxIgRC2bMmOHf2NgoEydO/BRA+4vDNsc3ImLDJXBxcdnt4uKC5ubm69rmZWdnB9bW1prbRh6WLVv2UmlpaVBaWpr7mDFj/lxUVPRAenp6ipPqWrvansViOVpXV3feP6c1NjaOsVqtZy9hN+4H4BHwmg0359nw11kmxAzTmq3wiIL15XNhfbxbIeotG7xfseH2Ahsevd6EO2K1Q3OoRRDkde7lZgJ83IAh7rw+T0TUlUE7XA8AV1999eOrV69+buLEiUWBgYEPFhQUbMrMzIyprq5+0WKx7PHx8flta2srXFxcqvLy8mpmz56dW1pa+o2u6nV1dT1QV1fnmp2dPSI/P//oxcY1b948W2xs7O69e/c+kZmZucRkMpUdOnRo+bBhwxo9PT3fy8zMjKmrq7vT29v7lfz8/MOpqamVACAiLe3rcnFx2SMiaGpqSgZQ4mx7/v7+z2zZsmVVUlLSG0OHDn2osbHx5j179iRcd911L19s7G3UIy4nAdwI4IK77ZJDtf+Jb/NRjrl9kQ6t+O6gPmSJiC7KoO3JA8Dy5ct/P3369NvPnj0buWrVqg0Wi0UVFxdvam5uDrBarW/l5+dXTp48+YNt27bNcXd3V3v27Pl7RETE6q7q9fLyejk8PLx8xYoVhz09PVVqaupDFxtbRETEND8/v/1r1qwpWblyZVVjY6PfhAkTZs2bN68FgEtZWdn3ly9ffshisajt27c/N3Xq1DcWLFhwQWx5eXk111133aKSkpIXPD091aRJkxa1L7NgwYLCpKSk/3fgwIFbly5d2rhly5a/XXPNNZ+sWrXqxxcbNxERXT5EqY6/hYyoF/T7AVZeqzDi9Uu6YkJ0RSi+oRLxY0cMdBjUN3rlmuSg7skTEREZGZM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFB8bmdZDhuZuDonO4/vpboSrXhy0MAH1BDneBT6Kiv8QAj6iMiAn6GGxafQkdEREQdY5InIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKD5PnojoClX0xQKg/ORAh2F8bi7AMJ+BjqJHmOSJiK5QI4NHACPuHugwjO/oWwMdQY9xuJ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSb6fJSYmvuXv79880HF0RURUWlrafQMdBxHRFevkWeCbzwFeNwMR9wAfruq47POfAbEPA963AKPmAHLTTy8oIzc9DLnpAOSmOshNuyA3je0qBCb5HoiIiDjl6uqq3N3dlYeHhwoODq6fNm3a8/0ZQ0ZGxhSTyaQiIiJq+nO7RETUzpMfa6/2HnhTe0xt5TvAP38M3Pc3YEeZ8zqUAv7+EFDzAbDgcQB4EHLTd79eLjfdDeD7ALIBWAHkAKjuKjQm+R6aMGHCkqamJklLS7OEhYUtLyoqeiQ9PX12f22/qqrq9x4eHqqsrGxof26XiIi6oa4R+M864OlbAKsHkDQeuGES8MFK5+V/9k1g4mjAxQyMGwEAnwNIBADITSYAvwbwY6i5O6HmKqi5+6DmnuwqDCb5S5SXl9c0fPjwu+x2OxoaGmYBQEJCwofDhw9v9vDwUKGhoWfT0tIe6Gj9nJwc90mTJi0cNmxYs6enpwoLCzszc+bM2zrbZm5urmtpaWlSbGxsfkBAQFNlZeWzjsujo6NLx48fvz82Nna3p6enGjJkiG3q1Kn/cCyTmJj4rr+/f4vFYlHjx4/fP378+P3R0dGlHW0zLS3tgfDw8NNeXl52f3//lsmTJ3+Rm5sr3WslIqJB5qtywMUEjA05Ny8uAthxuOt1lQKAZAA79Dmh+isWctNhfcj+N3ry7xST/CXKycnxqqqqetdsNsPDw6MgOTn5z9u2bftuXFzcnLS0NEtERMS/V69e/WpGRsZUZ+uXl5cvraioSIyPj5+VlpbmFh4e/p9169a9n5WVNbKjbZ45c+bJ2tpak5+f32MjR45cUFpael1OTo63Y5l9+/aN8vf3/3daWppLXFzcC19++eWtbTGkpqY+WFxcfGdcXNyjM2fOtAwbNix/7969ozra3uzZs3OLiopeHTVq1Kupqanu8fHxN+zduzezpqbm9a7ap6ysjNOc5nQfTVP/aLXZvp7u6Hdx+vTp8+fXNgI+nueVOWFrAs42dFpPWVkZ8OS/AC0/v6vPDtV/zgZwNYAUADdDG77vlCjtjIEuQkRExKmKioohLi4uMJvNaujQobWRkZEvrVy58olRo0ZV+/r67t64cWNSW/nQ0NCzwcHBRcXFxRmJiYlv7d69+/YTJ0645ebmyuLFi+1Tp0794bJly15tKx8YGNg4ZsyYd1avXn2/s+2PHj36uMlksu3duzc4Kytr7OLFi/dMmTLl1cLCwh8CWk++qanJZ9++fQFt61itVvt11133u5UrV/4yJiZmt1LKdefOnaPbloeHh5+2Wq1VO3fujAK0G+9SU1PvX7p06V+vueaarTabzbJjx46vb/KYPHnyF8ePH088cOCAfxfNxQOMqI+Ul2xHyKQnBjoM4zv6FhDid/68nGeB1bu06cYW7afFVfuZNB54+mYg8RdAvcO1+hc/B1bsAOb9ouNtvTofePEL4ODxMKi5RwAActO1ADYCmAE1d6U+7ycAkqDmfrOz0F26uYvUTlxc3JLi4uJZ7efX1dV5BwcHnzfs7e3tXdXY2BjavqzNZhvX1NSEtWvX/tnT0/PPDvPR3NzstGedkZEx9cCBA8OTk5OfBYD58+d/NW7cuCNlZWW3AfhhWzmLxXLetRo3Nze7zWbzBYCGhgZ/X1/fXY7Lvby8TnS0r3V1dUGHDx8e7unp+XXCttvtGDJkSFNH6xARGVreL89Nt9109+S5++RQ1wi02oG95cAYfch+y0EgJqzjOt9ZCjw3F1j1DBAZdMRhyR4AzTi/09StDhSH63uZl5fX2cbGxtGO82pra4dZLJYj7cuazeY9bm5uSExMvLO+vl7aXk1NTbJ+/fpMZ/VXVVU9p5TCxo0bf+Hj42Pz8fGxHTp0KLSsrGxIenp6Rndi9PDwONHQ0DDCcV5dXZ1fR+U9PT2PX3XVVdsdY2xsbJTKykpLd7ZHRDToeFmAmyYDT3ysJfyiXcDnxcBt052X/+dK4Bf/BBY/CUQGnb9Mza0H8C8AP4Pc5A25KRTAPQDyugqDPfleFhoa+vGmTZvunzlz5p0Wi+Wjmpqa1yorK71jYmKeal923rx5auLEiUV79+59OT09/ejChQuXZGdnB9bX13/fYrEUFBQUbHIsn5OT415aWpp47bXXrg0MDLzXcdmGDRtKKisrnwGwoKsYAwMDX1u9evXLqampP/by8nr11KlTz5eXlw8ZO3as03/HCA4O/lVRUdHc6dOnP+Xt7f07APbm5ubZra2to5YtW/bKxbUQEdEg8do9wF1/AQK+B/h7A3+9B4gJ15YV7gQynwFqP9Te/+oj4MRZYNLPtPd1jbUA/gE1d45e24MA/gagHMApAG8CeKerEJjke1lhYeGDCQkJAZs3b/5bXV3du35+fnVJSUk/XrBgwRpn5UNCQtJcXV0/2bRpU77FYnFzc3OzBwYGVkZFRS1qX/bs2bO/aWhoMAUFBd09f/78nY7Lpk6d+snWrVtvbX8DnjPLli17ZerUqfGbN2/+Q319/R9HjRp1YPTo0UdMJpPTL+lZtGjRZ2lpaffu27fv99XV1b9SSmHo0KH1o0eP7vLGOyIiw3Mcpnfk5w189pjzZcnR5xI8ABy44OPUet47NfcMgA421DHeeEcAvr45cE1xcXF6L1fNA4yoj/DGu37i7Ma7vtcr/6LMnvwgNW3atBe8vb2fF5HakydPvl5RUWGNiYnp12/tIyKivsUkP0gdP3782yUlJT+x2+3w9fVtSE5OfmrhwoVLBjouIiLqPUzyg9Tu3bsjBjoGIiLqW/wXOiIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKD4jfeERFdoQ5WHEXI0bcGOgzjc7tyUyWfQkd9jQcYUR8REfAz3LB65Sl0HK4nIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig7pyH5JLRINWdYNCs22goxh4XywpGugQ6DLHJE9EV5xmGzDidWb54hsiBjoEusxxuJ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IeuRkg8I3P7PB66VWRLzRig932Tssu7zMjpR/2TDklVaM/FvrBcvXHFW4/h+t8H65Fde814rVR1Rfhk40aDDJX8FERKWlpd13qfVkZGRMcXd3V5mZmdf2Rlw0ODyw1A43M1B5vxn/zDbjvsV27Kh2npy9XAV3xQqen37hR87JBoXc/9rw00kmnPqhGT+73oTc/9pQ08hET3SpBn2Snzlz5m1jxoypsFqtdovFooYNG9YcFxe3KTMzM66ndSYmJr7l7+/f3NHynJwcL29vb3tiYuI77ZdlZ2cHWiwWNW3atBd6uv2LtWDBgnVNTU1SUFCwCeg6fqK6ZoX/fKXwdKIJVjdBUqjghijBBzud9+avDxbcFmNC5NALl60pVwjyAr49zgSzSfB/0SYM9wTm7mWSJ7pUgzrJp6SkPLpy5cq/W63WQ8nJyZMaGxtl0qRJk1xdXU/U1tbe1ZM6c3JyPLoqk5eXVxcVFVV86NChb7dfVlNT87ybm5vdx8fn8Z5sn6g/fFUDuJiAsX7y9by44YId1T2rT6kL32/vYFSAiLpvUCf5bdu2PT127Ni9mzZtmlJQULABAAoKCraUlJTMLCwsfBgAkpOTXw4KCmqwWCzKx8fHFhsbuzM7O3t4Wx1+fn6t8fHxSyMiImrc3d1VZWXlf9evX//9mpoaV3d3d+Xu7q5SU1N/1H7bAQEBPy8vL7fOnj37Jsf5hw4d+lZUVNSXeXl5DbNnz74xMjKy2mq12n19fVsnTpy4prOTiGnTpj0fFBTU4OHhoYKCghqmT5/+nOPy1NTUh8LDw097eXnZrVarPTo6uhQAMjIykkREZWZmxqelpf3AWfyhoaG1kydP/syxvuuuu27lyJEjT158y9OVrrYF8HE7f94Qd+Bs88Un5oQQQXkd8NEuO1psCu9vt2PfKaC+pXdiJRrMBm2ST09Pn33ixAnXgICANzsrZzabq6+55prvzJo1yyUhIWHW8ePHR1dUVHzuWKa0tHT62LFj75s9e7YpMDDw5uuvv/5tX1/flqamJmlqapJly5a91L7ehQsXLgsPD685duzYE23zZs2a9Z2KigrPgICAn2dlZUUXFRXNDQoKWjBjxgzvKVOmTD527FjcsWPH8pzFmZaWds/atWsfueqqq36blpZmueqqq36/Zs2aR2fOnHmXXve3CgsLXw4LC/s0JSVlyIwZM/yDgoJebV/P0qVL33QWf0RExCcHDx7MaCuXm5tr3rdvX2JYWNgHnbVfWVkZp6/A6Rkft0JecP6a9F4drK7Amebz1z1UWQNvN+m0/uPHj18w399D8LeEavyxxI7Av9owd0ctZkYIQr2lw3psNhvofAN9zHC696d7g6j242SDRFpa2pxly5b9ddasWTmLFi3K7+56CQkJn5SVlWUdPXrUCmg9+cjIyJUlJSVpbWUSExPf2r179+0nTpxw67gmYNq0aX/asGHDwykpKb55eXmnr7766u11dXVB+/fvHzZ58uR5lZWViQcPHvRzKP/8jh07Hm6rV0RUamrq/UuXLv1rbGzsLpvNZtm1a9eotvJXXXXVIRcXl7rt27dHX3311dubm5uH7NmzJ6x9HBkZGUkLFy4szMjImFRQUFDiLP7s7Gz/ZcuWVScmJt69ZMmSt2fMmPF4SUnJb2bMmDEkLy/vbCe7OTgPMIOra1bwfdWGHd8zY4yvlthvn29DiBV4bpq5w/WWHLLj7oV2HLzHpcMyrXaFyDdteHO2CemjnPdDymsVRrzORF98QyXix44Y6DCob0jXRbo2aHvyLi4u+wGgpaUlurNyKSkpj4aHh59uuzFv48aN325oaHB3LGOxWEp7EoOPj88vTCaTOnXq1O+zs7OHl5aWxoSGhr4LAA0NDSOPHj3q6+npqdpexcXFj9TW1ro6q6u+vn6Yh4fHEcd5np6eZQ0NDcP05QFeXl6HehInAOTn558YM2bM9oqKip8CQHl5+ZyoqKhNXSR4MigvN8FNYwRPFNlR16xQdFTh81KF26Kdf6TYlUJjq0KLTbve3tiq0Gw7d/63qVKhxaZwpknhkRV2hHmjwwRPRN03aP+KFi5cuMjf37/l+PHjd3dUJicnx2vNmjXPjRgxYsGMGTP8GxsbZeLEiZ/iwjOs87oUItKtLkZeXl5DVFRUcVlZ2XdPnTr1B1dXV7uPj88TAGCxWMrCw8Or6uvrpe3V0NAgTU1NTs/uPD09qxsaGs47pa+vrw/z8PCo1pcfr6uri+hOXB3FHxgY+OS+ffvGZWRkJO/fvz8kICDgye7UR8b02kwTGlqBgNdsuDnPhr/OMiFmmHZ4Fh5RsL587v/hVx1W8HjJhqy5dpSdBTxesmH2p+cOsz8U2zHsLzaEvWFDRR3w3290PBpARN3X8ZjZIHD11Vc/vnr16ucmTpxYFBgY+GBBQcGmzMzMmOrq6hctFsseHx+f37a2tsLFxaUqLy+vZvbs2bmlpaXf6KpeV1fXA3V1da7Z2dkj8vPzj3ZWNiAg4OdbtmxZVltb+39RUVFr8/LyGgDA39//sV27dm1OTEx8x9fX98ciUtvS0pLU2NiYtGLFimfb1xMYGPhyYWHhX1NSUn5htVqfP3v27GP79u2LmD59+t0AEBQU9JsVK1Z8kpSU9PrQoUN/CsClvr7+Dmf3C3QU/+LFi/8TGhpau3Xr1oUhISFnFi1aNK/rViaj8vMQfHaj82ScHCqoffjcx8uMcBPUIx33KT7KYVIn6guDticPAMuXL//99OnTbz979mzkqlWrNlgsFlVcXLypubk5wGq1vpWfn185efLkD7Zt2zbH3d1d7dmz5+8RERGru6rXy8vr5fDw8PIVK1Yc9vT0VKmpqQ91VHbhwoXLIyIiTp46dcolICDgsbb5BQUF2xISEm6srKzMWrVq1cklS5a0bt68eUFTU9PVzupZunTp61OmTHlp586dTyxevLh5165dv0xISHhxyZIlbwPA4sWLP01OTv7JoUOHblm2bNmZFStWnKisrHT6RTqdxR8REfFJRUWFR3h4+L+6agciIhpYg/bGO+qZ1NTUH61bt+5PKSkpw/Lz8090YxUeYNTreOOdhjfeGRpvvKP+lZOTM6SsrOznY8eOLe5mgiciogHEJE/dMm3atN8vWrToVHNzs2XEiBEXfFMfERFdfgb1jXfUfatWrXoUwKMDHQcREXUfe/JEREQGxSRPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJEREQGxSRPRERkUPwyHCK64riZgaNz+OS6DV8eAvjd9dQJPqCG+hoPMKI+IiLgZ7hh8QE1RERE1DEmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIol4EOgIjoSlDdoNBsG+gozvfFkqKBDoEuc0zyRETd0GwDRrx+eWX54hsiBjoEusxxuJ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyqT5N8YmLiW/7+/s19uY3+EBERcWrSpEmLBzoOR9HR0aUxMTG7e6OuyMjIquuvvz6/N+oi6msnGxS++ZkNXi+1IuKNVny4y95h2eVldqT8y4Yhr7Ri5N9aL1j++Gobrn6vFS4vtuLJosvr4TNEvaHLJB8REXHK1dVVubu7Kw8PDxUcHFw/bdq05/sjuDYZGRlTTCaTioiIqOnP7faVuLi4TeHh4aecLYuJidkbFRVV2Z/x7N+/f/j69euz296LiEpLS7uvP2Mg6q4HltrhZgYq7zfjn9lm3LfYjh3VymlZL1fBXbGC56c7/6iLGir4wzQTsiOlL0MmGjDd6slPmDBhSVNTk6SlpVnCwsKWFxUVPZKenj67r4NrU1VV9XsPDw9VVlY2tD+32xtycnI82s8LDAx84vDhw0PS09MzHednZ2eHlZaWRo0YMeKN/ouQ6MpR16zwn68Unk40weomSAoV3BAl+GCn89789cGC22JMiBzqvL47Yk3IjDTB263vYiYaSBc1XJ+Xl9c0fPjwu+x2OxoaGmYBQEJCwofDhw9v9vDwUKGhoWfT0tIe6Gj9nJwc90mTJi0cNmxYs6enpwoLCzszc+bM2zrbZm5urmtpaWlSbGxsfkBAQFNlZeWzjsujo6NLx48fvz82Nna3p6enGjJkiG3q1Kn/cCyTmJj4rr+/f4vFYlHjx4/fP378+P3R0dGlAJCRkZEkIiozMzPeoXynlxliYmL2Dh06tNVisaiAgICm5OTkV9uWpaam/shsNqukpKTX/fz8WpYsWVLffv1FixbNCwsLO1NZWfm04/yTJ0++4OHhYfPx8Xk6Ozvbf8KECcV+fn4tXl5e9sjIyKr09PTUjmLKyMiYOmbMmAqr1WofOnRoa1xc3KacnBzftuVZWVlXxcTE7PH19W21WCwqJCSkPj09PQM4/3JEUFBQAwAUFha+5u7urmJiYnYnJCR82Da/TXp6eorZbFYZGRlTO4qJqLd9VQO4mICxfud63nHDBTuqBzAoosvYRSX5nJwcr6qqqnfNZjM8PDwKkpOT/7xt27bvxsXFzUlLS7NERET8e/Xq1a929MFfXl6+tKKiIjE+Pn5WWlqaW3h4+H/WrVv3flZW1siOtnnmzJkna2trTX5+fo+NHDlyQWlp6XU5OTnejmX27ds3yt/f/99paWkucXFxL3z55Ze3tsWQmpr6YHFx8Z1xcXGPzpw50zJs2LD8vXv3jrqY/W5v6NChq6dOnXrVrFmz3MaNG/f62rVrH5g9e3Zu23K73Y7q6urMhISEyLS0tGHO6ggPD/+otLR0Yk5OjlfbvIMHD94QFRW1et68eS379+8vqa2tDZ88efKElJQUb19f3x0bN24scDYykJOT475p06Zl7u7uNdOnTw9MSEiYXF1dPbasrGwVAOTm5pq3bt1a3NLSYp0yZUrcrFmzXGJiYm4zmUwXXBY4duyYBwAkJyff39TUJDt27LjKz8/vpzU1NZaZM2fe0VauqqrqmYiIiOoFCxas6aytysrKOM3pXpuubQF83M6fb6s9ibPNqtN1jx8/3mX9p0+f6bLM5exy+R1xuveme4VSqtNXeHj4KVdXV+Xh4aGsVqs9NDT0zLRp055SSmHkyJHV11577WrH8iNGjDgbHx+/QCmFqVOnvuXn59eslEJOTo64u7urlJSUBx3LBwQENCYmJr7W0fYjIyOPR0VFVSilkJmZOdbFxUUlJSX9uW35+PHjSyMjI487ruPl5WWfNm3as0opREdH7x4/fvw+x+VhYWGnx48fX6qUQnp6ehIAlZGREd+23DHutjaIj49f3FGMwcHBdVOmTPlYKYWUlJQfAVDp6elTOmvX7OzsIZ6envakpKSXlFJIS0u7y2QyqYyMjPjMzMyx7evIyckxWywWlZqael/bfkdHR+9WSiE1NfUeFxcXlZWVNbyt/IwZM37h6uqqcnJyJC0t7U6TyaSysrLCOvodO+4fgK+30/aKjo7eHRsbu0OPxdXHx8eWnJz8fFfHjyK6CNM/alF43vkr8cMWtfGYXXn8qeW8dV5Yb1M5/2nttN7FB20q4o2WDpffmteqfr268zqOnrV3GNtAvYr3HOl+49KVpsv83J2XS3dOBOLi4pYUFxfPaj+/rq7OOzg4uNRxnre3d1VjY2No+7I2m21cU1MT1q5d+2dPT88/O8xHc3Oz0551RkbG1AMHDgxPTk5+FgDmz5//1bhx446UlZXdBuCHbeUsFstJx/Xc3NzsNpvNFwAaGhr8fX19dzku9/LyOtGd/XYmNzfXXF5evqysrCyhtrbWVUTQ1NSEkJCQgLYyIgJXV9cvO6snLy/vdFxc3ObDhw/fDuBHFRUVj44aNaqyoKCgpK3HvGrVqrWenp5fr6O3VXT7upqbm6O9vLxs+fn5VQ5tsK6lpQU2m218c3NznNVqtefn5x/u6X4HBwf/rqio6L3s7Gz/2trah+12O3x8fJ7oaX1Ezqz4bucfSXXNCq12YG+Nwhhfbch+S5VCjNPxMiK6pH+h8/LyOtvY2DjacV5tbe0wi8VypH1Zs9m8x83NDYmJiXfW19dL26upqUnWr1+f2b48AFRVVT2nlMLGjRt/4ePjY/Px8bEdOnQotKysbEjb9eSueHh4nGhoaBjhOK+urs6vbdpkMh0DALvd/vXHRHNzc1hH9dXU1LxSWlqadO211/7vzJkzXerr6yUwMLAeDm0pIpg3b57z230dBAYGPl5WVuY7a9as75aWlo4NDQ19AwDc3NyKAWDGjBnjHduqublZCgsLf9i+Hjc3t511dXXm7Oxsf4d9uN7V1RVms3mXm5vbltraWlN2dvaI9us6I3LhncZLlix539fXt+H06dO/Ky8v/0FUVFRxXl5eg5PVifqMl5vgpjGCJ4rsqGtWKDqq8Hmpwm3Rzj/K7EqhsVWhxQYoBTS2KjTbzv1ptti05XYFtOrLbfYu/3SJrhjd6sl3JDQ09ONNmzbdP3PmzDstFstHNTU1r1VWVnrHxMQ81b7svHnz1MSJE4v27t37cnp6+tGFCxcuyc7ODqyvr/++xWIpKCgo2ORYPicnx720tDTx2muvXRsYGHiv47INGzaUVFZWPgNgQVcxBgYGvrZ69eqXU1NTf+zl5fXqqVOnni8vLx8yduzYagCYP39+qZ+fn+348eNP5ubmLm1qasrdt29fqog4/Uu32Wy+JpNJmc3mvUopl8TExDcrKys9Q0JCLqrtAGDRokX54eHhpzds2PCBp6enzdvb+xk9pp3jx48/uG/fvsWZmZk3FhQUbMjOzo6ora29z2q1/ik/P/+8a+keHh7v+/n5vVJWVrYyOzt7ut1uD9+7d++vxo0bt2PevHkqNzf3g+Dg4FdLS0vXZWVlpZvN5j2NjY03uri47G/f7gDg7e1tb2xsjG8/f+TIkf8tLS29rbq62pKWltbpDZNEfeW1mSbctdCOgNds8PcA/jrLhJhh2olp4RGFzP/YUPuw9tG26rBCyifn7rz3eMmG6aHnRgx+sMiO93ec+1N/dp0N72aYcGcs/6WOjOGSknxhYeGDCQkJAZs3b/5bXV3du35+fnVJSUk/7uhmrJCQkDRXV9dPNm3alG+xWNzc3NzsgYGBlVFRUYvalz179uxvGhoaTEFBQXfPnz9/p+OyqVOnfrJ169Zb29+A58yyZctemTp1avzmzZv/UF9f/8dRo0YdGD169BGTyfT13fMTJkz4fzt27PjDzp07m0NCQk6OHj16xf79+6c7q8/Pz++HAQEBycuXL9/u6upqHz169MawsLAe//9+eHj4P4uKiu6/7rrrls+bN6+lbX5kZGR8RUXFZ8XFxWstFourxWKxBQUFHbFarS+2ryMvL68pIyNjVmlp6acrV66scnFxsYeHh28PCwtLA4B58+bZsrKyrj948OAXRUVF25qamkx+fn71V1999TedxRQbG/vejh077vD09PxeZGTk7u3bt0cDgK+v7yOnTp26ZcSIEacWLly4pKf7THQp/DwEn91odrosOVS+TvAAMCPcBPVIxwOW72Wa8Z7TcUQiYxClBt/QVGho6Nng4OA1xcXF6QMdy5UkNzdXioqKmmNiYt4oLCx8sJurDb4DjAypvFZhxOuX17fiFd9Qifix3boKR1eeXhlOuqSe/JVi2rRpL3h7ez8vIrUnT558vaKiwhoTE9Ov39pnBDU1NX+x2WymIUOGPDrQsRARUdcGRZI/fvz4t0tKSn5it9vh6+vbkJyc/BSHmy+O1Wq1m0wmNXHixCfz8vLqBjoeIiLq2qBI8rt3744Y6BiudLW1tXxiIRHRFYYf3ERERAbFJE9ERGRQTPJEREQGxSRPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJEREQGNSi+8Y6I6FK5mYGjc5w//W6gbPjyEMAH1FAnBuVT6Khf8QAj6iMiAn6GG1avPIWOw/VEREQGxSRPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJEREQGxSRPRERkUPwyHOpTIrIAwDAni4YBqO7ncK4EbBfn2C7OsV2cM0K7VCulMi61EiZ5GhAiUqKUih/oOC43bBfn2C7OsV2cY7ucw+F6IiIig2KSJyIiMigmeRoofxvoAC5TbBfn2C7OsV2cY7voeE2eiIjIoNiTJyIiMigmeSIiIoNikqd+ISKeIvIvESkVkd0iktNJ2QkiskpEduqvzP6MtT9dTLvo5S0iskNESvorxoHQ3XYRkW+IyAYR2a63y0/6O9a+JiJjRWStiHyl/xzjpIxZRP4iIvv0Nrt7IGLtT91sl8f142KrfpykD0SsA8lloAOgQeMRAGeUUlH6H2OhiEQppWodC4mIF4C5AG5RSq0TERcAQwYg3v7SrXZx8CyAdQDi+i3CgdHddjkGIFcpVS4iQwBsEJH1SqnCfo+477wO4C9KqX+IyP8BeANAarsytwKIAjAGgD+ATSKyRCl1sF8j7V/daZf1AF5UStWLSByAlSISrJRq6O9gBwp78tRfvgPtjxBKqb0ASgA466HfAmC1UmqdXrZVKXWi36Lsf91tF4hIMrQP8Q/6LbqB0612UUp9qZQq16dPA9gFIKIf4+xTIhIAYCKAj/RZHwGYKCLD2xX9DoA3lVJ2pVQVgM8AfLvfAu1n3W0XpdRCpVS9/nYrAIF2EjRoMMlTfwkHcMjhfRmAMCflogG0iMh8EdksIm+LiG+/RDgwutUu+gjHSwDu65+wBlx3j5evichVAKYAWNaHcfW3MABHlVI2ANB/luPCtrjo9rrCdbddHN0OYJ9S6kg/xHfZ4HA99QoR2Qjtg8aZwIuoygwgDUACgEoAfwTwIoC7LinAAdKL7fI8tKHJo86uPV5perFd2uoLBvA5gPvbevZEbURkOoCnAcwa6Fj6G5M89Qql1MTOlotIGbRh1Cp9VjiA5U6KlgFYppSq0Nf7EMA7vRhqv+rFdkkCkCUiTwCwAPAVka1KqWt6M97+0ovt0jZ0uwTAH5RSn/ZmnJeBwwBGiIhZKWUTETOAEH2+o7b2Ktbft+/ZG0132wUikgDgHwC+oZTa089xDjgO11N/+RTAvQCg90QnAVjgpNwnACaLiLf+PgPAln6JcGB0q12UUtcopUYqpUYC+C6AbVdqgu+mbrWLiPgDWAzgVaXU2/0aYT9QSh0HsBnAzfqsmwFs0q+7O/oUwA9ExKRfl74RwL/7K87+1t12EZFJAP4F4H+UUhv7NcjLBJM89ZfnAQwVkVIAeQDuUUqdBQAReUpE5gCAUqoMwO8BrBWRrQCuA/D/Bijm/tCtdhmEutsujwEYC+Be/R6OzSLyvYEJuc/MAfBDEfkKwA/199DvW2l70toHAPYD2Avtvy+eUkodGIhg+1F32uU1AB4A3nA4Pq4emHAHBr/WloiIyKDYkyciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSKiSyAiI0VEiUjoQMdC1B6TPBH1CxGJFJFPReSYiNSKyGER+a+IuOnL79T/L779eh3Nv1VPrr92smyFiDTp2zktIptE5Ft9s2dEly8meSLqL/MBVAAYB8Ab2vMJFkJ7MlhP3AvgJIDv619r2t7TSikrtKeOfQTgXyIytofbIroiMckTUZ/Tv352HIDXlVKnleaIUup1pVRTD+obDyAZwB0AgtHB43kB7XHF0L75zAzggm87E5EHRGRzu3mjRMQmIiP19+/qIw9nRWSniNzSSWxPisiSdvNWiMivHN7HishCEakSkTIR+Z2IuHZj14kuCpM8EfU5pdQJADsAvCUit4tItIj0tAcPAPcA2KqUyoM2QnBvRwX1ywEPAGiB8+cgfAjgKhGZ4DDvTgArlFIH9ferAUwAMBTAUwDeE5HongSuP1BnJYC5AEZAG9GYBeDnPamPqDNM8kTUX2YAWAHgR9AeLlIpIo+3S/ajROSU4wtaL/xrImKB9mzwd/VZbwPIdHLj2y/19Y8A+AaAbymlLri2r5SqgfaY2u/p9Qu0EYJ3HMq8rZQ6oZSyKaU+BrBV35+euB3AFqXUG0qpZqXUUQC/0+cT9SomeSLqF0qpaqXUL/THzA4F8DMAT0BPrroDSqmhji8A97er6tsArNAeHwpoPfkqAHe3K/esXkeAUmqqUmpeJ+G9C+AWfcg8VY9vLgDoT3Z7SkT26DfxnQIQB2D4xey/g1EAEtudyLwDIKiH9RF1iEmeiPqdUqpeKfUetB7xhItc/R5o19e3i8gxaD11X3R8A153LAbQBCAX2lD9x0qpBn3ZzdBOIL4FwFc/8diCjm8YPAvAq928EIfpQwCWtDuZGaLfJEjUq5jkiajPiYivfnNZrIi4ioiL/i9tsQAKL6KeaABJAL4J7eSg7XU9tJ5wVk/iU0rZAPwdwEMAboLDUD0AHwCt0EYLTCJyF7SefEc2AJgoItfp+/kgtN57m78DiBeRu0TEoo8URIpIRk9iJ+oMkzwR9YdmAAHQhsBPQkuYvwLwkFLq04uo514AG5VS85RSxxxeWwF8ik5uwOuGdwFMh3bJYL3D/PcBfAmgFMBRANHo5MREKbUCwB8BLID2L4OBAIoclh8DkALgRgAHAdQA+C+AyEuIncgpPk+eiIjIoNiTJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoP6/3apsKf5VCRQAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAEZCAYAAACU8lxmAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAALEwAACxMBAJqcGAAAPZtJREFUeJzt3Xl8FdXZB/Dfk/UmuQlkIRtZIIQtiQYhLNkISQjZW4tdXOpa626rb31frVVf69JatW/VWlfUWlul2mKVhH0PASTsOxK2AAkhQALZt3veP2Yil8vNQoAEht/387kf5s6cOfPMucN9Zs6c3BGlFIiIiMh4HPo7ACIiIro0mOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ7IgETkWRH5e3/H0Vsi8o6IPN3fcXRHROaKyO0Xo+z57LOILBORu3sap9V6U0Tk8Pmu199E5EkRmdHfcVyJmOSJrIhIkoisEpFTInJSRIpFZLy+7A4RWWlnnQMiMtVm3hQRUSLyuM38Ifr8Ov11QESeuLR71bf0BFQtIq49LH9Ouyql7lNKPX8JYrtdRNaLyGkROSwiL4uIU2/rU0plK6U+Pt+yfbnPVxp7JyJKqd8ppc77pIaY5Im+IyJeAAoA/BmAD4DBAH4LoLkX1d0O4CSA2zpZPlApZQZwE4BnRCSrF9u47IjIEADJABSA7/VvNHa5A3gEgB+AiQDSATzWnwEZzYWcNNHFxyRPdMYIAFBKfaaUaldKNSqlFiiltpxPJSLiAeCHAB4EMFxE4jorq5RaDWA7gBg79cwVkYds5m0Wken69Osicki/Kl0vIsmdxHPOlZF174OIOIjIEyKyV0ROiMjnIuKjLzOJyN/1+TUiUiIiAV3s/m0A1gD4K7QTHetthorILBGp0ut7U0RGA3gHQLzes1Gjl/2riLxgte7PRaRU7135WkSCrZYpEblPRPboMf5FRMRecEqpt5VSRUqpFqXUEQD/AJDYxf50ybrbvOPqXERe1Xsy9otItm3ZnuyziHiLSIHeVtX6dEgPY3IVkddEpFx/vWbbq6J3fx/Xj4NbrObniMgOEakVkSMi8pjVsjwR2aS38SoRudZq2QEReVxEtgCo16f/ZbPN10XkDX36ThHZqW9nn4jcq8/3ADAXQLCc6e0KFpvbTyLyPRHZrseyTG9T61geE5EtovXI/VNETD1pOyNikic641sA7SLysYhki4h3L+uZDqAOwBcA5sMm2XUQTSKAaAAb7RT5DNqVfkf5KADhAAr1WSUAxkDrdfgUwBe9/DJ7GMD1AFIABAOoBvAXfdntAAYACAXgC+A+AI1d1HUbtMT5DwCZHScEIuIIrZfkIIAh0HpJZiqldup1rlZKmZVSA20rFJE0AL8H8GMAQXodM22K5QEYD+BavVxmD/d9MrSTrItlIoDd0HoKXgbwge0JR0/2Gdp380fQPu8waG3+Zg9j+A2ASdCOjVgAEwA8ZbU8UI9vMLTP9z0RGakv+wDAvUopT2gnnksAQESuA/AhgHuhHQfvAvja5uThJgC5AAZC+3xyRMRTX98R2ufyqV72GLTPzAvAnQD+JCJjlVL1ALIBlOttY1ZKlVvvnIiMgPZ/4xEAgwDMATBbRFysiv0YQBaAodCOiTt61HIGxCRPpFNKnQaQBK2r+X0AVfpVo/WV6yT96uG7F7QvYWu3A/inUqod2pfajSLibFPmOLTu/BkAnlBKLbYT0pcAxohIuP7+FgCzlFLNerx/V0qdUEq1KaX+CMAVwEg79XTnPgC/UUod1ut+FsAPRet2bYX2pR6p926s19vpHCKSBC0pfa6UWg9gL4Cb9cUToJ1A/LdSql4p1aSUOmd8QyduAfChUmqDHt+voV0FD7Eq85JSqkYpVQZgKbQE1yURuQtAHIBXexhHTxxUSr2vf/YfQzsp6arnwy79c/23UqpBKVUL4EVoJ2E9cQuA55RSx5RSVdBuOd1qU+ZppVSzUmo5tJPGH+vzWwFEiYiXUqpaKbVBn38PgHeVUt/ox8HH0G5jTbKq8w2l1CG9B+wggA0AfqAvSwPQoJRao+9foVJqr9IsB7AA2m2envgJgEKl1EKlVCu0z88NQIJNLOVKqZMAZqMHx4NRMckTWVFK7VRK3aGUCoF2JRMM4DWrImuUUgOtXwDKOhaKSCiAVGhXsgDwFQATtCsca35KKW+l1Gil1BudxFIL7Qv4Rn3WTVb1Qu+S3Kl3SdZAu+L268VuhwP40uqkZSeAdmjJ6RNovREz9a7fl+2csHS4HcACpdRx/f2nONOLEQotAbb1Ir5gaFfvAAClVB2AE9CuRDsctZpuAGDuqkIRuR5a70C2Vby2ZZ606jJ+p4exfheHUqpBn+wylk627S4i74rIQRE5DWAFgIH6FXF3zmovfTrY6n21fsVsb/kNAHIAHBSR5SISr88PB/Arm5PbUJt6D9nE8SnO9ETdjDNX8dB7ytaIdvulRt9mT49d2+PBom+718eDkTHJE3VCKbUL2r3lc+6Xd+FWaP+vZovIUQD7oCX5Hv2ZlR2fAbhJ/7I1QbtKhWj33/8H2hWYt36ycQqAvXvR9dAGnEFf1xFaN2eHQ9CSnfXJi0kpdUQp1aqU+q1SKgralVIe7AwmFBE3PZYUETmq7/ujAGJFJFbfRpjYH5TV3aMwy6ElmY5teUDrXTjSzXp2iTbI8X0A+UqprZ2V00d0d3QZ39ebbXWhu33+FbRemYlKKS9otxUA+5+vrbPaC1pPk3WXt7fehucsV0qVKKW+D8AfwH8AfK6XOQTgRZtjxF0p9VkX+/QFgCmijSX4AfQkr3fx/xvaFXiAfuzOsdq38z0eBNoJR6+OB6NjkifSicgoEfmV/qXUcVV+E7SBZD11O7Tu0TFWrxug3Z/07UVYc6B9oT0H7RaARZ/vCaANQBUAJxF5Btr9TXu+BWASkVz9KvwpaF37Hd4B8GLHbQERGSQi39enU0XkGv3E4DS07lwLznU9tKv/KKv9Hg2gCNpJwVoAFQBeEhEP0Qb0dQx4qwQQYnNP1dpnAO4UkTF6gvgdgG+UUgc6Kd8p/f7+PwDcoJRae77rX0Td7bMntPvwNaINgvzf86j7MwBP6Z+jH4BnANj+ZsJvRcRFP1nMgzaew0VEbhGRAXo3+Gmc+azfB3CfiEzUx5J46MeTZ2dB6LcKlkEbW7BfH4sAAC7Qjr8qAG2iDU6cZrVqJQBfERnQSdWfA8gVkXT9eP4VtFsHq7pvmqsPkzzRGbXQBk59IyL10JL7NmhfIt0SkUnQEvJflFJHrV5fAyiF1SC6ntLvQc8CMBVW3Z3QutDnQUvgBwE04dzu0o46TgF4ANr9/yPQruytR9u/DuBrAAtEpBbafk/UlwUC+Be0L/ydAJZD68K3dTuAj5RSZdb7Dm2w2C3QrtLyAURCu71xGNq9VUAb3LUdwFEROafrXCm1CMDT0K7+KgAMw5lbGOfraWi3NeZYdcXP7WVdF6LLfYZ2i8gN2tiNNdA+6556AcA6AFsAbIV2b/wFq+VHoQ2uLId2wnOf3msFaD1RB/RbBPdB++yglFoH4OfQPs9qaMfzHT2I5VPYHLv6bahfQEvW1dC68r+2Wr4L2onKPv3WgPUtASildgP4KbQ/dT0O7bjKV0q19CCeq44o1V3PCBEREV2JeCVPRERkUEzyREREBsUkT0REZFBM8kRERAbFBwnQpcaRnQYlIuDAXaJLpie/idAtXskTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRR/8Y6IeqX463lA+cn+DoPo0nNxAvy8+juKXmGSJ6JeGRI0GBh8d3+HQXTpHZnR3xH0GrvriYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiILoWTtcAPXgI8bgLC7wE+XdF52WdnAs4/Asw3ay+ZXgeZHnFOOZl+G2S6gkzv0dOhmOQNKCAgoCk5Ofm1Cy1DREQ98OxM7WXrwfe1x9RWfgj841Hg/veA7WWd1/OTRKDuU+2lZpmhZu07a7lM9wbwJIDtPQ3tqk/yU6dOvXX48OEVZrPZYjKZlJ+fX0tsbOzG7Ozs2N7WmZiYOMPX17elu3IiolxcXJSrq6sym82WoUOHHs/IyPhRb7fbobKy0lRUVPQIAGRlZSWJiMrOzo7rrAwREV1k9U3Av9cAz98MmN2ApNHA98YDnyy/kFp/D+ANAMd7usJVneRTU1MfX758+d/MZvPB5OTk8U1NTTJ+/Pjxzs7OJ+rq6u7qTZ15eXlu51M+OTn5gebmZklJSQl3cXFpXLdu3We92S4REV1Gvi0HnByAEcFn5sWGA9sPdb7O7HWAz21A9C8BmX7/Wctk+gQAcQDeOZ8wruokv3Xr1udHjBixZ+PGjZPmzp27HgDmzp27ed26dVOLiop+CQDJycmvBwYGNppMJuXl5dUeExOzIzc3d1BHHT4+Pm1xcXGLw8PDq11dXVVlZeWXa9eu/Vl1dbWzq6urcnV1VWlpaY90F0thYeGhoKCgD2tqahxzcnIic3NzfWNjYzcOHDiwzWw2W4YPH16RlZU1qaN8cnLy6/7+/s0mk0l5enq2R0VFlVrHlJSU9BYALF++vAgAlixZUuLq6qri4uIW2ZYBgLS0tF+EhITUubm5qUGDBjUnJCT83WrZI46Ojio5Ofk1X1/fVjc3NzVy5MhDubm5QRfQ/ERExlXXBHi5nz1vgAdQ22i//I8TgZ1vAFUfAe/fDwDPQKbfBACQ6Y4A3gLwENQsy/mEcdUm+czMzGknTpxw9vf3f7+rco6Ojsevvfban2RkZDjFx8dnHDt2bFhFRcVX1mVKS0tTRowYcf+0adMcAgICbpowYcIH3t7erc3NzdLc3CxLlix5rbt4cnJyhpSXl//M29u7bc6cOaUHDx4sOnHixPD4+PiJKSkpAS4uLqc2bdq0ND8/3zk3N9d31apVv4iJifl1U1OTTJ48OTAoKOhP9upNSUlJBoC0tLTxzc3Nsm7duqm2ZbKyspJXrlz5enh4+Mz09HRTbGzsA5s3b7558uTJf+woY7FYcPLkyexJkyaFTJkyJebEiRMBx44d+7i7/SorK+O0gaeJrgZt7e3fTX93/Oe9CMuAm4GBPwVe+hLqpVna9MCfojH9KcBsAk43nPX/pfpgOeDpdnY9HdNRoUCwD8qOHAESRgHA6wB+qBd5AMAWqFlrzjd2p/NdwSja2toiAMDZ2XlHV+WWLVv2vNXbJfHx8V+VlZXlWJeJjIxcvnDhwo5RF9WJiYk9jqO4uPgtd3f3t5ydnS2DBg2qiouLuyE/P99x9+7do5OSkp7o6GHIzc1N2b1799GGhoY7TCbTLEdHRzQ2Nk7IycmJmDNnzj4Af+nxRm2cOHHiqYCAgNri4uKO0ZofjB079meHDx++HcCvOsqFh4d/v7CwsBJA5ZgxYzbV1NTEdFd3WFgYpw08TXQ1cHJ0/G76u+O/4DdnrpKfnQkBgGdvBAC4Ado9+TYLwprPpFnvshogOvTsejqfVoBWLYB0ACmQ6R25xwfAdZDpY6BmPdRV7FftlbyTk9M+AGhtbY3qqlxqaurjYWFhpzoG5m3YsOFHjY2NrtZlTCZTaWfrdycxMfGBhoYGOXXqlGNpaWngggULvm5raxvV1tYGFxeXko5yhYWFlWazub21tTWqsLDwRFJS0qNVVVVTli1bVhocHFyfnJz8Zm9jaGpqGmw2m88ayGEymb6tq6vz7Hjv4OCAOXPm7Op47+Tk1NDW1nZWOxARkc7DBEyfCDwzU0v4xTuBr0qAW1Psl/9qLVBdBygFrN0DAL8A0NFrfAeA0QDG6K91AH4L4DfdhXHVJvn58+cv8PX1bT127Finf2uYl5fnsWrVqpcGDx48b8qUKb5NTU0yduzYL3Dm7KpDu/UbEWnHBXByctrl5OSElpaWcR3zcnNzA+rq6hw7eh6WLFnyWmlpaWB6errr8OHD/1xcXPxgZmZmqp3q2rrbnslkOlJfX+9rPa+pqWm42WyuvZD9ICK6qr11D9DYAvjfCdz0J+Dte4Bo/Uq9aIf29/AdZq4EIh8APG8BbnsDAP4ANUu7Japm1UDNOvrdC2gBcBpq1qnuQrhqu+sB4Jprrnl65cqVL40dO7Y4ICDgoblz527Mzs6OPn78+B9NJtNuLy+v37W1tcHJyamqoKCgetq0afmlpaXf765eZ2fn/fX19c65ubmDCwsLj5xvXLNnz26PiYnZtWfPnmeys7MXOTg4lB08eHCpn59fk7u7+1+zs7Oj6+vr7/D09HyjsLDwUFpaWiUAiEirbV1OTk67RQTNzc3J0M7+zuHr6/vC5s2bVyQlJb07cODAXzQ1Nd20e/fu+HHjxr1+vrETEV119G76c/h4Av95wv6y5Cjt7+E7fPZftiXe6HR7ataUnoZ21V7JA8DSpUv/kJKSclttbW3EihUr1ptMJlVSUrKxpaXF32w2zygsLKycOHHiJ1u3br3P1dVV7d69+2/h4eEru6vXw8Pj9bCwsPJly5Ydcnd3V2lpab8439jCw8Mn+/j47Fu1atW65cuXVzU1NfmMGTMmY/bs2a0AnMrKyn62dOnSgyaTSW3btu2lhISEd+fNm3dObAUFBdXjxo1bsG7dulfd3d3V+PHjF9iWmTdvXlFSUtJ/7d+//5bFixc3bd68+b1rr7328xUrVjx6vnETEdHlQ5RS/R0DGRsPMIMqX7cNweOf6e8wiC69IzOAYJ++3qrtbeFeuaqv5ImIiIyMSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCu6kfNElHvHag4guAjM/o7DKJLz+XKTZV8Ch1dajzADEpEwO8PokuGT6EjIiKizjHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGdSV+5Bcok4cb1Roae/vKIzv60XF/R0CEXWDSZ4Mp6UdGPwOs/ylVvK98P4OgYi6we56IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCZ5IiIig2KSJyIiMigmeSIiIoNikiciIjIoJnkiIiKDYpInIiIyKCb5PpaYmDjD19e3pb/j6I6IqPT09Pt7vf6rbT7yatuXHq+1IfzdNny609Jp2VfWWhDzURs8X2/D0Pfa8Mrac8u+vt6Coe+1weO1Noz+sA3fnlS9DY2I6KrBJN8L4eHhNc7OzsrV1VW5ubmpoKCghsmTJ7/SlzFkZWVNcnBwUOHh4dV9ud3z8BcALZUPOOIfuY64f6EF24/bT8wKwN9yHFH9sCPm/dARb260YOauM4l+xhYLPthqQeF0R9T90hEF0x3h59Y3O0FEdCVjku+lMWPGLGpubpb09HRTaGjo0uLi4scyMzOn9dX2q6qq/uDm5qbKysoG9uV2e0JebfMAcAOAp80ugqQQwfciBZ/ssH81/z8THDA2QODkIBjpI/h+pKD4iHZCYFEKv11twZ9SHRDlJxARDBso8HGTvtshIqIrFJP8BSooKGgeNGjQXRaLBY2NjRkAEB8f/+mgQYNa3NzcVEhISG16evqDna2fl5fnOn78+Pl+fn4t7u7uKjQ09PTUqVNv7Wqb+fn5zqWlpUkxMTGF/v7+zZWVlS9aL4+KiiodPXr0vpiYmF3u7u5qwIAB7QkJCX+3LpOYmPiRr69vq8lkUqNHj943evTofVFRUaWdbTM9Pf3BsLCwUx4eHhZfX9/WiRMnfp2fn99Zph0BoE095vRtx4zYQYLtx7vaK41SCkVHFKJ9taoP12qvbceB0He17vz/LW6HRbG7noioO0zyFygvL8+jqqrqI0dHR7i5uc1NTk7+89atW2+MjY29Lz093RQeHv6vlStXvpmVlZVgb/3y8vLFFRUViXFxcRnp6ekuYWFh/16zZs3HOTk5Qzrb5unTp5+tq6tz8PHxeWLIkCHzSktLx+Xl5Xlal9m7d+9QX1/ff6WnpzvFxsa++s0339zSEUNaWtpDJSUld8TGxj4+depUk5+fX+GePXuGdra9adOm5RcXF785dOjQN9PS0lzj4uK+t2fPnuzq6up3OlnFDOA0AJSVlQEABrgCx2ubvivQMd92+tG5NbAo4M4YQVlZGQ7XavO/2tWArbc7YulPHPHJ1lZ8sFV1WQ/1jc7an9Oc5vSFT18MonhFdN7Cw8NrKioqBjg5OcHR0VENHDiwLiIi4rXly5c/M3To0OPe3t67NmzYkNRRPiQkpDYoKKi4pKQkKzExccauXbtuO3HihEt+fr4sXLjQkpCQ8PCSJUve7CgfEBDQNHz48A9Xrlz5gL3tDxs27JiDg0P7nj17gnJyckYsXLhw96RJk94sKip6GNCu5Jubm7327t3r37GO2Wy2jBs37vfLly//TXR09C6llPOOHTuGdSwPCws7ZTabq3bs2BEJaAPv0tLSHli8ePHb11577Zb29nbT9u3bRwCAvNq2DEBKJ81TDOBhAMXqMSd3aLfc8ccSC5YdUpg93bHTdn1zgwV/XGdB0U2OCPHUruQ3ViqM/aQdy37iiJRQbd4fSyxYeUThy+vt11VepzD4nfZOt0MXR8n3KhE3YnB/h0FkVBflnqTTxajkahQbG7uopKQkw3Z+fX29Z1BQ0Fnd3p6enlVNTU0htmXb29tHNjc3Y/Xq1X92d3f/s9V8tLS02L2yzsrKSti/f/+g5OTkFwFgzpw5344cOfJwWVnZrdCSKwDAZDKdtF7PxcXF0t7e7g0AjY2Nvt7e3jutl3t4eJzobF/r6+sDDx06NMjd3V0BgBsAi8WCAQMGNFdWVppsy+v35J3k1bbh6jHtENtcpRDt19kWgA+3WvDSWgtW3HgmwQPASB/AxfHso114O56IqEfYXX+ReXh41DY1NQ2znldXV+dnMpkO25Z1dHTc7eLigsTExDsaGhqk49Xc3Cxr167Ntld/VVXVS0opbNiw4UkvL692Ly+v9oMHD4aUlZUNyMzMzOpJjG5ubicaGxvPugSrr6/36ay8u7v7sVGjRm2zjrGpqUnsJXgAUI851QOYBeC5+haF4iMKX5Uq3Bpl/3D7xw4LniyyYOGPHBEx8OwM7u4s+MlIwcslFtS2KByuVXhviwV5w5jpiYi6wyv5iywkJGTmxo0bH5g6deodJpPps+rq6rcqKys9o6Ojn7MtO3v2bDV27NjiPXv2vJ6ZmXlk/vz5i3JzcwMaGhp+ZjKZ5s6dO3ejdfm8vDzX0tLSxOuuu251QEDAvdbL1q9fv66ysvIFAPO6izEgIOCtlStXvp6Wlvaoh4fHmzU1Na+Ul5cPGDFihN2hcUFBQU8VFxfPSklJec7T0/P3ACwtLS3T2trahi5ZsuSNTjbzAIAP/d9qh68b8HaGA6L9tMRcdFgh+9/tqPuldvg9tdKCE03A+L+f6WL/aZTgnQytO/7NdAfcs8CC4LfbMdAE/PwaB9wVwyRPRNQdJvmLrKio6KH4+Hj/TZs2vVdfX/+Rj49PfVJS0qPz5s1bZa98cHBwurOz8+cbN24sNJlMLi4uLpaAgIDKyMjIBbZla2trf9vY2OgQGBh495w5c3ZYL0tISPh8y5Ytt9gOwLNnyZIlbyQkJMRt2rTp5YaGhv8bOnTo/mHDhh12cHCw+yM9CxYs+E96evq9e/fu/cPx48efUkph4MCBDcOGDets4B3UY04nAVwP/Z68teQQ+S7BA8D+e7o+DL1cBTPzO7+XT0RE9nHgHQH4bnDgqpKSksyLXHWfH2AceNc3OPCO6JLiwDvqvcmTJ7/q6en5iojUnTx58p2KigpzdHR0n/5qHxERXVpM8lepY8eO/WjdunW/slgs8Pb2bkxOTn5u/vz5i/o7LiIiuniY5K9Su3btCu/vGIiI6NLin9AREREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRR/8Y4Mx8UROHIfn1p3qa3/5iDAB9QQXdb4FDq61HiAGZSIgN8fRJfMRXkKHbvriYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKD4PHki6pXir+cB5Sf7O4z+5+IE+Hn1dxREdjHJE1GvDAkaDAy+u7/D6H9HZvR3BESdYnc9ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8EVF/OVkL/OAlwOMmIPwe4NMVnZdduhVIfQYY8FNgyL3nLl+1C5jwP4DnzcC1jwIrd166uOmKwSR/BRMRlZ6efv+F1pOVlTXJ1dVVZWdnX3cx4iIiG8/O1F62Hnxfe1Rt5YfAPx4F7n8P2F5mvw4PE3BXGvDKbecuO1kL5P8O+O/rgZpPgP+5XntfXXcx94KuQFd9kp86deqtw4cPrzCbzRaTyaT8/PxaYmNjN2ZnZ8f2ts7ExMQZvr6+LZ0tz8vL8/D09LQkJiZ+aLssNzc3wGQyqcmTJ7/a2+2fr3nz5q1pbm6WuXPnbgS6j5+ILoL6JuDfa4DnbwbMbkDSaOB744FPltsvP2E4cOsUICLg3GWrdgOB3sCPEgBHR+CnKcAgL2DWmku6C3T5u6qTfGpq6uPLly//m9lsPpicnDy+qalJxo8fP97Z2flEXV3dXb2pMy8vz627MgUFBfWRkZElBw8e/JHtsurq6ldcXFwsXl5eT/dm+0R0hfi2HHByAEYEn5kXGw5sP9S7+pSyeQ9gWye9AnTVuKqT/NatW58fMWLEno0bN06aO3fuegCYO3fu5nXr1k0tKir6JQAkJye/HhgY2GgymZSXl1d7TEzMjtzc3EEddfj4+LTFxcUtDg8Pr3Z1dVWVlZVfrl279mfV1dXOrq6uytXVVaWlpT1iu21/f/9fl5eXm6dNmzbdev7BgwdviIyM/KagoKBx2rRp10dERBw3m80Wb2/vtrFjx67q6iRi8uTJrwQGBja6ubmpwMDAxpSUlJesl6elpf0iLCzslIeHh8VsNluioqJKASArKytJRFR2dnZcenr6z+3FHxISUjdx4sT/WNc3bty45UOGDDl5/i1PRKhrArzcz543wAOobTz/uuJHAuUngc+KgNY24OOlwN6jQEPzxYmVrlhXbZLPzMycduLECWd/f//3uyrn6Oh4/Nprr/1JRkaGU3x8fMaxY8eGVVRUfGVdprS0NGXEiBH3T5s2zSEgIOCmCRMmfODt7d3a3Nwszc3NsmTJktds650/f/6SsLCw6qNHjz7TMS8jI+MnFRUV7v7+/r/OycmJKi4unhUYGDhvypQpnpMmTZp49OjR2KNHjxbYizM9Pf2e1atXPzZq1Kjfpaenm0aNGvWHVatWPT516tS79LpvKCoqej00NPSL1NTUAVOmTPENDAx807aexYsXv28v/vDw8M8PHDiQ1VEuPz/fce/evYmhoaGfdNV+ZWVlnDbwNGnstk/ei7AMuBkY+FPgpS+hXpqlTQ/8KRrTnwLMJuB0w1nrVh8sBzzdOq8TQOWxY+fO9/VE5Xt3Af/3NRBwF+pnrQSmXguE+F4Wxwmnezd9MYiy7eK5SqSnp9+3ZMmStzMyMvIWLFhQ2NP14uPjPy8rK8s5cuSIGdCu5CMiIpavW7cuvaNMYmLijF27dt124sQJl67qmjx58p/Wr1//y9TUVO+CgoJT11xzzbb6+vrAffv2+U2cOHF2ZWVl4oEDB3ysyr+yffv2X3bUKyIqLS3tgcWLF78dExOzs7293bRz586hHeVHjRp10MnJqX7btm1R11xzzbaWlpYBu3fvDrWNIysrK2n+/PlFWVlZ4+fOnbvOXvy5ubm+S5YsOZ6YmHj3okWLPpgyZcrT69at++2UKVMGFBQU1Haxm1fnAXYVKF+3DcHjn+m+oNEdmQEE+3RdpmPQ3bM3nplX3wR43wZsfw0YrnfZ3/a6VtdLt3Ze16LNwN1vAQfe7bxMWzsQcT/w/v1AJsfTXqHkYlRy1V7JOzk57QOA1tbWqK7KpaamPh4WFnaqY2Dehg0bftTY2OhqXcZkMpX2JgYvL68nHRwcVE1NzR9yc3MHlZaWRoeEhHwEAI2NjUOOHDni7e7urjpeJSUlj9XV1Tnbq6uhocHPzc3tsPU8d3f3ssbGRj99ub+Hh8fB3sQJAIWFhSeGDx++raKi4r8BoLy8/L7IyMiN3SR4IuqMhwmYPhF4ZqaW8It3Al+VALem2C9vsQBNLUBru3b/vakFaGk9s3zjPq2r/nQD8NhfgVBfJni6epP8/PnzF/j6+rYeO3bs7s7K5OXleaxateqlwYMHz5syZYpvU1OTjB079guce4bVbv1GRNrRAwUFBY2RkZElZWVlN9bU1Lzs7Oxs8fLyegYATCZTWVhYWFVDQ4N0vBobG6W5udnu2Z27u/vxxsbGwdbzGhoaQt3c3I7ry4/V19eH9ySuzuIPCAh4du/evSOzsrKS9+3bF+zv7/9sT+ojok68dQ/Q2AL43wnc9Cfg7XuA6DBtWdEOwHzzmbIrdgBuNwI5LwBlx7Xpac+dWf7yfwC/O4DQe4CKauDLx/tyT+gy5dTfAfSna6655umVK1e+NHbs2OKAgICH5s6duzE7Ozv6+PHjfzSZTLu9vLx+19bWBicnp6qCgoLqadOm5ZeWln6/u3qdnZ3319fXO+fm5g4uLCw80lVZf3//X2/evHlJXV3dTyMjI1cXFBQ0AoCvr+8TO3fu3JSYmPiht7f3oyJS19ramtTU1JS0bNmyF23rCQgIeL2oqOjt1NTUJ81m8yu1tbVP7N27NzwlJeVuAAgMDPztsmXLPk9KSnpn4MCB/w3AqaGh4XZ74wU6i3/hwoX/DgkJqduyZcv84ODg0wsWLJjdfSsT0Vnd9NZ8PIH/PGF/WXIUUPfpmfdTYgA1q/NtfPZfvY+PDOuqvZIHgKVLl/4hJSXlttra2ogVK1asN5lMqqSkZGNLS4u/2WyeUVhYWDlx4sRPtm7dep+rq6vavXv338LDw1d2V6+Hh8frYWFh5cuWLTvk7u6u0tLSftFZ2fnz5y8NDw8/WVNT4+Tv7//d//a5c+dujY+Pv76ysjJnxYoVJxctWtS2adOmec3NzdfYq2fx4sXvTJo06bUdO3Y8s3DhwpadO3f+Jj4+/o+LFi36AAAWLlz4RXJy8q8OHjx485IlS04vW7bsRGVlpd0f0ukq/vDw8M8rKircwsLC/tldOxARUf+6agfeUe+kpaU9smbNmj+lpqb6FRYWnujBKjzADIoD73Q9GXhHdP448I76Vl5e3oCysrJfjxgxoqSHCZ6IiPoRkzz1yOTJk/+wYMGCmpaWFtPgwYPP+aU+IiK6/FzVA++o51asWPE4AA7XJSK6gvBKnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKP4ZDRL1yoOIIgo/M6O8w+p8Lv0bp8sUH1NClxgPMoEQE/P4gumT4gBoiIiLqHJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGZRTfwdARFem4q/nAeUn+zuMM1ycAD+v/o6C6LLCJE9EvTIkaDAw+O7+DuOMIzP6OwKiyw6764mIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKAuaZJPTEyc4evr23Ipt9EXwsPDa8aPH7+wv+OwFhUVVRodHb3rYtQVERFRNWHChMKLURfRFelkLfCDlwCPm4Dwe4BPV3RedulWIPUZYMBPgSH3nrv86U+Bax4BnH4IPDvzkoVM1BPdJvnw8PAaZ2dn5erqqtzc3FRQUFDD5MmTX+mL4DpkZWVNcnBwUOHh4dV9ud1LJTY2dmNYWFiNvWXR0dF7IiMjK/synn379g1au3Ztbsd7EVHp6en392UMRH3i2Zn2E++D72uPqq38EPjHo8D97wHby+zX4WEC7koDXrnN/vLIIODl24DccRcvbqJe6tGV/JgxYxY1NzdLenq6KTQ0dGlxcfFjmZmZ0y51cB2qqqr+4ObmpsrKygb25XYvhry8PDfbeQEBAc8cOnRoQGZmZrb1/Nzc3NDS0tLIwYMHv9t3ERJd5eqbgH+vAZ6/GTC7AUmjge+NBz5Zbr/8hOHArVOAiAD7y29PBbLHAp7n/Ncn6nPn1V1fUFDQPGjQoLssFgsaGxszACA+Pv7TQYMGtbi5uamQkJDa9PT0BztbPy8vz3X8+PHz/fz8Wtzd3VVoaOjpqVOn3trVNvPz851LS0uTYmJiCv39/ZsrKytftF4eFRVVOnr06H0xMTG73N3d1YABA9oTEhL+bl0mMTHxI19f31aTyaRGjx69b/To0fuioqJKASArKytJRFR2dnacVfkubzNER0fvGThwYJvJZFL+/v7NycnJb3YsS0tLe8TR0VElJSW94+Pj07po0aIG2/UXLFgwOzQ09HRlZeXz1vNPnjz5qpubW7uXl9fzubm5vmPGjCnx8fFp9fDwsERERFRlZmamdRZTVlZWwvDhwyvMZrNl4MCBbbGxsRvz8vK8O5bn5OSMio6O3u3t7d1mMplUcHBwQ2ZmZhZw9u2IwMDARgAoKip6y9XVVUVHR++Kj4//tGN+h8zMzFRHR0eVlZWV0FlMRFeEb8sBJwdgRPCZebHhwPZD/RcT0UVyXkk+Ly/Po6qq6iNHR0e4ubnNTU5O/vPWrVtvjI2NvS89Pd0UHh7+r5UrV77Z2Rd/eXn54oqKisS4uLiM9PR0l7CwsH+vWbPm45ycnCGdbfP06dPP1tXVOfj4+DwxZMiQeaWlpePy8vI8rcvs3bt3qK+v77/S09OdYmNjX/3mm29u6YghLS3toZKSkjtiY2Mfnzp1qsnPz69wz549Q89nv20NHDhwZUJCwqiMjAyXkSNHvrN69eoHp02blt+x3GKx4Pjx49nx8fER6enpfvbqCAsL+6y0tHRsXl6eR8e8AwcOfC8yMnLl7NmzW/ft27eurq4ubOLEiWNSU1M9vb29t2/YsGGuvZ6BvLw8140bNy5xdXWtTklJCYiPj594/PjxEWVlZSsAID8/33HLli0lra2t5kmTJsVmZGQ4RUdH3+rg4HDObYGjR4+6AUBycvIDzc3Nsn379lE+Pj7/XV1dbZo6dertHeWqqqpeCA8PPz5v3rxVXbVVWVkZpw08fbnpScynTp06e35dE+DlflaZE+3NQG1jl/VUHjt2/tviNKfPY/qiUEp1+QoLC6txdnZWbm5uymw2W0JCQk5Pnjz5OaUUhgwZcvy6665baV1+8ODBtXFxcfOUUkhISJjh4+PTopRCXl6euLq6qtTU1Iesy/v7+zclJia+1dn2IyIijkVGRlYopZCdnT3CyclJJSUl/blj+ejRo0sjIiKOWa/j4eFhmTx58otKKURFRe0aPXr0XuvloaGhp0aPHl2qlEJmZmYSAJWVlRXXsdw67o42iIuLW9hZjEFBQfWTJk2aqZRCamrqIwBUZmbmpK7aNTc3d4C7u7slKSnpNaUU0tPT73JwcFBZWVlx2dnZI2zryMvLczSZTCotLe3+jv2OiorapZRCWlraPU5OTionJ2dQR/kpU6Y86ezsrPLy8iQ9Pf0OBwcHlZOTE9rZZ2y9fwC+207HKyoqaldMTMx2PRZnLy+v9uTk5Fe6O34UGdaRkq1K4QeXz+vICfuB5r6g1IBbtJfrj7VXx/vcF5TasFcpt5+cvc6r/1Eq78WuG2DhJqXC7+l8+S1/Uup/PzuvNiWy0m1+7snLqScnArGxsYtKSkoybOfX19d7BgUFlVrP8/T0rGpqagqxLdve3j6yubkZq1ev/rO7u/ufreajpaXF7pV1VlZWwv79+wclJye/CABz5sz5duTIkYfLyspuBfBwRzmTyXTSej0XFxdLe3u7NwA0Njb6ent777Re7uHhcaIn+21Pfn6+Y3l5+ZKysrL4uro6ZxFBc3MzgoOD/TvKiAicnZ2/6aqegoKCU7GxsZsOHTp0G4BHKioqHh86dGjl3Llz13VcMa9YsWK1u7v7d+vobRVlW1dLS0uUh4dHe2FhYZVVG6xpbW1Fe3v76JaWlliz2WwpLCzsdf9jUFDQ74uLi/+am5vrW1dX90uLxQIvL69nelsfUZ8p+M2Z6Y5Bd8/eeGZefRPQZgH2lAPD9S77zQeA6NA+C5HoUrmgP6Hz8PCobWpqGmY9r66uzs9kMh22Levo6LjbxcUFiYmJdzQ0NEjHq7m5WdauXZttWx4AqqqqXlJKYcOGDU96eXm1e3l5tR88eDCkrKxsQMf95O64ubmdaGxsHGw9r76+3qdj2sHB4SgAWCyW77rVW1paOv3fXV1d/UZpaWnSdddd9+OpU6c6NTQ0SEBAQAOs2lJEMHv2bNVdbAEBAU+XlZV5Z2Rk3FhaWjoiJCTkXQBwcXEpAYApU6aMtm6rlpYWKSoqeti2HhcXlx319fWOubm5vlb7MMHZ2RmOjo47XVxcNtfV1Tnk5uYOtl3XHhE5Z96iRYs+9vb2bjx16tTvy8vLfx4ZGVlSUFDQaGd1oiuLhwmYPhF4ZqaW8It3Al+VALem2C9vsQBNLUBrO6CUNt3SemZ5a5s2z6K0k4emFqC9vW/2hcjGBSX5kJCQmd9++23i1KlT78jLy3NNTEz8oLKy0tPX1/c527KzZ89W0dHRxXv27Hk9MzNzKgDk5uYGpKamPpmdnX2dbfm8vDzX0tLSxOuuu251UlJSbGJi4pjExMQxqamp1w4aNKilsrLyhZ7EGBAQ8FZpaWlEWlrao/n5+c7JycmvlZeXD+hYPmfOnFJvb+/2Y8eOPZufn+88bdq06Xv37u10gFt7e7u3g4ODcnR03KOUckpMTPywsrLSvbPyXVmwYEFhSEjIqfXr13/i7u7e7unp+YIe045Ro0Yd2Lt378Ls7OxxAJCbmxuekpLyUm5u7jlDet3c3D728fFpLisrW56bm+ubnZ193Z49e54aOXLk9tmzZys3N7dPgoKC6ktLS9fk5ORE5efnO2ZkZNxgr90BwNPT09LU1BRnO3/IkCFflpaW3rp///5Af3//p3qzz0SXpbfuARpbAP87gZv+BLx9DxAdpi0r2gGYbz5TdsUOwO1GIOcFoOy4Nj3N6ivv529r8z4rAl78lzbd2Uh9okusR931nSkqKnooPj7ef9OmTe/V19d/5OPjU5+UlPRoZ4OxgoOD052dnT/fuHFjoclkcnFxcbEEBARURkZGLrAtW1tb+9vGxkaHwMDAu+fMmbPDellCQsLnW7ZsucV2AJ49S5YseSMhISFu06ZNLzc0NPzf0KFD9w8bNuywg4PDd6Pnx4wZ81/bt29/eceOHS3BwcEnhw0btmzfvn12T+N9fHwe9vf3T166dOk2Z2dny7BhwzaEhob2+u/3w8LC/lFcXPzAuHHjls6ePfu7y4GIiIi4ioqK/5SUlKw2mUzOJpOpPTAw8LDZbP6jbR0FBQXNWVlZGaWlpV8sX768ysnJyRIWFrYtNDQ0HQBmz57dnpOTM+HAgQNfFxcXb21ubnbw8fFpuOaaa35gL6aYmJi/bt++/XZ3d/c7IyIidm3bti0KALy9vR+rqam5efDgwTXz589f1Nt9Juo31t301nw8gf88YX9ZchRQ9+mZ91NiADWr82389WHtRXQZEKW67VU2nJCQkNqgoKBVJSUlmf0dy5UkPz9fiouLW6Kjo98tKip6qIerXX0H2FWifN02BI+/jIZlHJkBBPt0X47oynDufdNeuKAr+SvF5MmTX/X09HxFROpOnjz5TkVFhTk6OrpPf7XPCKqrq//S3t7uMGDAgMf7OxYiIureVZHkjx079qN169b9ymKxwNvbuzE5Ofk5djefH7PZbHFwcFBjx459tqCgoL6/4yEiou5dFUl+165d4f0dw5Wurq6OTywkIrrC8IubiIjIoJjkiYiIDIpJnoiIyKCY5ImIiAyKSZ6IiMigmOSJiIgMikmeiIjIoJjkiYiIDIpJnoiIyKCuil+8I6KL70DFEQQfmdHfYZzhwq8zIltX5VPoqE/xADMoEQG/P4gumYvyFDp21xMRERkUkzwREZFBMckTEREZFJM8ERGRQTHJExERGRSTPBERkUExyRMRERkUkzwREZFB8cdw6JISkXkA/Po7jgvkB+B4fwdxGWK72Md2ORfbxL6u2uW4UirrQjfAJE/UDRFZp5SK6+84LjdsF/vYLudim9jXF+3C7noiIiKDYpInIiIyKCZ5ou69198BXKbYLvaxXc7FNrHvkrcL78kTEREZFK/kiYiIDIpJnoiIyKCY5IlsiIi7iPxTREpFZJeI5HVS7vsisl5EtonIdhH5VV/HeqmJyAgRWS0i3+r/DrdTxlFE/iIie/U2u7s/Yu1LPWyXp/XjYot+nGT2R6x9qSftYlV2pIg0iMirfRljX+tpm4jIj0Vkq/59slVEAi7G9pnkic71GIDTSqlIAPkAZoiI2U65owDylVIxABIA3C8iyX0YZ194B8BflFIjAPwFwLt2ytwCIBLAcADxAJ4VkSF9FmH/6Em7rAUwXil1LYC7APxTRNz6MMb+0JN2gYg46sv+03eh9Ztu20RE4gA8CyBD/z5JAnDqYmycA++IbIjIdgC3K6XW6e8LAHyslPqim/VmA/inUurvfRDmJSci/gC+BeCrlGrXv5hPABiulKqyKlcI4COl1L/0928COKiUeqU/4r7UetouNusIgBoA0Uqpw30WbB86n3YRkd8AaAZgBmBWSj3W5wH3gfP4P/QPAIuVUh9e7Bh4JU90rjAAB63elwEI7WoFERkFYBKAJZcwrr4WCuCIUqodAPR/y3FuW5x3e13hetou1m4DsNeoCV7Xo3YRkVgAmQD+1OcR9r2eHitRACJEZIWIbBCRp/QTwwvmdDEqIbqSiMgGaInJnvO+DyYiQQC+AvCAUqr8QmIj4xGRFADPA8jo71j6m4g4Q/vb8Dv1K9v+Duly4QjgWmjHiAuAedBOlv92oRUzydNVRyk1tqvlIlIGIBxAR3daGIClnZT1B7AIwMvddedfgQ4BGCwijlZdjcH6fGsd7VWiv7e9sjeanrYLRCQewN8BfF8ptbuP4+xrPWmXIADDAMzRE/xAaHczvJRS9/R1wH3gfP4P/Usp1QygWUS+AjABFyHJs7ue6FxfALgXAPSRsOOhnVmfRUR8ASwE8KZS6oM+jbAPKKWOAdgE4CZ91k0ANtq57/wFgJ+LiIOIDAJwPYB/9VWcfa2n7SIi4wH8E8APlVIb+jTIftCTdlFKlSml/JRSQ5RSQwC8BuB9gyb48/k/9CmAaaJxBpAOYPPFiIFJnuhcrwAYKCKlAAoA3KOUqgUAEXlORO7Tyz0BYASAe0Vkk/66s39CvmTuA/CwiHwL4GH9PURkjj4iGAA+AbAPwB4AawA8p5Ta3x/B9qGetMtbANwAvGt1fFzTP+H2mZ60y9WmJ20yE8AxADugnRRsB3BRLhw4up6IiMigeCVPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJERBdARIaIiBKRkP6OhcgWkzwR9QkRiRCRL0TkqIjUicghEflSRFz05Xfov01gu15n82/Rk+v/2lm2TESa9e2cEpGNInLDpdkzossXkzwR9ZU5ACoAjATgCe2xtPMB9PYHzO8FcBLAz/SfC7X1vFLKDMAXwGfQHvU6opfbIroiMckT0SWn/wTwSADvKKVOKc1hpdQ7+u91n299owEkA7gd2u+hZ3dWVinVBu3X5xwBnPOLcyLyoIhsspk3VETaRWSI/v4jveehVkR2iMjNXcT2rIgsspm3TESesnofIyLzRaRKRMpE5Pf6z5kSXVRM8kR0ySmlTkD7qc4ZInKbiERd4KM07wGwRSlVAK2H4N7OCuq3Ax4E0Ar7vwf+KYBRIjLGat4dAJYppQ7o71cCGAPtgSrPAfiriET1JnD9oUbLAcwCMBhaj0YGgF/3pj6irjDJE1FfmQJgGYBHoP0+d6WIPG2T7IeKSI31C9pV+HdExATt+ewf6bM+AJBtZ+Dbb/T1DwP4PoAblFLn3NtXSlVDe1TwnXr9Aq2H4EOrMh8opU4opdqVUjMBbNH3pzduA7BZKfWuUqpFKXUEwO/1+UQXFZM8EfUJpdRxpdST+qN+BwL4HwDPQE+uuv1KqYHWLwAP2FT1IwBmaI9wBbQr+SoAd9uUe1Gvw18plaCUmt1FeB8BuFnvMk/T45sFAPrT9Z4Tkd36IL4aALEABp3P/lsZCiDR5kTmQwCBvayPqFNM8kTU55RSDUqpv0K7Ih5znqvfA+3++jYROQrtSt0bnQ/A64mFAJoB5EPrqp+plGrUl90E7QTiBgDe+onHZnQ+YLAWgIfNvGCr6YMAFtmczAzQBwkSXVRM8kR0yYmItz64LEZEnEXESf+TthgARedRTxSAJAA/gHZy0PGaAO1KOKc38Sml2gH8DcAvAEyHVVc9AC8AbdB6CxxE5C5oV/KdWQ9grIiM0/fzIWhX7x3+BiBORO4SEZPeUxAhIlm9iZ2oK0zyRNQXWgD4Q+sCPwktYT4F4BdKqS/Oo557AWxQSs1WSh21em0B8AW6GIDXAx8BSIF2y2Ct1fyPAXwDoBTAEQBR6OLERCm1DMD/AZgH7U8GAwAUWy0/CiAVwPUADgCoBvAlgIgLiJ3ILj5PnoiIyKB4JU9ERGRQTPJEREQGxSRPRERkUEzyREREBsUkT0REZFBM8kRERAbFJE9ERGRQTPJEREQG9f/3/DU0nej/NAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "CARTPOLE_FEATURE_NAMES = ['Cart Position', 'Cart Velocity', 'Pole Angle', 'Pole Angular Velocity']\n",
    "import shap\n",
    "import matplotlib.pyplot as plt\n",
    "plt.close('all')\n",
    "explainable_values_action_1 = shap.Explanation(shap_values.squeeze()[: , 0], feature_names=CARTPOLE_FEATURE_NAMES)\n",
    "explainable_values_action_2 = shap.Explanation(shap_values.squeeze()[: , 1], feature_names=CARTPOLE_FEATURE_NAMES)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_title(\"SHAP values Action 1 - initial observation\")\n",
    "shap.plots.bar(explainable_values_action_1, ax=ax)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_title(\"SHAP values Action 2 - initial observation\")\n",
    "shap.plots.bar(explainable_values_action_2, ax=ax)\n",
    "\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}