notebooks/10_Object_Oriented_ML.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "10_Object_Oriented_ML-cn",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 面向对象的机器学习"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/logo.png\" width=150>\n",
        "\n",
        "本文中，我们将学习如何适当地创建和使用类与函数，基于 PyTorch 完成机器学习任务。在后续的 notebooks中，都会按照这样的实现结构。\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 概览"
      ]
    },
    {
      "metadata": {
        "id": "qWro5T5qTJJL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "我们将会用到的类和函数总计如下："
      ]
    },
    {
      "metadata": {
        "id": "f88fhoHFLIKg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*   **词汇表 Vocabulary**：用于原始输入和数字形式的相互转换。\n",
        "*   **向量器 Vectorizer**：输入和输出的 vocabulary 类实例，并为模型把数据向量化。\n",
        "*   **数据集 Dataset**：处理和拆分数据的 vectorizers。\n",
        "*   **模型 Model**：处理输入和返回预测值的 PyTorch模型。\n",
        "\n",
        "下面的以及后续课程中的代码实现结构，全部归功于 PyTorch 的[贡献者](https://github.com/joosthub/PyTorchNLPBook/graphs/contributors)。"
      ]
    },
    {
      "metadata": {
        "id": "I_1BrSavU0ek",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 配置"
      ]
    },
    {
      "metadata": {
        "id": "b6B0GSOuY_ty",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "首先，我们配置可重现的环境、参数、种子等。"
      ]
    },
    {
      "metadata": {
        "id": "WnV34uLSY4bZ",
        "colab_type": "code",
        "outputId": "923141ef-0d88-47ba-8b6a-ad14e50b789d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# 加载 PyTorch 库\n",
        "!pip3 install torch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "0-dXQiLlTIgz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "from argparse import Namespace\n",
        "import collections\n",
        "import json\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hnID97KCXuT-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# 设置 Numpy 和 PyTorch 种子\n",
        "def set_seeds(seed, cuda):\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    if cuda:\n",
        "        torch.cuda.manual_seed_all(seed)\n",
        "        \n",
        "# 创建目录\n",
        "def create_dirs(dirpath):\n",
        "    if not os.path.exists(dirpath):\n",
        "        os.makedirs(dirpath)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wmpwPJr5VtKP",
        "colab_type": "code",
        "outputId": "4ede8242-2dcd-4f30-aaa2-88090713b1a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# 参数\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    cuda=False,\n",
        "    shuffle=True,\n",
        "    data_file=\"names.csv\",\n",
        "    split_data_file=\"split_names.csv\",\n",
        "    vectorizer_file=\"vectorizer.json\",\n",
        "    model_state_file=\"model.pth\",\n",
        "    save_dir=\"names\",\n",
        "    train_size=0.7,\n",
        "    val_size=0.15,\n",
        "    test_size=0.15,\n",
        "    num_epochs=20,\n",
        "    early_stopping_criteria=5,\n",
        "    learning_rate=1e-3,\n",
        "    batch_size=64,\n",
        "    hidden_dim=300,\n",
        "    dropout_p=0.1,\n",
        ")\n",
        "\n",
        "# 设置种子\n",
        "set_seeds(seed=args.seed, cuda=args.cuda)\n",
        "\n",
        "# 创建保存路径\n",
        "create_dirs(args.save_dir)\n",
        "\n",
        "# 展开文件路径\n",
        "args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
        "args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
        "\n",
        "# 检查 CUDA\n",
        "if not torch.cuda.is_available():\n",
        "    args.cuda = False\n",
        "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
        "print(\"Using CUDA: {}\".format(args.cuda))\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using CUDA: False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1U_jyWA-ZIZF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 数据"
      ]
    },
    {
      "metadata": {
        "id": "R_Fg-TslU2Bs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "我们的任务是得到指定姓名对应的国籍，包括数据预处理，以及拆分数据为训练、验证和测试部分。"
      ]
    },
    {
      "metadata": {
        "id": "M7H6_m8XZkFX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "import urllib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Vq8Ooa6RZWhX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# 从 GitHub 上传数据到 notebook 的目录\n",
        "url = \"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/data/surnames.csv\"\n",
        "response = urllib.request.urlopen(url)\n",
        "html = response.read()\n",
        "with open(args.data_file, 'wb') as fp:\n",
        "    fp.write(html)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QkkdVRivZKd8",
        "colab_type": "code",
        "outputId": "79d7d550-f7e0-4eb7-9a62-ebac2c514fd1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# 原始数据\n",
        "df = pd.read_csv(args.data_file, header=0)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>surname</th>\n",
              "      <th>nationality</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Woodford</td>\n",
              "      <td>English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Coté</td>\n",
              "      <td>French</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Kore</td>\n",
              "      <td>English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Koury</td>\n",
              "      <td>Arabic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Lebzak</td>\n",
              "      <td>Russian</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    surname nationality\n",
              "0  Woodford     English\n",
              "1      Coté      French\n",
              "2      Kore     English\n",
              "3     Koury      Arabic\n",
              "4    Lebzak     Russian"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "diz8O5KKBbwj",
        "colab_type": "code",
        "outputId": "08fe865a-7220-4036-d07f-e43725637f04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        }
      },
      "cell_type": "code",
      "source": [
        "# 基于国籍拆分数据\n",
        "by_nationality = collections.defaultdict(list)\n",
        "for _, row in df.iterrows():\n",
        "    by_nationality[row.nationality].append(row.to_dict())\n",
        "for nationality in by_nationality:\n",
        "    print (\"{0}: {1}\".format(nationality, len(by_nationality[nationality])))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "English: 2972\n",
            "French: 229\n",
            "Arabic: 1603\n",
            "Russian: 2373\n",
            "Japanese: 775\n",
            "Chinese: 220\n",
            "Italian: 600\n",
            "Czech: 414\n",
            "Irish: 183\n",
            "German: 576\n",
            "Greek: 156\n",
            "Spanish: 258\n",
            "Polish: 120\n",
            "Dutch: 236\n",
            "Vietnamese: 58\n",
            "Korean: 77\n",
            "Portuguese: 55\n",
            "Scottish: 75\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "PYwA2DD7BoNv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# 拆分子数据集\n",
        "final_list = []\n",
        "for _, item_list in sorted(by_nationality.items()):\n",
        "    if args.shuffle:\n",
        "        np.random.shuffle(item_list)\n",
        "    n = len(item_list)\n",
        "    n_train = int(args.train_size*n)\n",
        "    n_val = int(args.val_size*n)\n",
        "    n_test = int(args.test_size*n)\n",
        "\n",
        "  # 给每个数据点添加子集属性\n",
        "    for item in item_list[:n_train]:\n",
        "        item['split'] = 'train'\n",
        "    for item in item_list[n_train:n_train+n_val]:\n",
        "        item['split'] = 'val'\n",
        "    for item in item_list[n_train+n_val:]:\n",
        "        item['split'] = 'test'  \n",
        "\n",
        "    # 添加到最终列表\n",
        "    final_list.extend(item_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oxYU4gnLCzcV",
        "colab_type": "code",
        "outputId": "56762991-d349-423a-8dfd-74e36f513508",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# 子集数据帧\n",
        "split_df = pd.DataFrame(final_list)\n",
        "split_df[\"split\"].value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "train    7680\n",
              "test     1660\n",
              "val      1640\n",
              "Name: split, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "metadata": {
        "id": "k_erGyQ24Tnb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# 预处理\n",
        "def preprocess_text(text):\n",
        "    text = ' '.join(word.lower() for word in text.split(\" \"))\n",
        "    text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
        "    text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
        "    return text\n",
        "    \n",
        "split_df.surname = split_df.surname.apply(preprocess_text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BnLGAU_7Dl7y",
        "colab_type": "code",
        "outputId": "d59eff3a-c6f9-4f47-fdc6-847f140b931b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# 保存到 CSV 文件\n",
        "split_df.to_csv(args.split_data_file, index=False)\n",
        "split_df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>nationality</th>\n",
              "      <th>split</th>\n",
              "      <th>surname</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Arabic</td>\n",
              "      <td>train</td>\n",
              "      <td>bishara</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Arabic</td>\n",
              "      <td>train</td>\n",
              "      <td>nahas</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Arabic</td>\n",
              "      <td>train</td>\n",
              "      <td>ghanem</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Arabic</td>\n",
              "      <td>train</td>\n",
              "      <td>tannous</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Arabic</td>\n",
              "      <td>train</td>\n",
              "      <td>mikhail</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  nationality  split  surname\n",
              "0      Arabic  train  bishara\n",
              "1      Arabic  train    nahas\n",
              "2      Arabic  train   ghanem\n",
              "3      Arabic  train  tannous\n",
              "4      Arabic  train  mikhail"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "-i28mtiYD4Xc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 词汇表 Vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "sj7DzE_YvjSr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "我们将为国籍（nationality） 和姓（surname）创建 Vocabulary 类。"
      ]
    },
    {
      "metadata": {
        "id": "QDFTRZ4nDzxp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Vocabulary(object):\n",
        "    def __init__(self, token_to_idx=None, add_unk=True, unk_token=\"<UNK>\"):\n",
        "\n",
        "        # 令牌（Token）到索引（index）\n",
        "        if token_to_idx is None:\n",
        "            token_to_idx = {}\n",
        "        self.token_to_idx = token_to_idx\n",
        "\n",
        "        # 索引（Index）到令牌（token）\n",
        "        self.idx_to_token = {idx: token \\\n",
        "                             for token, idx in self.token_to_idx.items()}\n",
        "        \n",
        "        # 添加未知 token\n",
        "        self.add_unk = add_unk\n",
        "        self.unk_token = unk_token\n",
        "        if self.add_unk:\n",
        "            self.unk_index = self.add_token(self.unk_token)\n",
        "\n",
        "    def to_serializable(self):\n",
        "        return {'token_to_idx': self.token_to_idx,\n",
        "                'add_unk': self.add_unk, 'unk_token': self.unk_token}\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        return cls(**contents)\n",
        "\n",
        "    def add_token(self, token):\n",
        "        if token in self.token_to_idx:\n",
        "            index = self.token_to_idx[token]\n",
        "        else:\n",
        "            index = len(self.token_to_idx)\n",
        "            self.token_to_idx[token] = index\n",
        "            self.idx_to_token[index] = token\n",
        "        return index\n",
        "\n",
        "    def add_tokens(self, tokens):\n",
        "        return [self.add_token[token] for token in tokens]\n",
        "\n",
        "    def lookup_token(self, token):\n",
        "        if self.add_unk:\n",
        "            index = self.token_to_idx.get(token, self.unk_index)\n",
        "        else:\n",
        "            index =  self.token_to_idx[token]\n",
        "        return index\n",
        "\n",
        "    def lookup_index(self, index):\n",
        "        if index not in self.idx_to_token:\n",
        "            raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
        "        return self.idx_to_token[index]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Vocabulary(size=%d)>\" % len(self)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.token_to_idx)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PJ56RLko0prE",
        "colab_type": "code",
        "outputId": "06161be2-181f-4436-b310-258bce6f349e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Vocabulary 实例\n",
        "nationality_vocab = Vocabulary(add_unk=False)\n",
        "for index, row in df.iterrows():\n",
        "    nationality_vocab.add_token(row.nationality)\n",
        "print (nationality_vocab) # __str__\n",
        "print (len(nationality_vocab)) # __len__\n",
        "index = nationality_vocab.lookup_token(\"English\")\n",
        "print (index)\n",
        "print (nationality_vocab.lookup_index(index))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Vocabulary(size=18)>\n",
            "18\n",
            "0\n",
            "English\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wt6sCsBu238H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 向量器 Vectorizer"
      ]
    },
    {
      "metadata": {
        "id": "lFSaEs4L2TkY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class SurnameVectorizer(object):\n",
        "    def __init__(self, surname_vocab, nationality_vocab):\n",
        "        self.surname_vocab = surname_vocab\n",
        "        self.nationality_vocab = nationality_vocab\n",
        "\n",
        "    def vectorize(self, surname):\n",
        "        one_hot = np.zeros(len(self.surname_vocab), dtype=np.float32)\n",
        "        for token in surname:\n",
        "            one_hot[self.surname_vocab.lookup_token(token)] = 1\n",
        "        return one_hot\n",
        "\n",
        "    def unvectorize(self, one_hot):\n",
        "        surname = [vectorizer.surname_vocab.lookup_index(index) \\\n",
        "            for index in np.where(one_hot==1)[0]]\n",
        "        return surname\n",
        "        \n",
        "    @classmethod\n",
        "    def from_dataframe(cls, df):\n",
        "        surname_vocab = Vocabulary(add_unk=True)\n",
        "        nationality_vocab = Vocabulary(add_unk=False)\n",
        "\n",
        "        # 创建 vocabularies\n",
        "        for index, row in df.iterrows():\n",
        "            for letter in row.surname: # char-level tokenization\n",
        "                surname_vocab.add_token(letter)\n",
        "            nationality_vocab.add_token(row.nationality)\n",
        "        return cls(surname_vocab, nationality_vocab)\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        surname_vocab = Vocabulary.from_serializable(contents['surname_vocab'])\n",
        "        nationality_vocab =  Vocabulary.from_serializable(contents['nationality_vocab'])\n",
        "        return cls(surname_vocab, nationality_vocab)\n",
        "\n",
        "    def to_serializable(self):\n",
        "        return {'surname_vocab': self.surname_vocab.to_serializable(),\n",
        "                'nationality_vocab': self.nationality_vocab.to_serializable()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IBf2a0nz4Ji_",
        "colab_type": "code",
        "outputId": "199a097e-bfa2-4205-e168-c643c6c1bc66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Vectorizer 实例\n",
        "vectorizer = SurnameVectorizer.from_dataframe(split_df)\n",
        "print (vectorizer.surname_vocab)\n",
        "print (vectorizer.nationality_vocab)\n",
        "one_hot = vectorizer.vectorize(preprocess_text(\"goku\"))\n",
        "print (one_hot)\n",
        "print (vectorizer.unvectorize(one_hot))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Vocabulary(size=28)>\n",
            "<Vocabulary(size=18)>\n",
            "[0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0.]\n",
            "['g', 'o', 'u', 'k']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "N0QYigLGZJ6G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**注意**：当我们向量化有独热编码形式的输入时， 我们丢失了所有名称相关的结构。这是用独热编码表示文本的主要劣势。后续我们会展示更多保留语义结构的编码方法。"
      ]
    },
    {
      "metadata": {
        "id": "vT7q4sh558yh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 数据集"
      ]
    },
    {
      "metadata": {
        "id": "4fYxY_Eq-Tso",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch.utils.data import Dataset, DataLoader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XFQf4ikx5pp1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class SurnameDataset(Dataset):\n",
        "    def __init__(self, df, vectorizer):\n",
        "        self.df = df\n",
        "        self.vectorizer = vectorizer\n",
        "\n",
        "        # Data splits\n",
        "        self.train_df = self.df[self.df.split=='train']\n",
        "        self.train_size = len(self.train_df)\n",
        "        self.val_df = self.df[self.df.split=='val']\n",
        "        self.val_size = len(self.val_df)\n",
        "        self.test_df = self.df[self.df.split=='test']\n",
        "        self.test_size = len(self.test_df)\n",
        "        self.lookup_dict = {'train': (self.train_df, self.train_size), \n",
        "                            'val': (self.val_df, self.val_size),\n",
        "                            'test': (self.test_df, self.test_size)}\n",
        "        self.set_split('train')\n",
        "\n",
        "        # Class weights (for imbalances)\n",
        "        class_counts = df.nationality.value_counts().to_dict()\n",
        "        def sort_key(item):\n",
        "            return self.vectorizer.nationality_vocab.lookup_token(item[0])\n",
        "        sorted_counts = sorted(class_counts.items(), key=sort_key)\n",
        "        frequencies = [count for _, count in sorted_counts]\n",
        "        self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32)\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_make_vectorizer(cls, split_data_file):\n",
        "        df = pd.read_csv(split_data_file, header=0)\n",
        "        train_df = df[df.split=='train']\n",
        "        return cls(df, SurnameVectorizer.from_dataframe(train_df))\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_load_vectorizer(cls, split_data_file, vectorizer_filepath):\n",
        "        df = pd.read_csv(split_data_file, header=0)\n",
        "        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\n",
        "        return cls(df, vectorizer)\n",
        "\n",
        "    def load_vectorizer_only(vectorizer_filepath):\n",
        "        with open(vectorizer_filepath) as fp:\n",
        "            return SurnameVectorizer.from_serializable(json.load(fp))\n",
        "\n",
        "    def save_vectorizer(self, vectorizer_filepath):\n",
        "        with open(vectorizer_filepath, \"w\") as fp:\n",
        "            json.dump(self.vectorizer.to_serializable(), fp)\n",
        "\n",
        "    def set_split(self, split=\"train\"):\n",
        "        self.target_split = split\n",
        "        self.target_df, self.target_size = self.lookup_dict[split]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Dataset(split={0}, size={1})\".format(\n",
        "            self.target_split, self.target_size)\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.target_size\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        row = self.target_df.iloc[index]\n",
        "        surname_vector = self.vectorizer.vectorize(row.surname)\n",
        "        nationality_index = self.vectorizer.nationality_vocab.lookup_token(row.nationality)\n",
        "        return {'surname': surname_vector, 'nationality': nationality_index}\n",
        "\n",
        "    def get_num_batches(self, batch_size):\n",
        "        return len(self) // batch_size\n",
        "\n",
        "    def generate_batches(self, batch_size, shuffle=True, drop_last=True, device=\"cpu\"):\n",
        "        dataloader = DataLoader(dataset=self, batch_size=batch_size, \n",
        "                                shuffle=shuffle, drop_last=drop_last)\n",
        "        for data_dict in dataloader:\n",
        "            out_data_dict = {}\n",
        "            for name, tensor in data_dict.items():\n",
        "                out_data_dict[name] = data_dict[name].to(device)\n",
        "            yield out_data_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QhAJn2H3-vvu",
        "colab_type": "code",
        "outputId": "a9aefac3-9827-4251-e099-ab22d12ecc62",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Dataset instance\n",
        "dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.split_data_file)\n",
        "print (dataset) # __str__\n",
        "print (dataset[5]) # __getitem__\n",
        "print (dataset.class_weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Dataset(split=train, size=7680)\n",
            "{'surname': array([0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,\n",
            "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32), 'nationality': 0}\n",
            "tensor([0.0006, 0.0045, 0.0024, 0.0042, 0.0003, 0.0044, 0.0017, 0.0064, 0.0055,\n",
            "        0.0017, 0.0013, 0.0130, 0.0083, 0.0182, 0.0004, 0.0133, 0.0039, 0.0172])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "q8CAcVWRCVtm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 模型"
      ]
    },
    {
      "metadata": {
        "id": "K4yDxHIe_hGv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_bbJqIPRCbuZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class SurnameModel(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, output_dim, dropout_p):\n",
        "        super(SurnameModel, self).__init__()\n",
        "        self.fc1 = nn.Linear(input_dim, hidden_dim)\n",
        "        self.dropout = nn.Dropout(dropout_p)\n",
        "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
        "\n",
        "    def forward(self, x_in, apply_softmax=False):\n",
        "        z = F.relu(self.fc1(x_in))\n",
        "        z = self.dropout(z)\n",
        "        y_pred = self.fc2(z)\n",
        "\n",
        "        if apply_softmax:\n",
        "            y_pred = F.softmax(y_pred, dim=1)\n",
        "        return y_pred"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "p0Hr9OohDmPI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 训练"
      ]
    },
    {
      "metadata": {
        "id": "UWYC2MfiKh8o",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.optim as optim"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TKlstCszC_PT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Trainer(object):\n",
        "    def __init__(self, dataset, model, model_state_file, save_dir, device, shuffle, \n",
        "               num_epochs, batch_size, learning_rate, early_stopping_criteria):\n",
        "        self.dataset = dataset\n",
        "        self.class_weights = dataset.class_weights.to(device)\n",
        "        self.model = model.to(device)\n",
        "        self.save_dir = save_dir\n",
        "        self.device = device\n",
        "        self.shuffle = shuffle\n",
        "        self.num_epochs = num_epochs\n",
        "        self.batch_size = batch_size\n",
        "        self.loss_func = nn.CrossEntropyLoss(self.class_weights)\n",
        "        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)\n",
        "        self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
        "            optimizer=self.optimizer, mode='min', factor=0.5, patience=1)\n",
        "        self.train_state = {\n",
        "            'stop_early': False, \n",
        "            'early_stopping_step': 0,\n",
        "            'early_stopping_best_val': 1e8,\n",
        "            'early_stopping_criteria': early_stopping_criteria,\n",
        "            'learning_rate': learning_rate,\n",
        "            'epoch_index': 0,\n",
        "            'train_loss': [],\n",
        "            'train_acc': [],\n",
        "            'val_loss': [],\n",
        "            'val_acc': [],\n",
        "            'test_loss': -1,\n",
        "            'test_acc': -1,\n",
        "            'model_filename': model_state_file}\n",
        "    \n",
        "    def update_train_state(self):\n",
        "\n",
        "        # Verbose\n",
        "        print (\"[EPOCH]: {0:02d} | [LR]: {1} | [TRAIN LOSS]: {2:.2f} | [TRAIN ACC]: {3:.1f}% | [VAL LOSS]: {4:.2f} | [VAL ACC]: {5:.1f}%\".format(\n",
        "          self.train_state['epoch_index'], self.train_state['learning_rate'], \n",
        "            self.train_state['train_loss'][-1], self.train_state['train_acc'][-1], \n",
        "            self.train_state['val_loss'][-1], self.train_state['val_acc'][-1]))\n",
        "\n",
        "        # Save one model at least\n",
        "        if self.train_state['epoch_index'] == 0:\n",
        "            torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "            self.train_state['stop_early'] = False\n",
        "\n",
        "        # Save model if performance improved\n",
        "        elif self.train_state['epoch_index'] >= 1:\n",
        "            loss_tm1, loss_t = self.train_state['val_loss'][-2:]\n",
        "\n",
        "            # If loss worsened\n",
        "            if loss_t >= self.train_state['early_stopping_best_val']:\n",
        "                # Update step\n",
        "                self.train_state['early_stopping_step'] += 1\n",
        "\n",
        "            # Loss decreased\n",
        "            else:\n",
        "                # Save the best model\n",
        "                if loss_t < self.train_state['early_stopping_best_val']:\n",
        "                    torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "\n",
        "                # Reset early stopping step\n",
        "                self.train_state['early_stopping_step'] = 0\n",
        "\n",
        "            # Stop early ?\n",
        "            self.train_state['stop_early'] = self.train_state['early_stopping_step'] \\\n",
        "              >= self.train_state['early_stopping_criteria']\n",
        "        return self.train_state\n",
        "  \n",
        "    def compute_accuracy(self, y_pred, y_target):\n",
        "        _, y_pred_indices = y_pred.max(dim=1)\n",
        "        n_correct = torch.eq(y_pred_indices, y_target).sum().item()\n",
        "        return n_correct / len(y_pred_indices) * 100\n",
        "  \n",
        "    def run_train_loop(self):\n",
        "        for epoch_index in range(self.num_epochs):\n",
        "            self.train_state['epoch_index'] = epoch_index\n",
        "      \n",
        "            # Iterate over train dataset\n",
        "\n",
        "            # initialize batch generator, set loss and acc to 0, set train mode on\n",
        "            self.dataset.set_split('train')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, shuffle=self.shuffle, \n",
        "                device=self.device)\n",
        "            running_loss = 0.0\n",
        "            running_acc = 0.0\n",
        "            self.model.train()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "                # zero the gradients\n",
        "                self.optimizer.zero_grad()\n",
        "\n",
        "                # compute the output\n",
        "                y_pred = self.model(batch_dict['surname'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['nationality'])\n",
        "                loss_t = loss.item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute gradients using loss\n",
        "                loss.backward()\n",
        "\n",
        "                # use optimizer to take a gradient step\n",
        "                self.optimizer.step()\n",
        "                \n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['nationality'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['train_loss'].append(running_loss)\n",
        "            self.train_state['train_acc'].append(running_acc)\n",
        "\n",
        "            # Iterate over val dataset\n",
        "\n",
        "            # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "            self.dataset.set_split('val')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, shuffle=self.shuffle, device=self.device)\n",
        "            running_loss = 0.\n",
        "            running_acc = 0.\n",
        "            self.model.eval()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "\n",
        "                # compute the output\n",
        "                y_pred =  self.model(batch_dict['surname'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['nationality'])\n",
        "                loss_t = loss.to(\"cpu\").item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['nationality'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['val_loss'].append(running_loss)\n",
        "            self.train_state['val_acc'].append(running_acc)\n",
        "\n",
        "            self.train_state = self.update_train_state()\n",
        "            self.scheduler.step(self.train_state['val_loss'][-1])\n",
        "            if self.train_state['stop_early']:\n",
        "                break\n",
        "          \n",
        "    def run_test_loop(self):\n",
        "        # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "        self.dataset.set_split('test')\n",
        "        batch_generator = self.dataset.generate_batches(\n",
        "            batch_size=self.batch_size, shuffle=self.shuffle, device=self.device)\n",
        "        running_loss = 0.0\n",
        "        running_acc = 0.0\n",
        "        self.model.eval()\n",
        "\n",
        "        for batch_index, batch_dict in enumerate(batch_generator):\n",
        "            # compute the output\n",
        "            y_pred =  self.model(batch_dict['surname'])\n",
        "\n",
        "            # compute the loss\n",
        "            loss = self.loss_func(y_pred, batch_dict['nationality'])\n",
        "            loss_t = loss.item()\n",
        "            running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "            # compute the accuracy\n",
        "            acc_t = self.compute_accuracy(y_pred, batch_dict['nationality'])\n",
        "            running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "        self.train_state['test_loss'] = running_loss\n",
        "        self.train_state['test_acc'] = running_acc\n",
        "    \n",
        "    def plot_performance(self):\n",
        "        # Figure size\n",
        "        plt.figure(figsize=(15,5))\n",
        "\n",
        "        # Plot Loss\n",
        "        plt.subplot(1, 2, 1)\n",
        "        plt.title(\"Loss\")\n",
        "        plt.plot(trainer.train_state[\"train_loss\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_loss\"], label=\"val\")\n",
        "        plt.legend(loc='upper right')\n",
        "\n",
        "        # Plot Accuracy\n",
        "        plt.subplot(1, 2, 2)\n",
        "        plt.title(\"Accuracy\")\n",
        "        plt.plot(trainer.train_state[\"train_acc\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_acc\"], label=\"val\")\n",
        "        plt.legend(loc='lower right')\n",
        "\n",
        "        # Save figure\n",
        "        plt.savefig(os.path.join(self.save_dir, \"performance.png\"))\n",
        "\n",
        "        # Show plots\n",
        "        plt.show()\n",
        "    \n",
        "    def save_train_state(self):\n",
        "        with open(os.path.join(self.save_dir, \"train_state.json\"), \"w\") as fp:\n",
        "            json.dump(self.train_state, fp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "O1_A24sGHslh",
        "colab_type": "code",
        "outputId": "10f14aa7-0092-4096-b37d-adcee95a26d1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialization\n",
        "dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.split_data_file)\n",
        "dataset.save_vectorizer(args.vectorizer_file)\n",
        "vectorizer = dataset.vectorizer\n",
        "model = SurnameModel(input_dim=len(vectorizer.surname_vocab), \n",
        "                     hidden_dim=args.hidden_dim, \n",
        "                     output_dim=len(vectorizer.nationality_vocab),\n",
        "                     dropout_p=args.dropout_p)\n",
        "print (model.named_modules)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Creating from scratch!\n",
            "<bound method Module.named_modules of SurnameModel(\n",
            "  (fc1): Linear(in_features=28, out_features=300, bias=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc2): Linear(in_features=300, out_features=18, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-5RrfYBpJFkg",
        "colab_type": "code",
        "outputId": "54348c27-4452-42aa-befc-05ae3de66c46",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "trainer = Trainer(dataset=dataset, model=model, \n",
        "                  model_state_file=args.model_state_file, \n",
        "                  save_dir=args.save_dir, device=args.device,\n",
        "                  shuffle=args.shuffle, num_epochs=args.num_epochs, \n",
        "                  batch_size=args.batch_size, learning_rate=args.learning_rate, \n",
        "                  early_stopping_criteria=args.early_stopping_criteria)\n",
        "trainer.run_train_loop()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[EPOCH]: 00 | [LR]: 0.001 | [TRAIN LOSS]: 2.73 | [TRAIN ACC]: 30.5% | [VAL LOSS]: 2.54 | [VAL ACC]: 36.7%\n",
            "[EPOCH]: 01 | [LR]: 0.001 | [TRAIN LOSS]: 2.32 | [TRAIN ACC]: 38.0% | [VAL LOSS]: 2.26 | [VAL ACC]: 40.4%\n",
            "[EPOCH]: 02 | [LR]: 0.001 | [TRAIN LOSS]: 2.10 | [TRAIN ACC]: 38.7% | [VAL LOSS]: 2.14 | [VAL ACC]: 36.9%\n",
            "[EPOCH]: 03 | [LR]: 0.001 | [TRAIN LOSS]: 2.00 | [TRAIN ACC]: 39.0% | [VAL LOSS]: 2.07 | [VAL ACC]: 39.1%\n",
            "[EPOCH]: 04 | [LR]: 0.001 | [TRAIN LOSS]: 1.92 | [TRAIN ACC]: 39.3% | [VAL LOSS]: 2.02 | [VAL ACC]: 41.9%\n",
            "[EPOCH]: 05 | [LR]: 0.001 | [TRAIN LOSS]: 1.88 | [TRAIN ACC]: 40.4% | [VAL LOSS]: 1.99 | [VAL ACC]: 37.4%\n",
            "[EPOCH]: 06 | [LR]: 0.001 | [TRAIN LOSS]: 1.83 | [TRAIN ACC]: 39.4% | [VAL LOSS]: 1.98 | [VAL ACC]: 40.7%\n",
            "[EPOCH]: 07 | [LR]: 0.001 | [TRAIN LOSS]: 1.80 | [TRAIN ACC]: 40.7% | [VAL LOSS]: 1.96 | [VAL ACC]: 42.1%\n",
            "[EPOCH]: 08 | [LR]: 0.001 | [TRAIN LOSS]: 1.77 | [TRAIN ACC]: 40.5% | [VAL LOSS]: 1.95 | [VAL ACC]: 43.2%\n",
            "[EPOCH]: 09 | [LR]: 0.001 | [TRAIN LOSS]: 1.74 | [TRAIN ACC]: 41.9% | [VAL LOSS]: 1.94 | [VAL ACC]: 38.3%\n",
            "[EPOCH]: 10 | [LR]: 0.001 | [TRAIN LOSS]: 1.70 | [TRAIN ACC]: 42.0% | [VAL LOSS]: 1.90 | [VAL ACC]: 39.6%\n",
            "[EPOCH]: 11 | [LR]: 0.001 | [TRAIN LOSS]: 1.69 | [TRAIN ACC]: 42.7% | [VAL LOSS]: 1.90 | [VAL ACC]: 38.1%\n",
            "[EPOCH]: 12 | [LR]: 0.001 | [TRAIN LOSS]: 1.66 | [TRAIN ACC]: 42.7% | [VAL LOSS]: 1.90 | [VAL ACC]: 40.4%\n",
            "[EPOCH]: 13 | [LR]: 0.001 | [TRAIN LOSS]: 1.64 | [TRAIN ACC]: 43.5% | [VAL LOSS]: 1.88 | [VAL ACC]: 39.6%\n",
            "[EPOCH]: 14 | [LR]: 0.001 | [TRAIN LOSS]: 1.61 | [TRAIN ACC]: 43.8% | [VAL LOSS]: 1.87 | [VAL ACC]: 39.7%\n",
            "[EPOCH]: 15 | [LR]: 0.001 | [TRAIN LOSS]: 1.60 | [TRAIN ACC]: 44.3% | [VAL LOSS]: 1.87 | [VAL ACC]: 41.4%\n",
            "[EPOCH]: 16 | [LR]: 0.001 | [TRAIN LOSS]: 1.57 | [TRAIN ACC]: 45.0% | [VAL LOSS]: 1.86 | [VAL ACC]: 42.3%\n",
            "[EPOCH]: 17 | [LR]: 0.001 | [TRAIN LOSS]: 1.56 | [TRAIN ACC]: 44.7% | [VAL LOSS]: 1.87 | [VAL ACC]: 40.1%\n",
            "[EPOCH]: 18 | [LR]: 0.001 | [TRAIN LOSS]: 1.52 | [TRAIN ACC]: 45.7% | [VAL LOSS]: 1.85 | [VAL ACC]: 42.2%\n",
            "[EPOCH]: 19 | [LR]: 0.001 | [TRAIN LOSS]: 1.51 | [TRAIN ACC]: 46.0% | [VAL LOSS]: 1.84 | [VAL ACC]: 41.8%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "fgfpWoNRN_wu",
        "colab_type": "code",
        "outputId": "91cf38b2-c040-4d9f-b596-bee9d1d41bc8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "# Plot performance\n",
        "trainer.plot_performance()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAE+CAYAAAD4XjP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XdgVGXa///3lEx6JhPSeyOhhAAK\nCEpHOqhgwYaurKv7c9Vdd59d3X3W9bvdsvq44trXsnYXIShgQwlNQOkpJCG9J5NkMpmUSTLl90cg\nirRAyuQk1+sfk3PmnPnMkJi5zrnv61Y5nU4nQgghhBBCCCFcTu3qAEIIIYQQQgghukiBJoQQQggh\nhBCDhBRoQgghhBBCCDFISIEmhBBCCCGEEIOEFGhCCCGEEEIIMUhIgSaEEEIIIYQQg4QUaEJcpOTk\nZKqrq10dQwghhBgQN954I1dddZWrYwgx5EmBJoQQQgghzikvLw9fX1/Cw8M5dOiQq+MIMaRJgSZE\nH2tvb+cPf/gDCxcuZPHixTz66KPY7XYA3nrrLRYvXsyiRYu47rrrOH78+Dm3CyGEEIPBhg0bWLRo\nEcuWLSMtLa17e1paGgsXLmThwoX8+te/pqOj46zb9+3bx/z587uP/f73a9eu5fe//z3XXXcdr7/+\nOg6Hgz/+8Y8sXLiQuXPn8utf/5rOzk4AGhoa+OlPf8q8efNYvnw5u3btIj09nWXLlp2SeeXKlWzd\nurW/3xoh+pzW1QGEGGreeOMNqqur2bx5MzabjVtvvZVNmzYxb948/vnPf7Jt2zZ8fHz45JNPSE9P\nJyws7IzbR44c6eqXIoQQQmC32/niiy/42c9+hkaj4cknn6Sjo4Pa2loee+wx0tLSCA4O5r777uM/\n//kPixYtOuP2cePGnfN5tm/fzsaNGwkICOCzzz5j//79bNq0CYfDwYoVK9iyZQtXX301Tz75JAkJ\nCbzwwgtkZ2dzxx13sHPnToxGIzk5OYwaNYrKykpKS0uZOXPmAL1LQvQdKdCE6GPp6emsWbMGrVaL\nVqtl+fLl7N69myVLlqBSqVi3bh3Lli1j8eLFAHR2dp5xuxBCCDEY7Nq1i3HjxuHj4wPAlClT2LZt\nG42NjUycOJGQkBAAnnzySTQaDR9++OEZtx84cOCczzN+/HgCAgIAWLhwIXPmzMHNzQ2AcePGUVZW\nBnQVci+//DIAY8aM4csvv0Sn07Fw4UI2b97MqFGj2Lp1K/PmzUOn0/X9GyJEP5MhjkL0sYaGBvR6\nfff3er2e+vp63NzceP311zl48CALFy7k5ptvJjc396zbhRBCiMFg/fr1pKenM2nSJCZNmsTnn3/O\nhg0bMJlM+Pn5dT/O3d0drVZ71u3n8/2/nQ0NDTz44IMsXLiQRYsW8eWXX+J0OgFobGzE19e3+7En\nC8elS5eyefNmALZu3cqSJUt698KFcBEp0IToY4GBgTQ2NnZ/39jYSGBgINB1pe+ZZ55hz549TJ8+\nnUceeeSc24UQQghXMpvNfPPNN+zbt4/9+/ezf/9+vv32WzIyMlCr1ZhMpu7HNjc3U1dXh8FgOON2\njUbTPScboKmp6azP+3//939otVo+/vhjPv30U2bNmtW9z9/f/5Tzl5eX09nZyeTJk7HZbGzbto3j\nx49z+eWX99XbIMSAkgJNiD42e/Zs1q1bh91up7W1lY0bNzJr1ixyc3O5//776ejoQKfTkZKSgkql\nOut2IYQQwtU2b97M1KlTTxkqqNVqmT59Oh0dHRw8eJDy8nKcTiePPPII69atY9asWWfcHhQUhNFo\npL6+Hrvdzscff3zW562vrycpKQmdTkdOTg6HDh2itbUVgLlz57JhwwYA8vPzWblyJXa7HbVazZIl\nS/jzn//M3Llzu4dHCqE0MgdNiF5YvXo1Go2m+/u//OUvrF69mrKyMpYuXYpKpWLRokXd88oiIyNZ\ntmwZbm5ueHt784c//IGkpKQzbhdCCCFcLS0tjdtvv/207fPnz+e5557jT3/6E7fffjsajYZx48Zx\nxx134O7uftbt1157Lddccw3h4eFcffXVHDt27IzPu2bNGh588EHWr1/PpEmTePDBB/nf//1fUlNT\n+fWvf82DDz7I3Llz8fb25h//+AceHh5A1zDH1157TYY3CkVTOU8O6BVCCCGEEELB6urqWLFiBenp\n6adcQBVCSWSIoxBCCCGEGBKeeeYZbrrpJinOhKJJgSaEEEIIIRStrq6OefPmUVdXx5o1a1wdR4he\nkSGOQgghhBBCCDFIyB00IYQQQgghhBgkpEATQgghhBBCiEFiwNvsG42WXp/DYPDCZGrtgzQDS3IP\nHCVmBmXmVmJmUGZuJWYOCvJ1dQRFkb+RysqtxMygzNxKzAzKzK3EzKC83Of6+6jIO2harTI780ju\ngaPEzKDM3ErMDMrMrcTMYuAp9edEibmVmBmUmVuJmUGZuZWYGZSb+0wUWaAJIYQQQgghxFAkBZoQ\nQgghhBBCDBJSoAkhhBBCCCHEICEFmhBCCCGEEEIMElKgCSGEEEIIIcQgIQWaEEIIIYQQQgwSUqAJ\nIYQQfcRqtXLllVeyfv16Ojs7+dWvfsV1113H7bffjtlsdnU8IYQQCiAFmhBCDCPp6V/26HH//OeT\nVFZW9HOaoef5559Hr9cD8MEHH2AwGFi3bh1Llixh//79Lk4nhBBCCaRAE0KIYaKqqpKtWz/r0WN/\n/vNfER4e0c+JhpaCggLy8/OZPXs2ANu2beOqq64CYNWqVcybN8+F6YQQQiiF1tUBLlRbu42PdxYy\nMSEAd7ehs2K4EEL0t6eeeoxjx7KYMWMyCxYspqqqkqeffo6///1PGI21tLW1sWbNXVxxxQzuvfcu\nfvnL37Bt25e0tDRTWlpCRUU599//K6ZNu8LVL2VQeuyxx3j44YdJS0sDoKKigh07dvDEE08QGBjI\nI488gr+/v4tTCiGEuBgOp5Py2mbyyhqJD9cTH+7Xb8+luALtWImJl9IyuHVBEnMviXR1HCGEUIyb\nblrN+vUfEBeXQGlpMc899womUwNTpkxl8eJlVFSU8/DDD3HFFTNOOa62toZ//OMZ9u79mo0bP5QC\n7QzS0tKYMGECUVFR3ducTidxcXHce++9PPfcc7z44os8+OCD5zyPweCFVtv7i49BQb69PocrKDG3\nEjODMnMrMTMoM7cSM0Pf566ub+HIcSNHjtdx5LiRppYOAC5PDeOy8f03ykRxBVrYCC8A8ivMUqAJ\nIRTrg6/y+Tan9oKO0WhU2O3Os+6fPCqYG+Ym9uhco0ePBcDX149jx7L46KP1qFRqmppOb2SRmjoB\ngODgYJqbmy8o83CRnp5OWVkZ6enpVFdXo9PpCAwMZPLkyQBMnz6dtWvXnvc8JlNrr7MEBfliNFp6\nfZ6BpsTcSswMysytxMygzNxKzAx9k7uptYOcEhPZxSayixuoM1u79xl83bk8JZQxsQYmjgzq9XOd\nq5hUXIEWEuCFj6cbhRVNro4ihBCK5ebmBsAXX3xKU1MT//rXKzQ1NXHnnatPe6xG890dHafz7AXi\ncPb00093f7127VoiIiKoq6tj586dXHvttWRlZREXF+fChEIIIX7I2mEjr8zMsZIGsotNlNV+dxHS\n013LxJGBjIkNYEysgdAAL1Qq1YDkUlyBplapSI4xcCCnlqbWDvy8dK6OJIQQF+yGuYk9vtt1Um+v\nDqrVaux2+ynbGhsbCQsLR61Ws337V3R2dl70+cWpVq9ezYMPPsi6devw8vLisccec3UkIYQY1mx2\nB0VVTWQXmzhW3EBBZRN2R9eFR61GzegYA2NiDYyOCSA21Be1emAKsh9SXIEGkBwTwIGcWgorm5iQ\nGOjqOEIIoQgxMXHk5uYQFhbe3axi9uy5PPTQL8nOzmTp0qsIDg7mtddednFSZbvvvvu6v37mmWdc\nmEQIIQSAydLOO1vzyCxqoL2j60KlCogN82V0TACjYw2MjNCjGyQNCBVaoBkAKKgwS4EmhBA9ZDAY\nWL9+8ynbwsLCeeON97q/X7BgMQB33PETAOLjv7vLFx+fyLPPvjQASYUQQoi+UVTVxNoPj9LY3EGI\nwZMxKQGMiTGQHG3Ax9PN1fHOSJkFWrQBFVBYKfPQhBBCCCGEEKfbm1XNa5/kYLM5uGFOIgunRA3Y\nPLLeUGSB5u3pRligN4VVTTgcTpeNDxVCCCGEEEIMLg6nkw07Ctm8pwRPdw0/W5FKaoJyRt0pskAD\niA/3o7Kuhcq6FiKDfVwdRwghhBBCCOFibe02Xv44m8P5dQT7e3L/damEB3q7OtYFUbs6wMVKOLF6\nd37l6Wv2CCGEEEIIIYYXY2Mbf3vrAIfz6xgdY+D3t09SXHEGSi7QIvQAsh6aEEIIIYQQw1xGQR1/\nfmM/FcYW5l0SyQM3jB+0TUDOR7FDHMNHeOOh01Agd9CEEEIIIYQYttIPV/D253kA3LYwmdkTI1yc\nqHcUewdNrVYRF+ZHVX0rLVZZWFUIIfrKddctp7W11dUxhBBCiHOyOxy8/Xke//k0Fy8PN/7nxgmK\nL85AwQUaQEJE1zy0Imm3L4QQQgghxLDR3NbJU+8f4cuD5UQEefPUL2aSHG1wdaw+odghjgAJ4V3z\n0Aoqm0iJH+HiNEIIMbitWXMLf/vbk4SGhlJdXcVvf/srgoKCaWtrw2q18sADv2bMmBRXxxRCCCHO\nqbKuhWc+PEqtqY0JiYH8ZPkYQkd4YzRaXB2tTyi6QIs/0clR5qEJIcT5zZw5h927d3DttTewc+d2\nZs6cQ0LCSGbOnM2BA9/y9ttv8Ne/PuHqmEIIIRSs1drJR7uLOVZiIjLIh5GRehIj9YQHeqPug0Wi\njxbU8eJHWbS121k6LYYVM+P75LyDiaILNF8vHcEGTwormnA4nUPuH0cIMXStz9/EodqMCzpGo1Zh\ndzjPun9i8DhWJi476/6ZM+fw7LNPc+21N7Br13buvfcB3nvvTd599006Ozvx8PC4oDxCCCHESQ6H\nk10ZVXy4vQBLaydqlYqy2mb2ZFUD4OmuJSHCj5ERehIj9MSH63HXaXp8fqfTyWfflPHfbfloNGru\nWj6GqWND++vluJSiCzToWg9tT1YNNQ2thI1Q3joHQggxUOLjE6ivN1JTU43FYmHnznQCA4N5+OE/\nk5OTzbPPPu3qiEIIIRQov9zM21vzKKm24O6m4dpZ8cyfFIXRbCW/vJH8cjP5FWYyCxvILGwAQK1S\nERXiQ2KEvusuW4SeAL8zXyjstDn4z6c57M6sRu+j4/5rU4kL8xvIlziglF+gRejZk1VDQUWTFGhC\nCMVYmbjsnHe7ziQoyLfX4+unTZvOSy89x4wZs2hsNJGQMBKA7du3YbPZenVuIYQQw4vJ0s669Hz2\nZNUAMG1sCNfNTsTg6w5ARKA3EYHezJrQ1VmxqaWD/IquYi2/3ExxdRMl1Ra+PFAOQICf+4mCzZ/E\nCD2Rwd40t3by7IYMCiqaiAvz5d6Vqd3nH6qUX6CdaBRSWGlmemqYi9MIIcTgNmvWHH760zW8/vq7\nWK1t/OUvj7Bt21auvfYGtm79nM2bP3J1RCGEEINcp83B59+WsunrEto77cSE+nLLlUkkRurPeZyf\nt45LkoK4JCnoxHnslFQ3c7ziu7ts3xyr5ZtjtQC46zRo1SparDamjgnhR4tHoXPr+bBIpVJ8gRYR\n5I1Oqya/QlrtCyHE+YwePZbt2/d1f//22+u6v54+fRYAS5deNeC5hBBCDH5Op5Mj+fW89+Vxahvb\n8PVy46YrRzJ9XBhq9YX3gnDTakg80USEy7rOX2tqI7/CzPETBVt9k5VrZ8WzZGoMqmHSb0LxBZpW\noyY2zI/j5Y20tdvwdFf8SxJCCCGEEGJQqapv4d2tx8ksakCtUjF/UhRXT4/Fy8Otz55DpVIREuBF\nSIAXV4zrGhnndDqHTWF20pCoZhLC/cgra6S42sLomKGxQJ0QQgghhBCu1mq18dHuIr48UI7d4WRM\nrIGbrkwiInBgej8Mt+IMhkiBFv+9eWhSoAkhhBBCCNE7DqeT3Ue72uY3tXYSqPfgxnkjmTgycFgW\nTQOpRwXa448/zoEDB7DZbNx9990sWLCge19VVRW//OUv6ezsZMyYMfzpT3/qt7BnkxBxYsFqmYcm\nhBBCCCFErxRUmHlnax5FVRZ0bmpWzIxn0ZQo3LRDv0HHYHDeAm3v3r0cP36c999/H5PJxIoVK04p\n0B599FHWrFnD/Pnz+eMf/0hlZSXh4eH9GvqH/H3cGeHnQUGleViOUxVCCCGEEKK3TJZ23tp6nK/2\nlwFw2ZgQrp+dcNb1yUT/OG+BNnnyZFJTUwHw8/Ojra0Nu92ORqPB4XBw4MABnnrqKQAeeeSR/k17\nDgkRfnxzrBaj2Uqwv6fLcgghhBBCCKEkNaZWPttXyq6Mamx2B9HBPtw8P4mkKH9XRxuWzlugaTQa\nvLy8AFi3bh0zZ85Eo+m6vdnQ0IC3tzd///vfycrKYtKkSfzqV7865/kMBi+0fXB7NCjI95TvU5OC\nuwq0pnbGjgzu9fn7yw9zK4UScysxMygztxIzgzJzKzGzEEKIwamk2sIn+0r4NqcWpxOC/T25YX4y\nE+IMF9U2X/SNHjcJ2bp1K+vWrePVV1/t3uZ0OqmpqeG2224jIiKCu+66i/T0dGbPnn3W85hMrb0K\nDF0fUIxGyynbQvy6VhQ/nFPL2OjBWe2fKbcSKDG3EjODMnMrMTMoM7dSMw8nVquVZcuWcc8997By\n5UoAdu7cyZ133klubq6L0wkhRNfn99zSRrbsLSGzqAGA6GAflkyLYVJyMCEhfor7WzPU9KhA27lz\nJy+88AKvvPIKvr7f/bE1GAyEh4cTHR0NwLRp0zh+/Pg5C7T+Eh3ii1ajoqDSPODPLYQQQgA8//zz\n6PX67u/b29t56aWXCAoKcmEqIYTo6sp45HgdW/aWUFDZ1VhvVLQ/S6bGMDYuQHo4DCLnLdAsFguP\nP/44r7/+Ov7+p96Z0mq1REVFUVxcTGxsLFlZWSxdurTfwp6Lm1ZNTIgvxdUWOjrt6Nyky4wQQoiB\nU1BQQH5+/ikXKV944QVuvvlmnnjiCdcFE0IMaza7g33ZNWzZW0JVfddItokjA1kyNYaECP15jhau\ncN4CbcuWLZhMJn7xi190b7vssstITk5m/vz5/O53v+Ohhx7C6XSSlJTE3Llz+zXwucSH6ymobKK4\n2iKTGoUQQgyoxx57jIcffpi0tDQAioqKyMnJ4ec//7kUaEKIAdfeYWfH0Uo++6aUhqZ2NGoVV6SE\nsmhqzIAtMi0uznkLtFWrVrFq1aqz7o+JieHdd9/t01AXKyHCjy/2Q2FlkxRoQgghBkxaWhoTJkwg\nKiqqe9vf//53fv/731/QefqrkZZSKDG3EjODMnMrMTMMfG5LawebdhXx8c5CLK0d6Nw0LJ8RzzUz\nEwgO8OrROeS9dq0eNwlRgoTwrtu0Mg9NCCHEQEpPT6esrIz09HSqq6vRarWo1Wr+53/+B4Da2lpu\nvfVW3nrrrXOep78aaSmBEnMrMTMoM7cSM8PA5m5osvL5t2VsP1xJe6cdbw8tV10Ry7xLI/H10oHd\n3qMs8l4PjHMVk0OqQAvwc0fvo6PwxMRHIYQQYiA8/fTT3V+vXbuWiIiI7i6OAHPnzj1vcSaEEBfK\n2NhGTqmJ7GIT+3NqsTuc+PvouGZGHDPHh+PpPqQ+6g8bQ+pfTaVSkRCu52CekYYmq6x6LoQQQggh\nhow6cxu5pY3klJjIKW2kvsnavS8kwIvFl0UzbWwoblq1C1OK3hpSBRp0zUM7mGekoLJJCjQhhBAD\n7r777jtt21dffeWCJEIIpWtospJTaiKnpJGcUhN15u8KMm8PLZckBTEq2p9R0QbCg7xRS6v8IUFx\nBZrT6aS0sQIPp+8Z12vonodWYWbyqOCBjieEEEIIIcRFMVnaTxRkJnJLG6ltbOve5+WuZeLIQEZF\nG0iO9icy2EcKsiFKcQVadkMezx35N3eMuYlJoRNP2x8T6otaJQtWCyGEEEKIwc3c0sGxkgZyShrJ\nLTVRY/quIPN01zIhMZBR0f4kRxuICvZBrZaCbDhQXIEW5BkAwP7aw2cs0NzdNEQF+1BS3UynzSFj\ncIUQQgghxKBiszvYvKeETV8XY3c4AfDQaUhNGMGoaAOjYvyJDvaVgmyYUlyBFuwVRJRfGMcajmO1\nteOhdT/tMQkRfpTUWCirbSY+3M8FKYUQQgghhDhdubGZf286RkmNBYOvO1deGsmoGAPRIT5o1HJj\nQYAifwomR07A5rBxrCHvjPtlPTQhhBBCCDGY2B0ONu8p5k+vf0tJjYXp48L4848vY/HUGOLC/KQ4\nE90U+ZMwJWICAEeMmWfcHx/RddesoEIKNCGEEEKIC1HVUkOnvdPVMYaUqvoW/v7WQT7cXoi3hxv3\nX5fKmqWj8fJQ3GA2MQAU+VMRZ4giwMNAZv0xbA4bWvWpLyPY3xMfTzdZsFoIIYQQ4gJsLd3OhvzN\nLI6dx7L4ha6Oo3gOh5PPvynlwx2FdNocTB0Tws3zk/DxdHN1NDGIKfIOmkqlYnzgWNpsVo6bCs+4\nPyHcjzqzFXNzuwsSCiGEEEIoy47yr9mQvxmAXFOBi9MoX62pld89v5v3vsrH3U3DPdekcNdVY6U4\nE+elyAINYHzQWAAO151tmGPXPDS5iyaEEEIIcW57qvbzfl4avm4+BHoEUGopp9Nhc3UsRXI4nXx1\nsJxHXv2WrMJ6Lk0K4i93XsYkWZ9X9JBiC7R4fSw+bt5kGLNwOB2n7U840b0xXxqFCCGEEEKc1YGa\nw7x97L94a724b+JPGBs4GpvDRpmlwtXRFKfebOXJ9w7z1ud5aDUqfnXLpdyzIgU/b52rowkFUWyB\nplFrSAkcjbnDQnFT2Wn748L8UAGFFXIHTQghhBDiTDLqsnk9+z3cNe78bMKPifAJI14fA0Chudi1\n4RTE6XSy80glD/97H8dKTKQmjOBPP76M2ZdEolLJWmbiwiiySchJE4JS2Fu1nyPGzO7/mZzk6a4l\nIsibouom7A6HtC4VQgghhPieYw15vJLxJlqVhnvGryHGLwqg+zNVkbnElfEUw2Rp541PczhaUI+n\nu4Y7loxi+rgwKczERVN0gTbKMBKdRscRYybXJCw57RchPlxPubGFCmML0SG+LkophBBCCDG45DcW\n8eLRN0Cl4u7UH5HgH9u9z+Duj7+7nkJzCU6nUwqNs3A6nezNruGdL/JosdoYG2vgR4tHM0Lv4epo\nQuEUfVvJTePG2IBkjG31VLXUnLb/5Dw0WQ9NCCGEEKJLfn0xzx95FbvTzk9SVjMqYOQp+1UqFXH6\nGJo6LNRbTS5KObiZWzr414ZMXv44G5vdyeqFyfxy1QQpzkSfUPQdNIDxQSkcMmZwxJhJuE/oKfsS\nTnRyLKhsYs4lrkgnhBBCCDF4VDRX8c/DL9Ju72BNyi2kBI4+4+MS9LEcqj1KobmYQM+AAU7ZPzKL\n6jmQa8Rmd+B0dnVbdDicOJzgdDhP+d7hdOL84ffdjwFjYxut7TaSovxZs3Q0wf6ern55YghRfIGW\nEjgKjUrDEWMmi+OuPGVf6AgvPN21FEirfSGEEEIMc9UttTxz6CVaOlu5bfQqLglOPetjv2sUUsKU\nUGVf5W5osvLul8c5kGu86HOoVKBWqVCrVahVKtx1Gm6cPpIrJ0WiliGgoo8pvkDz1HqSZEjgWEMe\n9W0NjPjeVR61SkV8uB9ZRQ00t3XKwoBCCCGEGJbq2up55tBLNHe2cOelNzFRP/Gcj4/0CcdN7abo\nTo42u4Mv9pfx0a5i2jvtJEbquX52AnpvXXexpeouuuguvn5YjKlUyDw8MaAUX6BB1zDHYw15HKnL\nYm7UjFP2JZwo0AorzaQmBLoooRBCCCGEa5isjTxz6CXMHU2sTFzGgsSZGI2Wcx6jUWuI8YukoLGY\nNpsVT62y5lbllpp48/M8Kuta8PF045b5SVw+LlTudglFUHSTkJNSA8eiQsURY+Zp+7rnocl6aEII\nIYQYZpo6LDxz+CXqrSaWxS1gXvTMHh8br4/FiZOSM6w3O1iZm9t5+eMsHnvnEFV1LcyeEM7f7prK\n9NQwKc6EYgyJO2h6d1/i9NEUNBZj6WjGV+fTvS8urKuTY2GldHIUQgghxPDR3NnC2kMvU9tax/zo\n2SyKnXdBx5+ch1ZgLj6t0+Ng43A42XaogvU7CmhrtxMT6sttC5O7PwcKoSRDokCDrmGOheYSMuqy\nuTx8Svd2H083QgO8KKxqwuF0ytUTIYQQQgx5bbY2/nX4FSpbqpkVeQVXJyy+4HlUcX7KWLC6oMLM\nm5/nUlrTjJe7ltULkpg1IQK1Wj7zuUqnvRNzh2XIdAAdaENiiCPA+MAUgDMPcwz3o63dTlVdy0DH\nEkIIIYQYUFZbO88deZVSSwXTwiZz3cjlF9XkwkfnTYhXEEXmUhxORz8k7Z3mtk5e/ySHv755gNKa\nZq5ICeVvd01lziWRUpy5UF1bA4/vX8sf9z5O4SAv7gerIVOgBXmNINw7lBxTPlab9ZR9318PTQgh\nhBBiqOqwd/JixhsUmkuYFDKBm0ddi1p18R/34vQxWO1Wqlpq+jBl7zicTnYcqeR3L+1lx5FKIoK8\neeiWS/jxsjH4eetcHW9YO24q4In9a6lsqcbhdPBRwSc4nU5XxwKgpqWW1s5WV8fokSEzxBG6hjl+\nUryV7Ia8U9b2iA//bh7azPHhroonhBBiiLNarSxbtox77rmHadOm8dvf/habzYZWq+WJJ54gKCjI\n1RHFEGZz2Hgl803yTPmMDxzLbaNX9ao4g655aHur9lNoLiHCJ6yPkl680hoLb36eS0FFE+46DTfM\nSeTKSZFoNUPmnoNi7arYy/t5aQDcmLySo3VZZNfnkmvKd/kcxvzGIp4++AIqlYpRhpFMDE4lNWgM\nPm7eLs11NkOyQDtizDylQIsI8sbdTSOdHIUQQvSr559/Hr2+a9TG008/zQ033MCSJUt4++23ee21\n1/jNb37j4oRiqLI77LyW9S5Z9TmMCUjmjpRb0Kg1vT5vgj4WgEJzMTMipvb6fBerpa2Td77I48uD\n5TidMHlUMDfOG4nB191lmURgUY0VAAAgAElEQVQXu8POh/mb2F6+G283L+5MWU2SIYEYv0iy63P5\nqPBTkg2JLltLzul0sj5/E06chHuHkt2QS3ZDLu/mqknyT+CS4FRSg8ae0mTQ1YZUgRbpE8YIDwOZ\ndTnYHDa06q6Xp1GriQvzJbe0kVarDS+PIfWyhRBCDAIFBQXk5+cze/ZsAB555BHc3bs+PBoMBrKy\nslyYTgx1645/zGFjBiP94/nJuNW4qfvms06wVxBeWs8Bm0vkcDppbu2ksbmdxuZ2TJZ2Gpra2ZVR\nhcnSTojBk1sXJDM2TppPDAYtna28mvk2OabjhHmH8NPUHxHoOQKAaN9IJgSN47Axg4y6bFKDxrok\n48HaI5Q0lTExOJU7U27F2FrPYWMGB2uPkmM6To7pOO/lbWCkfzwTg8cxPigFP52vS7KeNKQqFZVK\nxfigFL4q20muqYCxI5K79yVE6MkpbaSouomxsfJLLYQQom899thjPPzww6SldQ3x8fLyAsBut/PO\nO+/ws5/9zJXxxBC2s2IPOyq+Jtw7lJ+m/gidpu/mYalVauL0MWTV59DUYenVB9e2dlt30dVVgHV8\n7+t2Gi1d2+yO0+cs6bRqVsyIY9FlMbhpZTjjYFDdUssLR1/D2FbPuMDR3D7mptMWNF8Wv4Ajxkw2\nFX1OSuDoXg+5vVCdDhsbCz5Fo9JwdfxioKtvxfyY2cyPmU19WwOHjBkcqs0g15RPrimf93PTSPSP\nY2JwKhOCUtC7D/xSDT0q0B5//HEOHDiAzWbj7rvvZsGCBac95sknn+Tw4cO8+eabfR7yQpws0I4Y\nM08p0E7OQyuoMEuBJoQQok+lpaUxYcIEoqKiTtlut9v5zW9+w9SpU5k2bdp5z2MweKHV9n5YWlCQ\na6/+XqyByN3Q2oiHmztebp59cj5Xv9eZNbn8N28jvu4+/G7OvQR7j+jRcReSe1x4Eln1OdQ7a0kI\n6tlcfofDyftf5JJRUE9DUxsNTVba2u1nfbxGrcLg50FilD8Bfh6M0Ht0/3eEnyfRYb4YfD3Oevxg\n5uqfkYtxvsyHqjJ5+uC/aeu0cs3ohdyYchVq9enFV1CQLzOqp7CjZB/51jyuiJ7cX5G7n+/7NuV+\nSb21gSVJcxkTE3v64/FlVHQMN7GMupYG9pYfYl/ZQXLrCzneWMh/8zYyKiiBqZGXcFnkRAK8/Ps1\n/0nnLdD27t3L8ePHef/99zGZTKxYseK0Ai0/P59vv/0WNze3fgvaU/H6GHzcvDlal8WNzhXdlXpC\neNecgELp5CiEEKKPpaenU1ZWRnp6OtXV1eh0OkJDQ0lLSyMmJoZ77723R+cxmXrfYSwoyBej0dLr\n8wy0gcjd3NHCI3seJdmQyF2pt/f6fK5+r42t9Ty5/yVAxZ1jV6Nq1WFsPX+eC80dou1qDnK4LIc4\n94QeHbNxVxEbdxUB4OvlRpDeE39fd/x93PH30eHv647Bp+t7g687Pl5u51yr1uDrIT/XA+RcmZ1O\nJ1+V7WRD/mY0ag23j7mRKaGXUF9/9qWs5oXPZlfpt7x75CMS3Ef2ydzInuRu7WxlXeZmPLUezAqZ\n0YN/BzcuC5jCZQFTaGw3c6i2685ajrGAY8Z8Xjv0AfH6WCYGj+Oy0EvxdvPqdd6zOW+BNnnyZFJT\nuxpu+Pn50dbWht1uR6P57s199NFHeeCBB3j22Wd7FbQvqFVqUgPH8HXVtxSZS0nwjwXAz1tHkL8H\nBRVmnE6nyyYqCiGEGHqefvrp7q/Xrl1LREQEdXV1uLm5cf/997swmfi+PVXfYrW3c6wh75S56krU\nZrPywtHXaLG1csuo60j0j+u354rxi0KtUlNoLu7R4w/mGdm4q4hAvQe/v30Sfl7S+n4o6HTYeC9n\nPXur96PX+XJX6u3E+kWf97hAzxFcHjaZXZX72Fd9kMvD+/cu2kmflnxFq62NaxKWXHC3Rn93PXOi\npjMnajqN7WaOGLM4VHuU/MYiCs3FFDQW8ZNxt/VT8h4UaBqNpnsc/bp165g5c+Ypxdn69euZMmUK\nERER/RbyQo0PSuHrqm85YszsLtCg6y7a3uwaak1thAT0ruoVQgghzuWdd96hvb2d1atXA5CQkMD/\n+3//z7WhhjGH08HOir0AdDg6KW4q69eipj85nA5ez3qH6tZa5kRO5/LwKf36fO4aHZE+YZQ2ldPp\nsJ2zAUmFsZmXN2Wjc1Nz78pxUpwNEU0dFl7O+A+F5hKifSO5O/V2/N31PT5+Uew89lYfYEvRF0wO\nndhnTWzOpr6tge1luwnwMDA78opencvfXc+syMuZFXk55nYLWfU5RPr275ITPX53tm7dyrp163j1\n1Ve7tzU2NrJ+/Xpee+01amp6toDhQIyvvyJgIq9lv0NmQzZ3Bd7YfbcsNSm4q0CztJOSHNLrDBdD\nieOQQZm5lZgZlJlbiZlBmbmVmHm4ue+++wBYuXKli5OI78uuz6Xe2kCAh4EGq4k8U75iC7SNBZ+Q\nWZ/D6IAkViQuHZDnjNPHUmqpoMxSQbw+5oyPabF2snZ9Bu0ddn569ViiQ+T/V0NBmaWCF4++gam9\nkUuDx3Pr6BvQaS5sWpPBw5+ZEdP4qmwnuyv39bpoOp+PCj/F5rSzPH4hbheY9Vz07r4DcgewRwXa\nzp07eeGFF3jllVfw9f3ul23v3r00NDRwyy230NHRQWlpKX/729/43e9+d9ZzDdT4+jEByRysPcqR\n4uPdCyuG6LvaHR/OqWVcjKHXOS6UEschgzJzKzEzKDO3EjODMnMrNbMQg8HOij0A3DLqOp49/Ap5\npgKWxM13caoLt6/qAFtLtxPsFciasX2z1llPxOtj2F6+m0Jz8RkLNIfDyQsbs6g1tbF0WgxTRrvm\nQrjoWwdrj/Jm9vt0ODpZHr+IhTFzLnqa0IKYOeyq3MdnxV9xedjkPu02+n0lTWXsrzlMlG8Ek0Im\n9Mtz9Lfz9rq0WCw8/vjjvPjii/j7n9q5ZNGiRWzZsoUPPviAZ599lrFjx56zOBtI44NSADhszOze\nFhXsg5tWTUGl2VWxhBBCCDHA6tsayKrPJdYvmlEBI4n0CaPIXEKHvdPV0S5IobmEd3LW4an15Kep\nd/RZJ8qeOLlgddFZ1kNbt72ArKIGUhNGsGJG/IDlEv3D4XSwuegL/p35FqhU3DXudhbFzu1VDwdf\nnQ9zI6fT1GFhe/nXfZj2O06nkw35mwFYmbh0wNv695Xz3kHbsmULJpOJX/ziF93bLrvsMpKTk5k/\nf/BeeRo7YhRalYYjxkyWnrhCptWoiQn1pbCiifYOO+66gbnqJIQQQgjX2VW5DydOZkZ0LXWQZEik\nrLmSQnMxowJGujhdz5isjbyU8QYOnPw45RZCvIIG9PkNHv74u+spMBef1mxtb1Y1n+4rJSTAi7uW\nj0WtlkZsSma1tfNq5tscMmYwwsPA3ak/6h6N1lvzomexvWIPX5SkMz3iMjy1fXuRIbP+GMcbC0kZ\nMYokQ2KfnnsgnbdAW7VqFatWrTrviSIjI12+Btr3eWo9SApIJLs+l7q2BgI9u9Y+Swj3I7/cTHF1\nE8nRAz/MUQghhBADp9Nh4+vKb/DWenFJcFdX6iRDAl+W7eC4qUARBVq7vYMXj76OpaOZ60ZexeiA\nJJfkiNfHcLD2KPXWBgI9u9ZbK6m28NonOXi6a7j/2nF4eSi3M6bouhDwxJf/obixnET/OO5MWY2v\nzqfPzu/l5smV0bP4uPBTvirdydL409dWvlh2h50N+VtQoeKaAZqb2V+Ued+vhyYEdg1zPPK9YY4n\n10MrkPXQhBBCiCHvUO1RmjtbmBY+ubtZQIJ/HGqVmlxTgYvTnZ/D6eA/2e9T1lzJFeFT+r25wrnE\nnxjmWHhimGNTSwdr1x/FZnPwk+VjCRtxYa3MxeDSYe9k7eGXKW4s54rwKdw34Sd9WpydNDvyCnzd\nfPiqbCfNnWdfP+1CfVX4NTWttVwePpkwb2XPgRzSBdq4oDGoUJ1aoEWcKNAqZB6aEEIIMdTtrNiD\nChXTw6d2b/PUehDjG0mJpQyrzerCdOf3SdFWDhszSPSP44aka1y6juvJ5iCF5hJsdgfPpWXS0NTO\nNTPjmZAY6LJcom98XPgpNa1GFiXO5qbka/ttnUAPrTsLYudgtbfzRUl6n5zTarPyQdYmdBodS+P6\n7q6cqwzpAs1P50u8PoZCcwlNHV2dzwy+XSvWF1Y24XQ6XZxQCCGEEP2l3FJJobmE0QFJBHmNOGXf\nSEMCDqeDgh4uvuwKB2uPsqV4KyM8DPwk5TaXL6wd6ROOm9qNQnMx7355nLyyRiYlB7Fs2pnb7gvl\nyG8sYlvZLoK9Arll/Ip+vxAwI3wq/u56tpd/jbm996PatpbuwGxt4sroWejd/fogoWsN6QINuro5\nOnGSYczu3pYQ7oe5pYN68+C+aiaEEEKIi3eytf7MyGmn7Us+0UAg15Q/oJl6qtRSzn+y38ddo+Pu\n1B/ho3P98EGNWkOsXxSVzdVsO1xCZJA3a5aOduldPdF77fYO3jz2AQCrR6/CXdv/i4u7adxYHDuP\nTkcnnxZ/1atzNbab+bJ0O/4efsyLmtlHCV1rWBRoAEfqsrq3xcs8NCGEEGJIa7O18U3NIQzu/owd\nMeq0/fH6GDQqDccH4Tw0c7uFF4++gc1h40djbuqzDnp9wV8dihMnXgEW7rs2FQ+dNAVRuo0FW6hr\nq2de9MyzLkLeH6aFTSbQcwS7K/dR39Zw0efZXPgFHY5ObkhZjofWvQ8Tus6QL9ACPQOI8Akjt+E4\nbSfGmSeenIcm66EJIYQQQ9K+6oN02DuYETH1jGsh6TQ64vTRlFkqae1sdUHCM+u0d/Jyxhs0tpu5\nKn4RqUFjXR2pm8nSzqFDNgAmXepGkP/ArcMm+kduQz7by78m1CuYZQM8d0uj1rA0bj52p50txVsv\n6hyVzdXsqfqWUO8Q5sSdfqdcqYZ8gQZdd9FsTjvZ9TkARIf4oFGrKKiQO2hCCCHEUON0OtlZvgeN\nSsPl4VPO+rgk/wScODneWDSA6c7O6XTyTu6HFDWVMjlkIvNjZrs6UrdOm51n1x/FUt/V1a9ZVevi\nRKK3rDYrb+X8F7VKzW1jVnV3OR1Ik0ImEOodwr6qA9S0XPjPVFrBFpw4WZGwBI166KxvPCwKtAkn\nhzkau4Y56tw0RIf4UFpjodNmd2U0IYQQQvSx/MZCqltrmRg87pxtwk8uZJs3SOahbS3dzjfVB4nx\ni+KWUdcNmrldTqeT/3yaS1GVhWnJ0YR4BVFkLsXhdLg6muiFDfmbabCaWBA9mxi/KJdkUKvULI9b\ngBMnm4u+uKBjcxvyyarPIck/4YzDmJVsWBRo4d6hBHoEkFWfQ6ej69Z8Qrgeu8NJSU2zi9MJIYQQ\noi/tONEcZEbEuYc8xeqjcVO7kTcI5qFl1GWzseAT/N313D3udpfczTibrfvL2Z1ZTWyoL7cvSiZe\nH4vVbqWqpcbV0cRFOlafx67KfYR7h7Io7kqXZhkflEK0bwQHao9Qbqns0TEOp4MN+ZsAWJG4dNBc\nzOgrw6JAU6lUjA9KwWpvJ7fhOADxEV0tOGU9NCGEEGLoMLc3cdiYSbh3KAknFlY+Gze1lgR9LJUt\n1Vg6XHfBtrK5mtey3kGr1nL3uNsHVZvw7OIG3v8qHz9vHfeuHIfOTfO99dCKXRtOXJQ2W9upQxtd\nvHyDSqViWfwiADYVfdajY/bXHKasuZLJIROJ9ovsz3guMSwKNKB7ku3JYY5Jkf6ogP25MoZaCCGE\nGCq+rvwGh9PBzMhpPbqqnmRIAHDZXbTmjhZePPo67fYOVo++/qI/bFbUtfDSR1ms31HIgdxa6hrb\ner3eq7GxjefTMlGp4GcrUgjw8wBOXbBaKM+64x/T2G5mUew8onwjXB0HgDEBSSToY8moO0bReX6u\nOuydfFTwKVq1luUnCruhZtj0Ro3Xx+Dr5sPRuixucq4kwM+DlPgRZBTWU1pjITrE19URhRBCCNEL\ndoedXZX78NC4MzlkYo+O6S7QGgu4NGR8f8Y7o40FW6izNrA4dh6Xhky4qHOYLO089f5hTJb2U7Z7\ne2iJDvElJtSXmBBfokN8CAnw6tE52zvsrP0wgxarjdsWJTMy0r97X7BXEF5aTynQFCiz7hh7q/YT\n5RPOopi5ro7TTaVSsTx+EU8feoGPCz/j/ol3nfWx6eW7MLU3cmX0LEZ4GgYw5cAZNgWaWqUmNWgM\nuyu/odBcQqJ/HLMnhpNRWM/2w5WsXpjs6ohCCCGE6IXM+mM0tpuZGXE5HlqPHh0T7RuJh8bdJY1C\nOu2dHKzNwODuz5K4+Rd1jlarjf/74AgmSzvXTI8jIUJPSY2FkmoLJTUWjpWYOFZi6n68u05DQoSe\n8ACv7sItLNALjfq7QVVOp5N/bzlGubGZ2RMjmD3h1LssapWaeH0MmfU5NHVY8NPJRW4laOls5Z2c\ndWhUGlaPWTXouh6ONMQzOiCJYw155Jnyu5v4fF9zRwufFW/DW+vFwkFUYPa1YVOgQdckxN2V33DE\nmEmifxzjEwIJ8HPn66xqrpudgKf7sHo7hBBCiCFlR/nJ5iBTe3yMRq0h0T+OzPocGtvN+Lvr+yve\nabLqc7DarWddq+18bHYH/9qQQbmxmTmXRLD8ilhUKhVj4wK6H9NqtVFWa6GkppmSagulNRZyihvI\nLvpuYWCtRk1UsHfXXbZQX2pNbezPqWVkpJ6brxx5xueO08eSWZ9Dobmku1u2GNz+m/cR5g4LV8Uv\nGlSLn3/f8viFHGvI46OCz/jVpQmnDVP+pHgrVruVa0cux8tt6K7DN6wqkiRDIh4ad44YM1mZuAy1\nWsXM8eGk7SxiX3YNsycOjnG4QgghhCt12jtp7mzB4OF//gcPEjWtRnJMx0n0jyPcJ/SCjh1pSCCz\nPoc8UwFTQi/pp4Sn+7bmMMBFDW10Op28tiWHYyUmJo4M5JYrk844587LQ0tytIHk6O+GgvnqPTmc\nXU1JTVfBVlLdTGlNM0VVlu7HGHzduWfFOLSaMxeO328UMhwLtLq2BkZ4GBTTPfCIMZNvaw4S4xvF\nldGzXB3nrGL8ohgfOJYjdVlk1eeQEji6e19tax07KvYQ6DmCmefp0Kp0w6pAc1NrGTtiVFcbz+Yq\nonzDmZEazke7itl2qIJZE8IV84smhBBC9Jd1+R+zr2o/j0z9jWKKtF0VewEu6oNb8omhVLmm/AEr\n0NpsVjLrjxHqFUzkRdzN2LCzkD1Z1cSH+3HXVWNRq3v++cVDpyUhQk9CxHd3C212BxXGFkpqLFTV\ntzB9XBh6b91ZzxHjF4VapT5vQ4ehaGfFHt7L3cDYEaNYPfqGc661Nxg0d7Twbs56tGott425YdAN\nbfyhZfELOVqXzceFnzFmRHL33eWPCj7B4XRwdcJitC7uPNnfhk0Xx5PGdy9anQl0XSGaODKQstpm\nCquaXBlNCCGEcDm7w86h2qN0OmwcPvG3crDrsHewp2o/vjqf7r/zFyLCJwwvrSfHe9jJ0WZ38J/P\ncnn0jW9pbuu84OeDrs8hNoeNSSETL/jicPqhCjZ9XUKwwZP7r0vF3a33H7i1GjUxob7MHB/Oqrkj\niQg6d9HhrtER6RNGaVN59xqzw0GnvZNPir4Euoao/nXfU2TV57g41bl9kJeGpbOZ5fELCfUOcXWc\n8wr3CeXSkPGUN1d2/z+o0FzCIWMGcX7RTAwa5+KE/W/YFWhjRySjVWu7CzSge2hj+qEKV8USQggh\nBoWiplJaOlsBTvlbOZgdqDlCm62NK8Ivu6gr62qVmpGGBOqtJuraGs75WGuHjX/+9wjphyrYfbSS\nP73+LeW1F76G2v7u4Y0X1jnycH4db36ei4+nGw/cMB4/r7Pf5epv8fpYbE47ZZbh8/lpT9V+zB1N\nzI2awcrEZbTa2njuyKv8N28jnfaLK9b708HaoxyoPUK8Poa5UTNcHafHlsYtQK1Ss6nwc+wO+/cW\npV42LEa7DbsCzUPrwShDIpUt1Rhb6wEYHWsg2ODJN8dqabEOvl8uIYQQYqBk1GUDXXdI8huLXLqA\nc0/tqNiDChXTwy+76HP0ZD00S2sHT7x7iKxiE+MTRrDqyiTqzFb++uYBDlzAuqpNHRZyTfnE+EUR\n7BXY4+OKqpp4YWMmbho1P78+lRBDz1rm95fhtmC1zWHj85JtuKm1zI+Zzbzomfx60n2EeAWTXr6b\nx/evpbK52tUxuzV1WHgvdz1uajduHX3DRTWicZVgr0Cmhk6iprWW17LeodBcwvigFBL8Y10dbUAo\n51+qD3UPc6zrujKoVqmYPSGCTpuD3RmD5xdLCCGEslitVq688krWr19PVVUVq1ev5uabb+bnP/85\nHR0dro7XIxl12ejUbiyImYsTJ0frslwd6ZxKmsootZQzLnBMr+bLJfmfLNDO3G6/3mzl0bcPUlRl\n4YqUUO69dhy3Lh7NPdek4MTJvzZkkrazEEcPFoc+WHsUh9PBpAtoDlJrauXp/x6h0+bg7qvHkhA+\ncN0mzyZeHwsMnwWr91UdwNTeyPSIqd1LC0T5hvPQ5PuZHjGVypZqHt//DOnlu3u9SHhvOZ1O3svd\nQEtnK1cnLCbEK8ileS7G4rh5aFUaDhkzUKvUXJ2w2NWRBsywLNDGBY5BheqUoRtXjAtFq1GRfqjC\n5b9UQgghlOn5559Hr+/64PzMM89w880388477xATE8O6detcnO78alqN1LQaGR2Q1F08HDEO7gLt\nZGv93nZ1C/MOwdfNhzxT/mmfAyrrWvjbWweoqm9l4ZQo7lg6unvdsEmjgvnf1ZMI1Hvw0e5intuQ\nSVv7uedkHag5jAoVlwb3bHijpbWD//vgCJbWTm6dn8TEkYPjw7bBwx9/dz2F5uIh/9nJ7rDzWclX\naNXa07og6jQ6bkpeyV3jbken0fHfvI28cPQ1l9593l9zuHtZqVmRl7ssR28EeBiYfmLJjOnhUxVZ\nZF6sYVmg+ep8SPCPpchcirm9q6Wsr5eOSaOCqW5oJbe00cUJhRBCKE1BQQH5+fnMnj0bgH379jFv\n3jwA5syZw549e1yYrmdODm9MCRxDoGcAkT7h5DYcp83W5uJkZ9bS2cqB2sMEeY4gOeD0RW0vhEql\nIsmQgLnDQm2rsXt7QaWZv791AJOlnevnJLBq7kjUP5gDExXsw8O3T2JUtD8H84z87a0D1Dae+T2r\nb2ug0FzCSEMCene/8+bq6LTzzIdHqTG1sWRqDHMuiezV6+xr8foYLB3N1FvPPXdP6b6pOUS91cQV\n4VPOulbe+KCx/G7KA4wyjCSzPoe/fvMUh6sG/gJHY7uZD/LS0Gl0rFbY0MYfWh6/kOtGXsXVCYtc\nHWVAKfdfrJfGB3UNSThizOjeNudks5DDw2eyqxBCiL7x2GOP8dBDD3V/39bWhk7X1cBhxIgRGI3G\nsx06aGTUZaNCRUrgKAAmBKVgc9rJqhucXer2Vu2n02FjRsS0PvkQenIeWu6JeWiZRfX8493DtLbb\nuGPJKBZfFnPWY329dPxy1QTmXRJJhbGFP7/+LdnFpxctB2qOADC5B8MbHQ4nL32cTUFFE1PHhrBy\nVvzFvKx+NRyGOdoddj4r/hKtSsP86NnnfKy/u56fTfgxKxKX0trZxt92PMu6vI8GrIGI0+nk3Zz1\ntNraWJGwlEDPEQPyvP3FQ+vBnKjpeGg9XB1lQA3tRQTOYWLQODYWfMKmws8ZO2I0IzwNJEboiQjy\n5kCuEXNLxznX/xBCCCFOSktLY8KECURFRZ1xf0+HfxkMXmi1vW+ZHhTke8HHWNqbu+7sjIgjISIc\ngDm6y9hU9DnHLLksHjez17nO50JyO5wOvv5mH24aN5alzMbH3bvXzz/NYzzv5q6npK2EY+VjeGbd\nUVQqFb/70RSmppx5rbIfZv7FLZcyOiGQF9Yf4akPjvDj5WNZPiO+u/PcoQNH0ag1zBs9FR/d2TM7\nnU5e2pDBwTwjqYmB/Oa2Kbhp++66+sX8jJzJpZoxrDv+EZXtFQQF9e8CyH2V+ULtKN6Hsa2eKxNm\nkHSW3/Efuil4GVPjx/PMnlfZVr6LQksR909bQ5Q+vF+zphftIbP+GONCklkx4cqLvnDhqve6t5Sa\n+4eGbYFm8PDn+pFX8W7uev6d9Ra/vOT/Q6vWMntCBG9/kceuo5UsnRbr6phCCCEUID09nbKyMtLT\n06murkan0+Hl5YXVasXDw4OamhqCg4PPex6TqbXXWYKCfDEaLRd83DfVB3E4HYzSJ3Uf7+70Idgz\nkIOVmVRUN6DTuPU639lcaO5j9XlUNxuZGjqJtiYHbVz4a/4htdMDf3c9B8uzSd8XhIe7lvuvTSUh\nxOeM2c6W+ZKEAH5z0yU8uyGDlzdmcqywntULkzFaayk1V5AaOJY287kzf7qvlE27i4gI8uauZWNo\nNLX0+vWdL/fF8HbocVO7kV2T32fnPJO+zHwhHE4H/83YjFqlZmbwFReUwQd/Hl3wW17c8w67Kvfx\n0Od/Z0XiMmZGTOuXVvEmayOvHvgAD40718evpL7u4n5mXPVe95bScp+rmBy2QxwBrgi/jCmhl1DS\nVMb6/M0ATBsbis5NzfbDlTgcQ3vCqxBCiL7x9NNP8+GHH/LBBx9w/fXXc88993D55Zfz2WefAfD5\n558zY8bgXoPo6In5Z+MCx3RvU6lUjA9KocPeQU5DnquindGOihPNQSJ71xzkhzw7QuhwWvEJsPLg\nzZeQHG24qPMkRur5w+2TiA31ZVdGFY+/c5BdZfsBmHSetc/2ZdfwwbZ8DL7uPHD9eLw8Bu/1dI1a\nQ6xfFJXN1bTZrK6O0+cO1h6lptXI1NBLGeEZcMHHu2t13DTqWu4adxs6jY4P8tJ44ejrfd5AxOl0\n8nbOOqx2KytHLmOE58X93IrBYVgXaCqVihuTVxLmHcL28t0cqDmMl4eWqWNCqDNbySwa2hNehRBC\n9J/77ruPtLQ0br75ZmL95OkAACAASURBVBobG7nmmmtcHemsbA4bx+pzCfQIIMw75JR9E4JPLE0z\niLo5NlhNZNRlE+0bSYxfz4acnY/D4eStL/IoKeia3jBvlifRIb0bLhXg58FDt1zC1LEhFFSa2V6y\nHzeV2ylF8A/llpr49+ZsPN01PHD9eAL8Bv/cmzh9DE6cFDeVujpKn3I4HXxa/CVqlZoFMXN7da7x\nQSn8bsoDJBsSyaw/xl+/eYrs+tyLPp/T6aTd3oG5vYmallq2lm7nWEMeYwKSuTxsSq+yCtcbvJdk\nBoi7RsedKat5bP8zvJ2zjkifcGZPjGDHkSrS/3/27jw+zrLe///rnj2TZLZksu9726RJutJSurAL\nuGDBVuQLiOLhoAIufMUvHjw/PccjIOrBDbUgCmKRigiIrC1QCnRPmqRNszb7MkkmeybLzPz+yNKW\nLtkzmfTzfDx4zGTmvu/5tCTpvOe6rs91uI6lyf69uFIIIcTc+vrXvz52/w9/+IMPK5m40vYKXO5+\n1kSuPGPqVVxwDBa9mYKWo7g9btSq6a+Rm6499fvw4p12a/1Rg0Metr1ylP3FzURGxtJOIfX9MxM2\ndFo1d1y3mOCQHt539dLfEsX+o61cnHXmmrY6Rze/+FsBXi987fosYsKCZqSG2ZZ8SqOQRbY03xYz\ng/IdRTT0NLE6Yjl24/TfD1r0Zr6W82V21uzmpfLX+FX+E2yKXceSkAxcQ/30DblwuV24hly4hvpH\n7vfTN3LrGnLhcp+89Xg9p10/QGPgpozNszJ9UsytCz6gAUQEhvGFjBv4Q9GzbCt8hvtWfI3EyGDy\ny1to7XARYp7/n14JIYQQU1XQcgzgrCM7KkVFtn0J79Z+QGl7BRm21Lku7zRDniH21O/FqAlg+ThT\nBSeir3+IX/29gKMnnKTFmLn7hqU8dGgvZe0VeLyeGekOqSgKmtAGqAVVRzRP/PMYNc3d3LgpeWw/\nNWdXPz97Pp/e/iHuuG4xixImP53OVxLMcQBUtJ+YlesPuAfo7p+5NXgT4fF6+NeJt1BQuCpheqNn\np1IpKi6P20CaNZmniv7Crpr32VXz/oTOM6j1GDTD6yQNGgMGjZ4A9fCtQW1gRXjOtDZrF/OHBLQR\nK8JzKG8/wXt1H7D9+N/ZkL2OyobjvJdfz/Xr519bWyGEEGImeL1eClqOEqAxkGJJPOsxOfZM3q39\ngHxHoc8DWr6jkK6Bbi6NvQSdenrdlrt6B/j58/lUNnSRkxLKnZ9egk6rJs2awgcN+6jpqpuRKZRu\nj5uDzfkEao3cc/3V/PqFIt7YX0Odo5t/+3QmapXCz5/Pp62zn80bkliTGTHt15xLQdpAwo1hnOis\nnrFQO6pzoItHD/yKPo+Lby/7GmHG0Bm79vkUtByjrruBFeE5s7JBclxwDN9ZeQ976vcy4B7AMBq0\nNAYMaj0BGsPIfQMBGj1alVZGxi4gF/QatI/7bOp1xAfHsrfxIF5bDQF6De8dqWfI7Rn/ZCGEEMIP\n1fc00uZystiWfs7pi8nmRAK1RvIdRWdMq5pro81B1kVfNK3rtHa4+J9nDlHZ0MXFWRF89bOZ6LTD\nf/70kf3QSkb2Q5uu0vYKuga6yQ1bSnRIMN+7ZQXZySEUnXDyX388wGM7jlDT3M3G3Giuuejce63N\nZ0nmeFzufhp6mmbsmgPuAR4/8hQtrjZ6Bnr5fcGf6HcPzNj1z8Xr9Y6Nnl2dcNmsvY5erePS2Eu4\nOuEyNsZezEWRK8ixZ5JhSyXeFEu40Y5ZH4xOrZNwdoGRgHYKrUrDlzJvxqgJ4IWKl8jO1NDRPUB+\nWYuvSxNCCCFmRcFI98al52lcoVapyQpdTMdAJ1WdNXNV2hnquxspa68kw5o6rVGNupYefvTMQRrb\nerl6VRy3X7NobKohQOoMB7T9TYcBWBmeC0CAXsPXNy/l2jXxNLf3cbymnZyUUL5wRarfvhFPMg8H\ny4qOEzNyPY/Xw1NHt1PVWcPqiOVcmbye+p5G/nzs+QnvKzhVRa3F1HTVkRuWdUbTHCHmwoQC2sMP\nP8yWLVvYvHkzb7zxxmnPffTRR3zuc59j69atfPe738Xj8e/RppAAK7cu3sqQZ4jqgHdBPcg7h+t8\nXZYQQggxK460HEWlqFgckn7e43Lsw90c8xyFc1HWWe2u+wiYemt9r9dLSU07P37mIM6ufm7clMzn\nLk05IxSZ9SYijGGUdVTi9rinVfOge5C85kIsevNYiAFQqRQ2b0jmq9dncfmKGP7tU0tOC4n+JumU\nRiEz4e9l/yTfUUiaJZmbMjZzW+6NJJnjOdicz66a3TPyGmfj9Xp59cRbALM6eibE+Yz7m+Cjjz6i\ntLSU5557jm3btvGjH/3otOcffPBBHnvsMbZv305PTw+7d8/eD81cyQxdxJXxm3AOOLEtKaboRBtN\nbdPfPFQIIYSYTzr6u6jqrCHFnIhRazzvsRnWVPRqHfmOwlkfwTgb15CLfY0HsejNZIYsmtA5Q24P\nlQ2dvLG/hl+/WMi3frWHH//5EL39Q3zxmgw+sfrc0wnTrMkMuAeo6preiGFR23FcbhcrwnPOujZr\nebqdmy5PQ6/zfXfM6QgzhhKoMc5Io5B3avews2Y3EcYw7si6BY1Kg0Y9PMvJpAvm7+WvUuIsm37R\nZ1HcVkpVZw059kyig87stCnEXBi3ScjKlStZunQpACaTib6+PtxuN2r18C+SF154gaCg4TawNpsN\np9M5i+XOnesSr6Syo4rS9go0EUG8mxfP5y5N8XVZQgghxIwpbB3dnHr8wKNVa1kSksGh5iPU9zTO\n+ZvX/U2Hcbn7uTxuwznXynX3DVJe10FZXQdltR1UNnQyMHRyZo8pUMfyNDsbc6NZknj+Lolp1hTe\nq/uQ423lY6NDU3GgcXh644rwnClfwx+oFBWJ5jgKW4vp6O/CrJ/aHnIFLUfZUfISwdog7sq+HaM2\nYOw5i97MlzJv5n8P/5YnCv/M/SvvmdGuhcOjZ28CcHXC5TN2XSEma9yAplarMRqHP1XbsWMH69ev\nHwtnwFg4a25uZs+ePdxzzz3nvZ7VakSjmf6nRHb79DaPnIj71n+F//vGj2iPLeH9ilDu+OzSsQXE\nUzUXdc8Gf6zbH2sG/6zbH2sG/6zbH2sW89fo+rOs0CUTOj7Hnsmh5iPkOQrnNKB5vV7eq/0QlaJi\nbdSqsceanX2U1nZQVtdOWV0n9S0nW7ErQLQ9kJQYCynRJlJiLNjNhgmv8Uq1DndwLmkv5xNMbapb\n35CLwtZjhBvDiAmKmtI1/EmiOYHC1mIqO6vGpsRORnVnLU8W/hmNSsO/Z3+RkIAzQ3SKJZHNqZ/k\n+ZJ/8PuCp/nGsjvRqrUzUT4lznIqOqrICl1MbPDC//8l5q8Jt9l/66232LFjB08++eQZz7W2tnLn\nnXfy/e9/H6vVet7rOJ3TnypotwfjcHRN+zrjU3Hbos/z88O/YyjmAC/uzmJT1tRb7s9d3TPLH+v2\nx5rBP+v2x5rBP+v215rF/DTgHqC4rYyIwPAJb8K7JCQDjaIm31HItYlXzHKFJ5U4y6nvaSQ1aBEf\n5Dkpqz1BWV0HXb2DY8fotWoWxVtJiTaTGmMmKcqE0TD1N+5B2kCigyKp6DjBoHtwSiHgiKOIQc8Q\nK8Kz/bb5x2Sc2ihksgGtzeXkN0f+wKBniDuybjnv9gYbotdS3VnL3saD/LXkH3xh0Q3TqnvUv0bW\nnn1C1p4JH5tQQNu9ezePP/4427ZtIzj49H9su7u7ueOOO7j33ntZt27drBTpS6nWZC6Luoy369/i\n5dq/syHzGzO6v4cQQgjhC8edZQx6Bsma4HouAIPGQIYtlcLWYhy9rRMOdlPV1NZLfnkLr7f+HXRQ\nsNfCkZ7hzoo2k55Vi8JIjbGQEm0mJixwxptspFtTqOtuoLKzijTr5Jc5HGjKAxb+9MZRCaZYVIqK\nykk2Cukb6uPX+U/SOdDFDamfItt+/hFdRVHYmv5Z6rsb+KBhHwmmWC6OXj2d0il1VlDaXsHikPQZ\n2ftOiOkYN6B1dXXx8MMP89RTT2GxnDnP98c//jG33nor69evn5UC54PPpF/Oh5VH6dXX81zhq3w+\n6zpflySEEEJMy1h7ffu52+ufTbY9i8LWYvJbCrk8bsOM1jQw6Ka4up2C8lYKKlppbu9DZWpFn+FA\n2xvB2owlpMaYSYk2YzMZZvS1zybNmszOmt2UOMsnHdC6BropdpYSFxxD2CxsdDwf6dQ6YoKiqO6s\nnfCoo9vjZlvBMzT0NLEx5mI2xU7sw36dWssdWbfw0P7H+GvJi0QFRZJojpty7a+deBuAT8jaMzEP\njBvQXn31VZxOJ/fee+/YY6tXryY9PZ1169bx4osvUlVVxY4dOwC47rrr2LJly+xV7AMqRcVnEz/L\nnyqf4P3m98hpTWNRSJqvyxJCCCGmxOP1UNByjCBtIAmmyb2pXRq6mGdRyHfMTEBrcvZSUN5KcW0H\nBWUtDI409TDo1OSmheIIzad1CL65/nPEmWKm/XqTkWJJREHhuLOcyX40e7j5CB6vh5UXyOjZqCRz\nPNVdtdR0143bXMXr9fKX4y9Q7CwlK3Qxm1M/OanXCgmw8cXMm/hV3hNsK3ya76y8G5Nu8tOqKzpO\nUOwsJcOaetpWCEL4yrgBbcuWLecNXIWFvtsPZS6tSovhr3tW0h//Pk8WPcv/W3XvjHYOEkIIIeZK\ndVctnQNdXBSxYtLT9oN0gaRYEiltr6CjvxOz3jSp8wcG3RyvGR4lO1LRSrOzb+y5aHsgS5NCyEoK\nISXGTGl7Gb/Mb2Bp6JI5D2cAAZoA4kwxnOispt89gF6tm/C5+5vyUFBYFp49ixXOP0nmeN6p3UNF\nR9W4Ae31qp182LCfuOAYvrjkpiktIVlkS+NTyVfzj/J/8UThM9yd85Vzdvk8l39VjoyeJcromZgf\nJtwk5EKnVqnYmL6EV0pa6E04yhOFf+Yby+6c9C8BIYQQwtcKWo4BkDXJ6Y2jcuxZlLZXkO8omtCm\n0c3OXgoq2jhS3srxaudY63u9Tk1uaihLk0PYsCIehobGzvF6vfyz8g0ArpnDhiQfl2ZJpqqzhvL2\nynE38x7V2uekouMEaZZkLHrzLFc4v0x0w+r9jYd5ueJ1bAYrdy794qTC78ddEbeRqs5a8hwF/L38\nn9yQ+qkJn3uis5qjbcdJsySTYkmccg1CzCQJaJOwPjuKl/fEobd3UUkVL5a/OunheCGEEMLXClqO\nolHUZFhTp3R+tn0Jz5f+g3xH4TkDmtfrZd+xZv7xfiWNbSc7OEeHBpKVPDxKlhpjRqMeHjWxWwNO\n61J6tO04lZ3VZNszfdryPN2awpvV71DiLJ9wQDvYPNIcJOLCmt4IYDVYsOotVHScwOv1nrV7Zamz\ngmeO/ZUAjYF/X/rFKe+ZNkpRFP7Pohtp7GliV837xAfHsjIid0LnyuiZmI8koE2CNVhPdkooh4+l\nE722j501u0kyJ5AbluXr0oQQQogJae1zUtfdwOKQdAwa/ZSuYTVYiA+OpaS9nJ7BXgK1xtOe7+ju\n5+k3SjhU4kCrUZGbGjocyhJDCDGP39zD6/Xyz4rhDYPnsp3/2SRZElArakqc5RM+50BTHmpFTa79\nwnx/kGSO52BzPi19bWd0+mzqaeZ3BX/Eg5c7Mm8hKihiRl7ToDHwlaxbePjAL/hz8Q4iA8OJGSfY\nV3fVUth6jGRzAqmWqW+jJMRMk37xk7QpNxo8GiK6LkGn0vLMsedp7nX4uiwhhBBiQgpaR7o3hk5t\neuOoHHsmHq+HwpHpkjAcrD4qauR72/ZyqMRBeqyFH35pFV/fvJSNOdETCmcARa3FVHXVkGvPmtMN\nsc9Gr9aRYIqluquWvqG+cY9v6GkaC8DGjwXXC0XiKfuhnaproJtf5z9J71AfX8i4gXTb5LcuOJ/w\nwDBuWbyVQc8gvyv4Ez2D599797UTO4Hh0bMLYZ864T8koE3S4kQbdouBgqJ+Nid/BpfbxbbCZxhw\nD45/shBCCOFjBY7hgJY5if3PziY7bHgj4jzHcLOw9u5+fvlCAb97+SiDbg9fuCKN+27KJcw6uZAy\nuvZMQfHp2rNTpVlT8OKlrL1y3GMvtL3PzmZsw+rOk+vQBtyD/PbIU7S42vhEwuVcFLliVl47276E\nqxMuo9XVxlNFf8Hj9Zz1uLruBvIdhSSa4qY81VeI2SIBbZJUisLGnGgGhjy4miJYF7Wauu4G/nh0\nu4Q0IYQQ81rfkIvS9gpig6On3Yk43GgnIjCcY23H2V1Qw39s28vh0hYy4iz84EuruWx5DKopjEoU\ntByluquOZWFLZ2z623SlWZOB4c29z8fr9XKg8TA6tY6saY5Q+rOYoCh0Ku3YhtUer4c/Ht1OZWc1\nqyKWzfq01WsTr2CxLZ2jbcf5Z+WbZz3mXydOrj2T0TMx30hAm4KLl0aiVinsOlzH5pRPkmROIM9R\nwE8P/ZrWPqevyxNCCCHO6lhbCW6vm6xpjp6NyjBlMOgZ4o8f7GbI7eXmK9P49udzCbMETOl6w6Nn\nb6KgzKumDYmmOLQqzbjr0Kq6amhxtbE0dPG0uhL6O7VKTbwplvruRvqGXLxY/ip5jgJSLUl8IeOG\nWQ9EKkXFbUs+T6jBxmsn3ibfcfqWUPXdjeQ1FxAXHMNi28QavwgxlySgTYHJqGNFRhgNrb1U1vdw\nd84drIlcSU1XHQ8feIyScT5hE0IIIXyhoGV4euNU2+uP8nq97Clo4J13h6ePWWOc/OBLq7h02dRG\nzUbltxRR213P8vBsIgPDp1XjTNKqtSSaE6jrbqB7oOecxx1oHJ7euDJ8Yh0EF7IkcwJevDxbvIO3\nq98j3BjGV7JuQaOam/50gVojd2Tdglal5U9Hn6Opp3nsuderduLFyzUyeibmKQloU7QxZ7gz0Dt5\n9WjVWr6QcQNb06+nb8jFL/K2sbNmN16v18dVCiGEEMPcHjdFLcVY9GZig6KnfB1nVz//u+MIT/zz\nGJ7eYIyqYDxBjVhN2mnV5/F6eHV09Cxh/oyejUofmeZY0n72UTSP18PB5nwCNUYybLKmaXQd2qHm\nIwRrg7gr+/Y5b5oSExzFFzJuwOXu53cFf8I15KKpp5mDTfnEBEVNex2mELNFAtoUpcVaiAoN5EBx\nM509AyiKwiXRa7gn998I0gbyt9KXR9alDfi6VCGEEIKKjip6hnrJDF00pVEDr9fL+0ca+N62vRwp\nb2VRvJUf3r6ai6JzcLn7OT6JNvRns682j7ruBlaE5xIRGData82G0XVopef4c5Y4y+kc6CI3LGvO\nRonms0RzPAoKWpWWO7NvIzTA5pM6Vkbksil2HY29zTx97HleGxk9+0TCZTJ6JuYt+Q0yRYqisDEn\nimffKuX9ggauuWj4k6JkSwLfWXk32wqeZn/TYRp6mvhK1i2E+OgXkxBCiNnX19fH/fffT2trK/39\n/dx1110EBQXx05/+FI1Gg9Fo5OGHH8ZsNvusxum013d29fPH14o5Ut6KXqfmlqvT2ZAdhaIoZJPJ\nzprd5DsKWDLBjZw/zuP18HzRP0fWnl02pWvMtvjgWHRq3TmDqHRvPF2g1shti7diNVhJMMX5tJbr\nk6+ltquePEcBAFGBESy1L/FpTUKcj4ygTcPazAh0GhXvHK7Dc8p0RovezD3L7uTiqNXUdtfz0IHH\nKG4r9WGlQgghZtOuXbvIzMzkmWee4ec//zk//vGP+Z//+R/++7//m6effprc3Fyee+45n9ZY0HIU\nnVpHmiV5wud4vV52H6kfGzVbkmDlh19axcac6LHRhyRzPMHaII44jp6zpfl4DjcXUNNRz6qIZYQb\n7VO6xmxTq9SkWBJp6m2mo7/ztOcGPUPkOQqw6M0kWxJ9VOH8syIil2RLgq/LQK1Sc3vmF7Dohz8g\nuTrhMlSKvAUW85d8d06D0aBl1eJwWjpcHK1sO+05rUrDTRmb+Xz6Z3EN9fPLvG28XPyWrEsTQogF\n6JprruGOO+4AoKGhgfDwcKxWK+3t7QB0dHRgtVp9Vl9TTzPNvS0ssqWhVU9srVhbp4ufPZ/PH14t\nxuv1cuvV6XxzSw6h5tM7NKoUFUvtS+ga7Kaio+ocVzu30bVnKkU1L9eenWo03H68m+PR1mL6hlws\nD8+WN/7zlEkXzD25X+HmjBvJDcvydTlCnJdMcZymTbnRvH+kgVc+rCIj3opGffov5nXRFxEVFMm2\ngj/xdP7fOBZezhcybkB3AbffFUKIhWrr1q00Njby+OOPo9VqufnmmzGZTJjNZr71rW/5rK6C1mMA\nE96b61iVk1++cIS+fjdLEm3cdnUGIWbDOY/PsWeyp34veY4CUiY5gnSoKZ/G3mY2Jq7BbgyZ1Llz\nLd2aAkCJs4yVESc7Ncr0Rv8QZrQTNk9HaIU4lQS0aUqMNJGTEkpeWQt/fK2Y2685c/F1kjme76y8\nhz8WP8uBpryRdWm3+mzBrBBCiNmxfft2jh07xn333YfNZuOXv/wly5cv56GHHuLZZ5/llltuOe/5\nVqsRjUY97Trs9uDTvi4uOI6Cwoa05ZgNwec466THXzpKX7+br96QzVUXxY/bTOFiWw5/OBpAQetR\n7lxz04SbL3g8Hl7fvxO1omLz4k9gDxq/Nl8KCUknMD+A8s7Ksb/jIIuWgtZjRAWHsywxw28aT3z8\ne8Qf+GPN4J91+2PN4L91f5wEtBnwlU8t5pG/HGZPQSOWID2bN5w5v9+sN/H9Td/gNx8+y+66D3l4\n/2N8MfMmFtnSfFCxEEKImVRYWEhISAiRkZEsWrQIt9vN3r17Wb58OQBr167l5ZdfHvc6TmfvtGux\n24NxOLrGvu4e7KHYUU6iOY6BLgVHV9d5zh5W1dhJoEHD8pQQWlq6J/S6S2wZ7G86zKHKYuKCYyZ0\nzr7GQ9R3NbE2chXhQfbT6p6vks1JHGkpori6moy4OHYW72XQPUhO6NIJ/1352se/R/yBP9YM/lm3\nP9YM/lf3+cKkTJSeAQadhntuzCbMGsA/P6zi7YO1Zz1Oo9awNf16vpBxA/3ufn6V9wRvVr0j69KE\nEMLPHThwgCeffBKAlpYWent7SU1NpaysDICCggLi4+N9UltRSzFevBOe3jjk9tDS3kdEyOT2rMqx\nZwKQ31w4oePdHjf/qnwLlaLi6oRLJ/VavpT2sf3Q9jcdBmR6oxBi5sgI2gwxGXV8c0sOP3r6IM++\nWYI5UMeKjLPv47I2ahWRgRFsK3yaF8tfpbqrlpsXfQ69rEsTQgi/tHXrVh544AFuuukmXC4XDz74\nIBaLhe9973totVrMZjM/+tGPfFLbZNefOdr7cHu8RNgmF9AWhaSjVWnIcxTyyeSrxz3+QFMezX0t\nrIta7Vdb0YwFNGcZna4VFLeVEhccPW+7Twoh/I8EtBkUZgngGzdm89Czh/jdy0UEBWjJiD97165E\ncxz/d8XdPFH4NIeaj9DY08xXsm6d9wukhRBCnMlgMPDoo4+e8fj27dt9UM1Jg54hjrUeJzQghAjj\nxDZ/bmwbnmY52YCmV+tYbEsnv6WIxp7m82427fa4efXEW6gVNVf50egZQGRgOEHaQEqc5XxUewiP\n18OK8NzxTxRCiAmSKY4zLD4imK9+NguvF37xwhFqms89H92sD+bu3K+wPnot9T2NPHTgMYpai+ew\nWiGEEAtZmbMCl7ufpaGLJ9y84mRAC5z062WPTnN0nH+a477GQ7T0tbI2ahU2g++2H5gKlaIi1ZpM\ne38H/yh+EwWF5eHZvi5LCLGASECbBUsSbHzpukX09bv52V/zaOnoO+exGpWGLemf4eaMGxn0DPLr\n/Cd5qmg7nQP+s8hRCCHE/FTQehSArNBFEz6nsXUkoE1yDdro66gUFXnnCWhuj5t/nXgbjaLmqvhN\nk36N+SB9ZJqjo6eVFEvi2AbIQggxEySgzZKLFkew9dIU2rsH+Nlf8+nuGzzv8WuiVvLt5V8lLjiG\n/U2H+MFHj/Bu7Qd4vJ45qlgIIcRC4vV6OeI4SoAmgGTzxPcma2zrRVGGp+1PllFrJM2STHVXLW0u\n51mP+ajxAK2uNi6OXo3VYJn0a8wHaSP7oYE0BxFCzDwJaLPoylVxXL0qjobWXv53Rz6ugaHzHh8b\nHM19K77GlrTrAfhryYs8cuAXVHXWzEW5QgghFpD6nkac/e0sCUlHrZr43mqNbb3YzQFoNVN7i5AT\nNjrNseiM54Y8Q7x2YicalYYr/XT0DCAsIBSL3oxaUZETluXrcoQQC4wEtFl2w6ZkLloSTnldJ488\nfRC35/wjYipFxfqYNTx40X2silhGdVcdjxz4JduP/53ewenvjyOEEOLCcMQxOr1xYt0bAbr7Bunq\nHZzS9MZRS0OXoKCcdR3aRw0HaHM5uSTqIr+eFqgoCrct3sq3Lv4KQdrJr9UTQojzkYA2y1SKwu3X\nLGJJgpV9Rxt5+vXjE9r3zKQL5tbFW7kn998IN9rZXfchP/joJ+xtOCj7pgkhhBhXQctRVIqKxbb0\nCZ8z1Q6OpzLrTSSa4yhrr6Rr4GSjrMGR0TOtSsMV8RunfP35ItWazIpoaQ4ihJh5EtDmgEat4q7r\ns0iOMfNefgP/eL9ywuemWZP57qp7+XTyJ3C5+/nTsef438O/paGnaRYrFkII4c86+jup6qohxZKE\nUTvxtWTTaRByqmx7Jl68FLQcHXvsw/r9OPvbuSR6DWa9aVrXF0KIhUwC2hwJ0Gv4/pcvwm4x8NKe\nE7xzuG7C547O1f+P1d9maegSStsr+NG+n/Fi2av0uwdmsWohhBD+qLBleHPqpZOY3ggnR9AipzGC\nBpAz0m5/tJvjoHuQ16t2olVpF8TomRBCzCYJaHPIGmzgm1tyCDZqefqN4xwqcUzq/JAAK/+29Fbu\nXHobFr2ZN6vf4Ycf/eSsC7GFEEJcuKbSXh9mZoojQGhACNFBkRxvK6VvyMWehn2093ewPmYNJl3w\ntK4thBALnQS02LTQ9gAAIABJREFUORZuNXLvjdnoNGp++1IRJTXtk75GVuhi/mP1t7gyfhOdA138\nruCPPH7kD7T2tc1CxUIIIfxJ/9AAxW2lRAaGExoQMqlzG9t6CdCrMQXqpl1Hjj2TIa+bPEchb5zY\nhU6l5Yq4jdO+rhBCLHQS0HwgMdLEXddn4vF4eWzHEeoc3eOf9DE6tY5PJ3+C/7fqXtIsyRS0HOOH\nex/ltRM7GfKcv52/EEKIhaug6RiDnqFJdW8E8Hi8NDt7ibAZURRl2nXk2Ifbz+8oeYmOgU42xFxM\nsC5o2tcVQoiFTgKaj2QlhfDFazLo7R/ip3/Np63TNaXrRASGc3fuV7h18VYMGj0vV7zGj/b9nGNt\nJdLtUQghLkAH6guAybXXB2jp6GPI7Z329MZRkYHh2ANCcLld6NU6Lo/bMCPXFUKIhW5CAe3hhx9m\ny5YtbN68mTfeeOO05z744ANuuOEGtmzZwq9+9atZKXKhWpsZyY0bk3F29fOzv+bT4xqc0nUURWFV\nxDIeXH0f66PX0tzr4Jd523jwwx/zQukrVHZUS1gTQogLgMfr4WB9AcHaIBJMsZM6d6bWn41SFGVs\nFG1DzMUE6WS/MCGEmAjNeAd89NFHlJaW8txzz+F0Orn++uu58sorx57/r//6L5544gnCw8O5+eab\nueqqq0hJSZnVoheSq1fH4ezu560DtfxixxG+uSUHnVY9pWsZtQFsSf8MayJXsLNmNwUtR3m75j3e\nrnkPq95CTlgmy8KWkmCKQ6XI4KkQQiw01V21dLg6WRO5ctK/50+22J+5IHV5/AaMmgDWx6ydsWsK\nIcRCN25AW7lyJUuXLgXAZDLR19eH2+1GrVZTU1OD2WwmMjISgA0bNvDhhx9KQJsERVHYelkqHd0D\n7C9u5rcvFfHvn8lEo556gIozxXDbks8z6B6k2FnK4eYCjrQUsavmfXbVvI9FbybHnklu2FKSzPES\n1oQQYoEocEyteyPM/AgaQJA2kCsTNs3Y9YQQ4kIwbkBTq9UYjcO/rHfs2MH69etRq4dHeBwOBzab\nbexYm81GTU3NLJW6cKkUhS9ft5juvkEOl7bwmxcLufPTmWg10wtOWrWWrNDFZIUuZsgzRHFbKYcd\nBRxxFPFO7R7eqd2DSRdMjj2L3LAsUiyJEtaEEMKPlbZXolVpyLClTfrcxrZeFCDcOvGNrYUQQsy8\ncQPaqLfeeosdO3bw5JNPTusFrVYjGs3UpvCdym73z31Uzlf3D+5cy38/uY/DpQ5++8pR/t9tq9BP\ncbrj2USGr2LTolUMedwUNh3no9pD7K/N4726D3iv7gPM+mBWxeRwUewyFttTUatOvrY//n37Y83g\nn3X7Y83gn3X7Y81i7lyVcCmBwVr06sm3yW9o68VmMkx5mr0QQoiZMaGAtnv3bh5//HG2bdtGcPDJ\nNwdhYWG0tLSMfd3U1ERYWNh5r+V09k6x1JPs9mAcjq5pX2euTaTuf//0Yn7190IOFTfzH7/Zw92b\nl6LXzfw/ltGaWDYnxPKZuOsoba/gcPMR8hyFvFm+mzfLdxOkDSTbvoRsexZL41IY7Fb8anRtIX+P\nzDf+WDP4Z93+WrOYO0tC0qf0fdLXP0RH9wBLEm3jHyyEEGJWjRvQurq6ePjhh3nqqaewWCynPRcT\nE0N3dze1tbVERESwa9cufvKTn8xasRcCrUbN1z6bxeP/KOJQiYOf/jWPe2/MJkA/4cHOSVGr1GTY\nUsmwpbIl/XrK2is43FzAYUcBe+r3sad+H+SDWlFjNViwGayEGKzYDBZCDDZsBgs2gw2L3nTaiJsQ\nQgj/Mbr+LHIG158JIYSYmnHf9b/66qs4nU7uvffescdWr15Neno6V1xxBf/5n//Jt771LQCuueYa\nEhMTZ6/aC4RGreLOTy9h2ytH2XesmZ9sz+ObW7IJNGhn9XVVioo0awpp1hRuTPs0FR1VFLUW0+3p\npL7DQZvLSYmz7JznWvTmU4Kb9ZQwZ8VqMKNRzU7IFEIIMT0nOzhKQBNCCF8b9x3zli1b2LJlyzmf\nX7lyJc8999yMFiWGQ9pXPrkErVrFnsJGHvnLYb61JYdg4+TXFUyFSlGRYkkkxZJ42nSZAfcgTpeT\nNlc7ra62027bXE7K209QRuUZ11NQsBkshBnthBnthI/8F2YMxaI3+9X0SSGEWGgaZqGDoxBCiKmR\nIY15TKVS+OK1i9BoVLybV8/DfznMt7fmYg6cm5B2Njq1lvDAMMIDz77WcMgzhNPVcUpoG75t6WvD\n0dfCsbYSjrWVnHaOVqUlzBh6RnALN9oJ0Eg3MSGEmG2z0WJfCCHE1EhAm+dUisItV6WjVat462At\nD/35EPd9PhdrsN7XpZ2VRqXBbgzBbgw56/N9Qy6aex009Tpo7m2hudcx/HVfC3XdDWccH6wLIixg\nJLgF2gkLCMVuDCXEYEU3hS5lQgghztTY2oteq563/7YIIcSFRAKaH1AUhc9fnopWo+Jfe6t56M+H\n+Pbncwg1+9/oUoDGQLwplnhT7GmPe71e2vs7aO5tGQlvDpr6HDT3OKjoOEF5x5nTJoN1QYQabIQE\nDK95G70fYrBh9cinwEKIudPX18f9999Pa2sr/f393HXXXaxbt47777+fqqoqAgMDeeyxxzCbzb4u\n9Qwer5dmZy8RIUYURfF1OUIIccGTgOYnFEXhho3JaDUqXtpzYmwkLcy6MIKIoihYDRasBgvptpTT\nnhv0DNHS1zoc3HoctLhaae1z0uJqo6qrlsrO6jOvt1fBojMTEmAlZCS4nQxwVsx6k6x7E0LMmF27\ndpGZmckdd9xBXV0dt99+O7fccgtWq5VHH32U5557jgMHDnDZZZf5utQztHW6GBjyyPRGIYSYJySg\n+RFFUfjMJUlo1CpeeK+CH4+EtMiQQF+XNqu0Kg2RgeFEBoaD/fTnPF4P7f0dtPY5aXW10drXRqvL\nSae7g4ZOxzmblqgVNaEBNiJGrhtpDCMyKIKwgFC06tntlimEWHiuueaasfsNDQ2Eh4eza9cu7r77\nboDzNtvyNVl/JoQQ84sEND903doEdBoV23eW8dCzh/n21hxi7EG+LssnVIpqrKV/Kkljj492nhz0\nDOEc6TQ5Gt5a+9pocbWNTafMdxSOnaegYDeGEGkcDm6jAS7caJfgJoQY19atW2lsbOTxxx/nG9/4\nBu+99x6PPPIIoaGhfP/73z9jP9H5QFrsCyHE/CIBzU9duSoOjUbFM2+U8PCzwy344yOCfV3WvKNV\naUY6RIae8ZzX66VzoIuGniYaeppoHLlt6Gkiv7eI/JaisWMVFOwBIWOBLSIwjMjACMKNdnQS3IQQ\nI7Zv386xY8e477778Hg8JCYm8rWvfY1f//rX/Pa3v+U73/nOec+3Wo1oNOpp12G3T/zfg46+IQAW\nJdsndd5s8PXrT4U/1gz+Wbc/1gz+Wbc/1gz+W/fHSUDzY5cui0GrVvHUv4p55C+H+eaWHJKiTL4u\ny28oioJZb8KsN5FhSx17fDi4dZ8MbL1NNHQPB7gjLUUc+VhwCzfaybZnsiI8h6igCF/8UYQQPlZY\nWEhISAiRkZEsWrQIt9uNSqVi5cqVAKxbt45f/OIX417H6eyddi2n7l05EZV17QDoFe+kzptpk617\nPvDHmsE/6/bHmsE/6/bHmsH/6j5fmJSA5ucuyY5Co1Gx7ZWj/GT7Yb7xuWxSY+bfFBp/MhzcgjHr\ng09rWOL1eukaHA5u9T1NNPY009DTSHVnLa9X7eT1qp1EBUawPDyHFeHZhAacfasBIcTCc+DAAerq\n6njggQdoaWmht7eXrVu3snv3bjZv3kxRURGJiYm+LvOsGtt6sQbrMejkLYEQQswH8tt4AVizJAKN\nWsXvXirip8/lc/cNS1kUb/V1WQuOoiiYdMGYdMGkWU8Gt373AIUtRznYlE9RazEvV7zGyxWvkWCK\nY0V4DsvClmLWy8imEAvZ1q1beeCBB7jppptwuVw8+OCDrFmzhu985zvs2LEDo9HIQw895Osyz9A/\n4Kats1/+zRBCiHlEAtoCsTIjDI1a4TcvFvLz5/P5+mezyEySEZy5oFfrWB6ew/LwHHoH+8h3FHKg\nKY/jzjJOdFbzt9KXSbUksSI8h5ywLAK1shBfiIXGYDDw6KOPnvH4Y4895oNqJq7JKQ1ChBBivpGA\ntoDkptr5+ual/PKFAn7+/BFWLQ7jqpVx0jxkDhm1AayJWsmaqJV0DnRxqPkIB5vyKGkvp6S9nOdK\nXmSRLY0V4TlkhS7GoNH7umQhxAVMWuwLIcT8IwFtgclKCuGbn8vmmTdL+KioiY+KmsiIs3D16jgy\nk0JQKYqvS7xgmHTBbIy5mI0xF9Pa5+RQcz4HmvIobD1GYesxtCotS0MXszw8h8Uh6WhV8uMohJhb\nDSMt9iMloAkhxLwh7wgXoPQ4Kz+4fRVFlW28vq+aohNOiqvbiQwxctWqONYsCUc7A22cxcSFBFi5\nIn4jV8RvpLGnmQNNeRxsyuNgcz4Hm/MJ0BhYGrqE2JAIPP0KARoDBo2BAPXIrUaPQWPAoDagV+tQ\nJGgLIWaAjKAJIcT8IwFtgVIUhcykEDKTQqhu6uKN/TXsPdrEU/8q5oV3y7l0eQybcqMJNup8XeoF\nJyIwjOuSruTaxCuo6arjQHMeB5vy2dt4kL2N45+vUlQY1PqR4GbAoNaPBbrRUKdWVONfaAIhz6I3\nkWCKJyowHLVKQr0QC01jay9ajQqb2eDrUoQQQoyQgHYBiAsP5svXLWbzhmTeOljDO4freXF3Ja9+\nWMXFWZFcuTKWcPn0dM4pikKcKYY4UwyfSb6Ghp4mNEYvja1OXEMu+tyu4duhU27d/ad93eZy4hrq\nx4t3VmvVqbTEmWJINMWTYI4j0RQnnSmF8HNer5dGZy/h1gCZ/i6EEPOIBLQLiDVYz40bU7huTQLv\nH2ngjf017DpcxzuH68hJDeWqVXGkxphl+pwPqBQV0UGR2O3BhKsmt8mix+thwD1wWoDzeD3Trsnr\n9eDoa+NEZxWVHdWUt5+grL1y7Hmr3kKiOY7MqDTs6nBig6LQqrXTfl0hxNxo7x6gf8At0xuFEGKe\nkYB2AQrQa7hiZSyXLo/m4HEHr++r5nBpC4dLW0iKMnHVqjiWpYWiVk1gmpzwOZWiGpveONNSrcms\njVoJQN+Qi+rOWio7q8dC26HmIxxqPgKAWlETExxFgml4hC3RHEeIwSaBX4h5qrG1B5AW+0IIMd9I\nQLuAqVUqVi0KZ2VGGKW1Hby+r5q80hZ+82IhoWYDV6yM5ZKlkb4uU8wTARoD6bYU0m3Dm3R7vV5a\nXW20eh0cqT1OZWc1tV31VHXW8C57AAjSBpJojiM6KAqzzoRZH4xp7DYYjXSuFMJnpEGIEELMT/Lu\nSKAoCmmxFtJiLTS29fLG/hr2FDTwl7dK+cfuSj65PomLF4cTFCDT18RJiqIQGhDCInsC6cYMAAbd\ng9R013Oio4rKzmoqO6opaDlGQcuxs14jUGvEpAvGrDNh0p+8HX4sGJPehFkXPK3RQY/Xg8frwe31\n4PG6cXs8ePES4gmc8jWFWAgaxgKa/CwIIcR8IgFNnCbCZuSWq9L5zCWJvHOojrcO1vLcmyW8+E45\nm3KjuWpVLOYg2VxZnJ1WrSXJHE+SOX7ssY7+Tpp6m+no76JjoJPOgS46+7voGOiis7+T9v5OGnqa\nzntdnVqHSRdMkDbwlMDlxu114/GcEr5Ggtjw48Nfn6uBilpRYdFbsBks2AxWrAYLNv0p9w0WdOqZ\n6XLq8XroHuyhvb+DdlcH7f2dw/f7O+gYuR+gCSDFkkiKJZEkcwJGbcCMvLYQ5yIjaEIIMT9JQBNn\nZTLq+NS6RK5aFcfB8lZ2vF3Ca/uqeetgLeuzI/nE6nhCpC2zmACz3jRux8cB9yBdAydD22m3p9yv\ndbWjKCrUihq1SoVq9L6iQqvSD3+tGv5apYzeDh+jUo0eqwa89Hp6ae5upay9Ei8VZ60rSBuIzWDB\narCOhLeR+yOhLkgbiNvrHglZnbT3t58Wvkbvd/R34va6z/nnN2oCaO5robKzijer30FBIToociSw\nJZFiSSRYFzSd/w1CnKGxtRdToA6jQd4KCCHEfCK/lcV56XVqPr0+mZWpoewpaODVj6rYeaiOd/Pq\nWZMZwbUXxUuLfjFtOrWWkAAbIQG2OXtNuz0Yh6OLIc8Q7f2dtLmcOF3ttLmctLnacfYP32/oaaa6\nq+6s19CoNAx5hs75GipFhUkXTGxwNBa9CYvefMp/Jix6C2a9CZ1ai2uon8rOKsraKylrr+BEZw21\n3fW8Uzu8ni/cGEZmRBoxhhhSLInYDNZZ+XsRF4bBITetHS5SYy2+LkUIIcTHSEATE6LVqNiYG826\npZHsPdrEqx9V8f6RBvYUNLBqUTjXroknxi6f8Av/o1FpCA2wEXqOcOj1euke7DkZ3vrbx+47XR3o\n1FosesvJAGYwj9036YJRTWTTcMCg0bPIlsYiWxowvJ6vqqt2LLBVdJzg7Yr3x463GaxjUyJTLEmE\nBYRKx0wxYU3OPrxApHRwFEKIeUcCmpgUjVrFxVmRrFkSwcESB698cIK9R5vYe7SJ3NRQrlubQGKk\nbGAsFg5FUQjWBRGsCyLOFDNnr6tVa8cCGFyK2+OmR9vB/spCytorKW+vZF/jIfY1HgIgWBdEijmR\neFPs2Do6q96MWW+acEgUF47GVll/JoQQ85UENDElKpXCyowwVqTbyS9v5ZUPToztpZaZaOO6tQmk\nydQZIWaMWqUm2RaPyW3jsrj1eLweGnuax0bYytorOewo4LCj4LTzVIoKs840FtisBsvIfQtWgxmr\n3kKQNlBG3y4w0iBECCHmLwloYloURSEnJZTs5BCOVTl55YMTFFa2UVjZRlqshU+uTWBxglXe/Akx\nw1SKiqigCKKCIlgfswav10tLXxv1PQ04XR04R6ZiDt92cKKzmgqv56zX0qo0WPUWLKeEuOigSHLs\nmTL6tkA1jI6gyRRHIYSYdySgiRmhKAqLE2wsTrBRVtvBKx+e4Eh5K48+l0diZDDXrU0gOyUUlQQ1\nIWaFoijYjSHYjSFnfd7tcdM50HVKcOsYvj0lxDU7y0475/9b8x1CA85+PeHfGtt6UasUQqUbrxBC\nzDsS0MSMS4kxc++N2VQ1dvHKhyc4eNzBL/5WgDVYz7JUO8vS7aTFmlGr5JN5IeaKWqUem96I+ezH\nDLoHae/vxNnvBJBwtkB5vV4a23oJswbI72EhhJiHJKCJWRMfEcxXr8+irqWHN/ZVc6jEwduHann7\nUC1BAVpyUkNZkW5nUbwNrUbeJAjha1q19ryjcGJh6OwdpK9/iIw4WScshBDzkQQ0MeuiQwP54jWL\n+D9XpVNS087BEgeHShy8f6SB9480YNCpyU4JZXmancwkGwadfFsKIcRsaWztAWT9mRBCzFfyTljM\nGY1aNbZO7QtXpFFR38nB480cPO4Ya9Wv1ajITLSxPN1OdkoogQatr8sWQogFRTo4CiHE/DahgFZS\nUsJdd93Fbbfdxs0333zac3/+85956aWXUKlUZGZm8sADD8xKoWJhUSkKKdFmUqLNfG5TCjXN3Rw8\n7uBgiWOsXb9apZARb2V5mp3cNDvmQJ2vyxZCCL83GtAibYE+rkQIIcTZjBvQent7+eEPf8iaNWvO\neK67u5snnniCN954A41Gw+23305eXh45OTmzUqxYmBRFIS48mLjwYK5fn0RDaw+HShwcPO6gqLKN\noso2nn79OKkxZpalh7EyIwxrsN7XZQshhF9qlBb7Qggxr40b0HQ6Hb///e/5/e9/f8ZzWq0WrVZL\nb28vRqORvr4+zOZztAcTYoIiQwK5dk0g165JoKWjj0MlLRw63kxpbQcltR38dWcZy9PtXLY8htQY\ns+yxJoTwub6+Pu6//35aW1vp7+/nrrvuYtOmTQDs3r2bL3/5yxw/ftzHVQ5rbOslKEBLUIBMIRdC\niPlo3ICm0WjQaM5+mF6v56tf/SqXX345er2ea6+9lsTExBkvUly4Qs0BXLkylitXxtLRM8Ch483s\nOlzP/uJm9hc3ExcWxGXLY1i9OBydVu3rcoUQF6hdu3aRmZnJHXfcQV1dHbfffjubNm2iv7+f3/3u\nd9jtdl+XCMCQ24Oj3UVSlMnXpQghhDiHaTUJ6e7u5re//S2vvfYaQUFB3HrrrRQXF5ORkXHOc6xW\nIxrN9N9I2+3B076GL0jd06kBUhJCuPHKDIoqWnnl/Uo+LGzgD/8qZse7FVx1UTyfWJtAmNU4crzv\na54Kf6zbH2sG/6zbH2u+EFxzzTVj9xsaGggPDwfg8ccf56abbuKRRx7xVWmncbT34fF6ZXqjEELM\nY9MKaOXl5cTGxmKz2QBYsWIFhYWF5w1oTmfvdF4SGH6D4nB0Tfs6c03qnjnhJj1fuiaD69clsOtw\nHe/m1bNjZyl/21XKsjQ7my9LIzxY53fTH+fj3/V4/LFm8M+6/bXmC8nWrVtpbGzk8ccfp7KykuLi\nYu655555E9BG159FSgdHIYSYt6YV0KKjoykvL8flcmEwGCgsLGTDhg0zVZsQ47KZDGzekMwn1yaw\n91gTbx+sHe4GedxBjD2Qy5bHcNGSCPQy/VEIMQe2b9/OsWPHuO+++4iMjOR73/vepM6f7VkmXQWN\nAKQlhszL8DwfaxqPP9YM/lm3P9YM/lm3P9YM/lv3x40b0AoLC3nooYeoq6tDo9Hw+uuvc+mllxIT\nE8MVV1zBl770JW655RbUajW5ubmsWLFiLuoW4jQ6rZpLlkaxLiuSsroO3i9sYk9+PX987Tg73inn\nkuwoLs2NJtQS4OtShRALUGFhISEhIURGRrJo0SJ6enooKyvj29/+NgDNzc3cfPPNPPPMM+e9zmzP\nMimrcQJg1CjzbjTWX0eI/a1m8M+6/bFm8M+6/bFm8L+6zxcmxw1omZmZPP300+d8fuvWrWzdunVq\nlQkxwxRFITXGwtrcWEoubhmZ/ljHa3ureX1fNTkpoVy2PIZF8Va/m/4ohJi/Dhw4QF1dHQ888AAt\nLS14PB527tyJSqUC4NJLLx03nM2FxtZeVIqCXT6sEkKIeWtaUxyFmM+swXo+uz6JT66NZ9+xZt4+\nWDu2CXZ0aCCXZEexOMFKdGighDUhxLRs3bqVBx54gJtuugmXy8WDDz44Fs7mk8a2XuwWAxr1/KtN\nCCHEMAloYsHTatRcnBXJ2swIKuo7eftgLfuLm9n+dikAwUYt6XFWFsVZyIi3EmEzSmATQkyKwWDg\n0UcfPefzO3funMNqzq67b5DuvkGSpcW+EELMaxLQxAVDURSSo80kR5vZcmkK+eWtFFc7Ka5ycqC4\nmQPFzQCYg3RkxFnJGAlsYZYACWxCCL832sFRWuwLIcT8JgFNXJDMQXrWZ0exPjsKr9dLs7OPYyNh\nrbi6nb1Hm9h7tAkYniqZEWclI97CojirNBoRQvilhrYeACKkxb4QQsxrEtDEBU9RFMJtRsJtRjbm\nROP1emlo7R0bXSuubufDokY+LBpuTx1qNowFtow4KzaTwcd/AiGEGF9j28gImgQ0IYSY1ySgCfEx\niqIQFRpIVGggly6LweP1Uu/oGRthK6lp5/2CBt4vaAAgMsTIxtxoLlkaiUEnP1JCiPnp5BTHQB9X\nIoQQ4nzk3aQQ41ApCjFhQcSEBXHFilg8Xi+1zd1jo2tFJ9r4y1ul/GN3JZuWRXPZ8hgsQXpfly2E\nEKdpbOslQK/BZNT6uhQhhBDnIQFNiElSKQpx4cHEhQdz5ao4OnsHeOdQHW8fquWfH1bx2t5qLloS\nzlWr4oixB/m6XCGEwO3x0OzsIy48WJoeCSHEPCcBTYhpMhl1fGpdIlevjuODokbe2FfDnoJG9hQ0\nkplo46rVcSyWjbGFED7U0uHC7fESKR0chRBi3pOAJsQM0WnVbMyJZn12FEfKWnl9XzWFlW0UVrYR\nGxbEVatiWbUoXDaIFULMubH1Z9IgRAgh5j0JaELMMJWikJMaSk5qKJUNnby+r5r9xc1se+UYf3u3\ngsuXx7AhJwqjQdaBCCHmhnRwFEII/yEBTYhZlBhp4s5PZ3LDhj7ePFDLe/n1PP9OOS99cIIN2VFc\nviKGULPsqyaEmF1jAU2mOAohxLwnAU2IORBqCeDzl6fy6XUJvJtXz5sHanhjfw1vHahlRYadq1bF\nkRhp8nWZQogFqrG1FwUIt8oHQkIIMd9JQBNiDhkNWj5xUTxXrIxl79EmXt9Xw75jzew71kxajJkV\nSyKxBWqJDQsixGxAJY1FhBAzoKGtlxCzAa1G7etShBBCjEMCmhA+oFGruDgrkrWZERw94eS1fdUU\nVbZRUtsxdoxBpybGPrz/Wqw9cHgvNnsQAXr5sRVCTFyva4jOngEyk2y+LkUIIcQEyDs9IXxIURSW\nJNpYkmjD2dVPZ7+bwtJmah091DZ3U1HfSVldx2nnhJoNxI6EtdiRDbTDLAGoVDLaJoQ4kzQIEUII\n/yIBTYh5whqsJy0pmPjQk2+iBoc8NLT2UNPcTU1zN7WO4dvDpS0cLm0ZO06nURFtDxwLbRnxVqJD\nA2XvNSEEjW09AERKQBNCCL8gAU2IeUyrUREXHkxcePBpj3f0DFDT3EVtc89YcKtu6qayoWvsGLvF\nQG6qndzUUFJizKhVsv+aEBciGUETQgj/IgFNCD9kDtRhTgwhMzFk7LEht4fGtl6qGrs4Ut7KkYpW\n3tg/3C0yKEBLdkoIual2liTa0GulUYAQF4qxTapDAn1ciRBCiImQgCbEAqFRq4abitiDuDgrksEh\nD8XVTg6XODhc1sKegkb2FDSi1ahYkmAjNzWU7NRQTEadr0sXQsyixrZe9Do1liD5WRdCCH8gAU2I\nBUqrUZGVFEJWUgg3e72caOjicKmDw6Ut5JUN/6e8BinR5uGpkGmhhFtlCpQQC4nH66XJ2UdUiKxJ\nFUIIfyHLC0UMAAAYaklEQVQBTYgLgEpRSIoykRRlYvOGZJraekcajTgoq+2gtLaDv+4qIyo0kNzU\nUHJT7SREBss+bEL4ubYOF4NDHiJD5MMXIYTwFxLQhLgAhduMXL06jqtXx9HZM0B+2XBXyKITbfzz\nwyr++WEV5iAdSxJsY8Euxh6ERi2NRoTwJ9IgRAgh/I8ENCEucKZAHZdkR3FJdhT9A26KTrRxuNRB\nflkrHxQ28kFhIzC8xi0+PIjEKBNJkcOhzW4J8HH1QojzaRgNaDKCJoQQfkMCmhBijF6nZlmanWVp\ndjweLw1tvVTUd1DZ0EVlfScnGrsor+8cOz7QoCE9wUZMiJGkKBMJkSZpOiLEPCIjaEII4X8koAkh\nzkqlUogODSQ6NJBLlg4/NjDoprqpm4qGzpHg1smh4mYOnXKe3WIgMXJ0lM1MXHgQOmnrLxa4vr4+\n7r//flpbW+nv7+euu+4iIyOD7373uwwNDaHRaHjkkUew2+1zWtdoi31pACSEEP5DApoQYsJ0WjUp\nMWZSYsz/f3v3Hh1lde9//D2TyYWEXCZhEpICQQJouARB0ARIgCi2xFZtdXGAg7QFsQoRF6WEqOXS\n4zkigvz0h7YULFoFq6fo6aHV9YOiSJFCBFQwActNBXIjTBKSkAvJ8Pz+CIzEXLgkzMwjn9daWcw8\n+3kmH/Z6yOabvWcP0L3xWKcAducWcLSggi8LKzlacJqPD5zk4wMnAfCzWujZNZRb+8VwW2IMYSGa\nYZPvni1btjBgwACmT59Ofn4+U6dO5eabb2b8+PFkZGSwbt06XnnlFbKysjyaq6i0msiwQAID9EsS\nERGzUIEmIu0S3jmQpIQuJCV0AcAwDErKa87PslXwZUEFRwsrOFJQwVvvH2ZAr0iS+8cwuI9DH5gt\n3xkZGRnux4WFhcTExLBw4UICAwMBsNvt5OXleTRT7dkGyirr6NfT7tHvKyIi7aMCTUQ6lMViIdoe\nTLQ9mOR+XQE4XVXHxwdOsiOviH1HnOw74iQwwI9b+jpI6d+VxHg7Vqu29BfzmzBhAkVFRaxcuZLg\n4MZlhS6XizfeeIOZM2d6NEtxaQ2g95+JiJiNCjQRuebCOwcydlh3xg7rTqHzDDvyitmZV+TeJTI8\nJIDb+sWQ0r8rPWI66wN1xbTefPNNDhw4wNy5c9mwYQPnzp0jKyuL5ORkUlJSLnm93R6Mzdb+mWWH\nI5T9x08D0LtHJA5HaLtf0xPMkvNiZswM5sxtxsxgztxmzAzmzf1tKtBExKNio0L4SVovfpx6A4fz\nT7Mjr5hdB4rZtOs4m3YdJ65LCCn9Y7itXwxdwrWNv5hDbm4uUVFRxMbGkpiYiMvlorS0lCVLlhAf\nH09mZuZlvU5ZWXW7szgcoZSUVHLwKycAnQOtlJRUtvt1r7ULuc3EjJnBnLnNmBnMmduMmcF8udsq\nJi/rU2cPHjzIHXfcwdq1a5u1FRYWMnHiRO6//34WLFhw9SlF5LpisVjo0y2CKd+/keWZI3n0JwMZ\neqODk2U1vL31KFm/28Ez6z5h62f5nKmt93ZckTbt3r2bNWvWAHDq1Cmqq6vZvn07/v7+zJo1yyuZ\ntMW+iMiV+fDD9y/rvBdeeI6CgvxrluOSM2jV1dU89dRTrS7NeOaZZ5g6dSpjx47lN7/5DQUFBcTF\nxXV4UBH57vK3WRnc18Hgvg6qa+vZ/a8SduYV8cWxcg4eL2fd3w8yKKELwxKjiesSgiO8k3alE58y\nYcIEnnzySSZNmkRtbS0LFixg1apV1NXV8cADDwCQkJDAokWLPJapqLSaAJuVyLAgj31PERGzKiws\nYPPmjYweffslz33ssTnXNMslC7SAgABWr17N6tWrm7WdO3eOPXv2sHz5cgAWLlzY8QlF5LoSHORP\n2qA40gbF4TxdS86BYnbkFrHnYAl7Dpa4zwsLCcAREYQjohOO8E5E2zs1Po7oRHjnAKx6H5t4UFBQ\nEM8991yTY+np6V5K07ibanFpDTGRwfq3ICJyGZYvX8KBA3mkpg7jzjvHUVhYwPPP/5bFi/+DkpKT\n1NTUMHXqQ4wYkUpm5kP88pdZbNnyPmfOVHHs2Nfk559g1qw5pKSMaHeWSxZoNpsNm63l00pLSwkJ\nCWHx4sXk5eUxdOhQ5sy5thWliFw/osKDyEiOZ9xtPTh+soq8L0spKa85/1XLV4WVHMmvaHadzc/a\npHhzRAThuFDAafZNrgNllXXU1bu0vFFETOm/PzjMri9OXtE1fn4WXC6j1fZhN0UzPr13q+0TJz7A\nO+/8NzfckMCxY1/x29++TFlZKbfemsy4cT8kP/8E8+dnM2JEapPrTp4sZtmy/8vOnf/kf//3bc8U\naG0xDIPi4mKmTJnC9773PR566CE+/PBDRo8e3eo1HblDlRkpt+eYMTOYM7cnMkdHh3HLgKbLp12u\nc5SU11DsrKao9AxFzmqKnGcoKq2m2HmGQmfLGy7YQwMZmNCFof1iGHJjNOGdA695/o5ixvtDPE/v\nPxMRuXqJif0BCA0N48CBPDZseAeLxUpFxelm5yYl3QxAdHQ0VVVVHfL921Wg2e124uLi6NGjBwAp\nKSkcOnSozQKtI3eoMhvl9hwzZgZz5vZ2Zj8gzh5EnD0IEqKatFXX1lNSXnvRrFvjV/6pM/zjs3z+\n8Vk+FqBXXBhJCVEkJXTx6W3+vd3XV0MFpXe4C7QoFWgiYj7j03u3OdvVko4cI/39/QH4+9//HxUV\nFbz00stUVFTw4IMPNDvXz++biSfDaH0G70q0q0Cz2Wx0796dr776ip49e5KXl8ddd93VIcFERNor\nOMif+K7+xHdtWiQYhkGNCz7cfYx9h09xOL+CIwUV/M+2LwnvHMDAXlEMSoiiX89IOgXq00jEfIqc\nmkETEbkSVqsVl8vV5Fh5eTmxsXFYrVa2bv2A+nrP7Cp9yf955ObmsmTJEvLz87HZbGzcuJH09HS6\ndevG2LFjeeKJJ8jOzsYwDPr27evVN0WLiFwOi8VCfGwoGcnxZCTHc6a2nrwvS9l3xMnnR518tK+Q\nj/YV4me10Ld7xPnZtSi6Rgb77OyayMW0xFFE5MrEx9/Av/71BbGxcURERAAwenQ62dm/ZP/+XO66\n626io6N55ZXmGyd2NIvRUXNxl6kjph7NuMwHlNuTzJgZzJnbjJmh9dznDIMvCyv4/IiTvUecfF30\nzTmOiCCSErowKCGKG3tE4N8B76e9Embsay1xvDIdNUb+/D82Uu86x//JHNkBqTzDrPe32TKDOXOb\nMTOYM7cZM4P5crc1PmrtjojIRawWCwlx4STEhXNvai9OV9Wx76iTfUec5H1Zyvt7TvD+nhME+Fvp\nFx9J9+jO2MMCiQwNxB4ahD00kJAgm2baxGvq6l04T9dyY48Ib0cREZGroAJNRKQN4Z0DSU2KIzUp\njgbXOQ6dOH1+du0Unx1u/Pq2AJsVe2jg+a8gIsMaH0eeL+DsYYGEdvJXESfXREFJFQZa3igiYlYq\n0ERELpPNz0pivJ3EeDvj03tTWlHLybIayirrKK2spbSyjrKKOsoq6yirrKW4rKaN17J8U8CFBtIz\nNozk/jGEBQd48G8k30X5JY3bPKtAExExJxVoIiJXKTIsiMiwoFbb6xvOUV5V5y7gyi4q4ErPHzt0\nvBwD2Lm/mD9vOczNvbswMimWAb0i8bNaPfeXke+M/JPnCzRtsS8iYkoq0ERErhF/mxVHRCccEZ1a\nPafBdY6yyjo+O3SKbfsK2HOwhD0HSwjvHMCIAbGMTIrVTIhckROaQRMRMTUVaCIiXmTzayzixg7r\nzh1Du/F1cSXb9hWyM6+Y93Z+zXs7v6ZPt3BGJsUybmSCt+OKCeSfrMLmZ6FLeOu/GBAREd+lAk1E\nxEdYLBZ6dg2jZ9cw/m1Mbz45VMJH+wrZ/1UZh06c5k+bDzH0pmhSk2Lp/b1wbTIizRiGQX5JFTH2\nYKxW3R8iIh3t/vt/xGuvvUVw8LVbpaACTUTEBwX4+5HcryvJ/bpyqryG7blF7Mgrcn+IdkxkMKlJ\nsQwf0JWIzoHejis+ouLMWaprG0jsYfd2FBERuUoq0EREfFyXiE7cM/IGpt4zkG17jvHRvkJ2/6uE\n9R8e4Z2tRxnYK5KRSXEM6h2FzU8bi1zPikqrAW0QIiJypaZO/Xeefvo5unbtSlFRIY8/PgeHI5qa\nmhpqa2uZPXsu/foN8EgWFWgiIiZhtVro1zOSfj0j+ffaenL2F7NtXyF7jzjZe8RJaLA/g/s4iLZ3\nIiosqPErPIjwzgFYtRzyulB4oUDTBiEiYmLvHP4bn578/Iqu8bNacJ0zWm0fHD2Qn/T+YavtaWlj\n2L79H9x333i2bdtKWtoYEhL6kJY2mj17drFu3R/5r/9aekWZrpYKNBEREwoJ8id9SDfSh3TjWHEl\nH33euLHIP/YWNDvXz2ohMiywSdHm/jM8iMjQIPxtmnn7LihyqkATEbkaaWljePHF57nvvvF89NFW\nMjNn8+abr/OnP71OfX09QUGtf6xOR1OBJiJicj1iQpkUE8r4Mb0pOHWG0oo6nBW1OE/XNv55/vEX\nx8pbfY3wkIBvCrewILpHd+a2/jGaeTMZLXEUke+Cn/T+YZuzXS1xOEIpKam86u/Zq1cCTmcJxcVF\nVFZWsm3bh3TpEs38+U/xxRf7efHF56/6ta+UCjQRke8Im5+VHjGh9IgJbbG9vsFFaWVdY+H2reKt\ntKKOr4sqOVpQ4T4/oVs40W18hpv4npq6BqLtnQgJ8vd2FBER00lJGcmqVb8lNXUU5eVlJCT0AWDr\n1i00NDR4LIcKNBGR64S/zY8YezAx9pZnV84ZBqerzuKsqAUDFWcm9Iu7+xMeEQznznk7ioiI6Ywa\nNYaHH57Kq6/+idraGv7zPxeyZctm7rtvPJs3b+Lddzd4JIcKNBERAcBqsWAPDcQeqm37zSoyLAhH\nVEi7lvmIiFyvEhP7s3Vrjvv5unXr3Y9HjhwFwF133X3Nc+hd4SIiIiIiIj5CBZqIiIiIiIiP0BJH\nERGRdqqpqSE7Oxun00ldXR0zZszgpptuIisrC5fLhcPhYOnSpQQEBHg7qoiI+DgVaCIiIu20ZcsW\nBgwYwPTp08nPz2fq1KkMGTKESZMmMW7cOJYvX8769euZNGmSt6OKiIiP0xJHERGRdsrIyGD69OkA\nFBYWEhMTQ05ODrfffjsAY8aMYceOHd6MKCIiJqEZNBERkQ4yYcIEioqKWLlyJT//+c/dSxqjoqIo\nKSnxcjoRETEDFWgiIiId5M033+TAgQPMnTsXwzDcxy9+3Ba7PRibza/dORyOlj+s3NeZMbcZM4M5\nc5sxM5gztxkzg3lzf5sKNBERkXbKzc0lKiqK2NhYEhMTcblchISEUFtbS1BQEMXFxURHR1/ydcrK\nqtudxeEINeXnoJkxtxkzgzlzmzEzmDO3GTOD+XK3VUzqPWgiIiLttHv3btasWQPAqVOnqK6uZvjw\n4WzcuBGATZs2kZqa6s2IIiJiEppBExERaacJEybw5JNPMmnSJGpra1mwYAEDBgxg3rx5vPXWW8TF\nxXHvvfd6O6aIiJiAxbjchfEiIiIiIiJyTWmJo4iIiIiIiI9QgSYiIiIiIuIjVKCJiIiIiIj4CBVo\nIiIiIiIiPkIFmoiIiIiIiI9QgSYiIiIiIuIjfP5z0J5++mn27t2LxWLhiSeeICkpyd32z3/+k+XL\nl+Pn50daWhozZ870YtKmnn32Wfbs2UNDQwO/+MUvuPPOO91t6enpdO3aFT8/PwCWLVtGTEyMt6IC\nkJOTw2OPPUafPn0A6Nu3L/Pnz3e3+2pf//nPf2bDhg3u57m5uXz66afu5/3792fIkCHu56+++qq7\n3z3t4MGDzJgxg5/97GdMnjyZwsJCsrKycLlcOBwOli5dSkBAQJNr2rr/vZn78ccfp6GhAZvNxtKl\nS3E4HO7zL3UveSNzdnY2eXl5REREADBt2jRGjx7d5Bpf7OtZs2ZRVlYGQHl5OTfffDNPPfWU+/x3\n3nmHF154gR49egAwfPhwHnnkEY/nFu/Q+OgZGh89Q2Ok9zJrjPRBhg/LyckxHnroIcMwDOPw4cPG\n+PHjm7SPGzfOKCgoMFwulzFx4kTj0KFD3ojZzI4dO4wHH3zQMAzDKC0tNUaNGtWkfcyYMUZVVZUX\nkrVu586dxqOPPtpqu6/29cVycnKMRYsWNTl26623eilNU2fOnDEmT55s/PrXvzZef/11wzAMIzs7\n23jvvfcMwzCM5557zli3bl2Tay51/3tCS7mzsrKMd9991zAMw1i7dq2xZMmSJtdc6l661lrKPG/e\nPOODDz5o9Rpf7euLZWdnG3v37m1y7O233zaeeeYZT0UUH6Lx0XM0Pl57GiM9R2OkOfj0EscdO3Zw\nxx13AJCQkMDp06epqqoC4Pjx44SHhxMbG4vVamXUqFHs2LHDm3Hdhg0bxgsvvABAWFgYNTU1uFwu\nL6e6er7c1xd76aWXmDFjhrdjtCggIIDVq1cTHR3tPpaTk8Ptt98OwJgxY5r1aVv3v6e0lHvhwoV8\n//vfB8But1NeXu7RTJfSUuZL8dW+vuDo0aNUVlZ65TeW4ps0PvoGX+7ri/ny+AgaIz1JY6Q5+HSB\ndurUKex2u/t5ZGQkJSUlAJSUlBAZGdlim7f5+fkRHBwMwPr160lLS2u2bGDhwoVMnDiRZcuWYRiG\nN2I2c/jwYR5++GEmTpzI9u3b3cd9ua8v2LdvH7GxsU2WEQCcPXuWOXPmMGHCBF555RUvpQObzUZQ\nUFCTYzU1Ne7lGlFRUc36tK3731Nayh0cHIyfnx8ul4s33niDH/3oR82ua+1e8oSWMgOsXbuWKVOm\nMHv2bEpLS5u0+WpfX/Daa68xefLkFts+/vhjpk2bxk9/+lP2799/LSOKD9H46FkaH68tjZGeozHS\nHHz+PWgX85Uf1Jdr8+bNrF+/njVr1jQ5PmvWLFJTUwkPD2fmzJls3LiRH/zgB15K2ahnz55kZmYy\nbtw4jh8/zpQpU9i0aVOz9d6+av369fz4xz9udjwrK4u7774bi8XC5MmTGTp0KAMHDvRCwrZdzr3t\nS/e/y+UiKyuL5ORkUlJSmrT54r10zz33EBERQWJiIqtWreLFF19kwYIFrZ7vS3199uxZ9uzZw6JF\ni5q1DRo0iMjISEaPHs2nn37KvHnz+Otf/+r5kOJ1vnTPXg6Nj55j9vERNEZeaxojfY9Pz6BFR0dz\n6tQp9/OTJ0+6fwP07bbi4uIrmq691rZt28bKlStZvXo1oaGhTdruvfdeoqKisNlspKWlcfDgQS+l\n/EZMTAwZGRlYLBZ69OhBly5dKC4uBny/r6FxKcTgwYObHZ84cSIhISEEBweTnJzsE319QXBwMLW1\ntUDLfdrW/e9tjz/+OPHx8WRmZjZra+te8paUlBQSExOBxk0Ivn0f+HJf79q1q9VlGwkJCe43cg8e\nPJjS0lJTLxeTy6fx0XM0PnqHxkjP0Rjpe3y6QBsxYgQbN24EIC8vj+joaDp37gxAt27dqKqq4sSJ\nEzQ0NLBlyxZGjBjhzbhulZWVPPvss/z+979374hzcdu0adM4e/Ys0HhjXdjJx5s2bNjAH/7wB6Bx\nyYbT6XTvnOXLfQ2NP7hDQkKa/fbp6NGjzJkzB8MwaGho4JNPPvGJvr5g+PDh7vt706ZNpKamNmlv\n6/73pg0bNuDv78+sWbNabW/tXvKWRx99lOPHjwON/1n59n3gq30N8Pnnn3PTTTe12LZ69Wr+9re/\nAY27W0VGRnp1FzbxHI2PnqPx0Ts0RnqOxkjfYzF8aZ6yBcuWLWP37t1YLBYWLlzI/v37CQ0NZezY\nsezatYtly5YBcOeddzJt2jQvp2301ltvsWLFCm644Qb3sdtuu40bb7yRsWPH8sc//pG//OUvBAYG\n0q9fP+bPn4/FYvFiYqiqquJXv/oVFRUV1NfXk5mZidPp9Pm+hsatg59//nlefvllAFatWsWwYcMY\nPHgwS5cuZefOnVitVtLT0722vWpubi5LliwhPz8fm81GTEwMy5YtIzs7m7q6OuLi4li8eDH+/v7M\nnj2bxYsXExQU1Oz+b+2HkCdzO51OAgMD3T+cExISWLRokTt3Q0NDs3tp1KhRXs08efJkVq1aRadO\nnQgODmbx4sVERUX5fF+vWLGCFStWcMstt5CRkeE+95FHHuF3v/sdRUVFzJ071/2fLG9tfSzeofHR\nMzQ+eianxkjvZdYY6Xt8vkATERERERG5Xvj0EkcREREREZHriQo0ERERERERH6ECTURERERExEeo\nQBMREREREfERKtBERERERER8hAo0ERERERERH6ECTURERERExEeoQBMREREREfER/x83WZ+Ot8th\n/AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f4e12d1c710>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "RAQROqCgfcQM",
        "colab_type": "code",
        "outputId": "ef6c7f89-50b8-4823-ab44-4203bafba3f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Test performance\n",
        "trainer.run_test_loop()\n",
        "print(\"Test loss: {0:.2f}\".format(trainer.train_state['test_loss']))\n",
        "print(\"Test Accuracy: {0:.1f}%\".format(trainer.train_state['test_acc']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test loss: 1.93\n",
            "Test Accuracy: 42.5%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bT0hnmy2vS26",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save all results\n",
        "trainer.save_train_state()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EvWFdAz0jh4B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 预测"
      ]
    },
    {
      "metadata": {
        "id": "76p9UmpYjjp5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Inference(object):\n",
        "    def __init__(self, model, vectorizer):\n",
        "        self.model = model\n",
        "        self.vectorizer = vectorizer\n",
        "  \n",
        "    def predict_nationality(self, surname):\n",
        "        # Forward pass\n",
        "        vectorized_surname = torch.tensor(self.vectorizer.vectorize(surname)).view(1, -1)\n",
        "        self.model.eval()\n",
        "        y_pred = self.model(vectorized_surname, apply_softmax=True)\n",
        "\n",
        "        # Top nationality\n",
        "        y_prob, indices = y_pred.max(dim=1)\n",
        "        index = indices.item()\n",
        "\n",
        "        # Predicted nationality\n",
        "        nationality = vectorizer.nationality_vocab.lookup_index(index)\n",
        "        probability = y_prob.item()\n",
        "        return {'nationality': nationality, 'probability': probability}\n",
        "  \n",
        "    def predict_top_k(self, surname, k):\n",
        "        # Forward pass\n",
        "        vectorized_surname = torch.tensor(self.vectorizer.vectorize(surname)).view(1, -1)\n",
        "        self.model.eval()\n",
        "        y_pred = self.model(vectorized_surname, apply_softmax=True)\n",
        "\n",
        "        # Top k nationalities\n",
        "        y_prob, indices = torch.topk(y_pred, k=k)\n",
        "        probabilities = y_prob.detach().numpy()[0]\n",
        "        indices = indices.detach().numpy()[0]\n",
        "\n",
        "        # Results\n",
        "        results = []\n",
        "        for probability, index in zip(probabilities, indices):\n",
        "            nationality = self.vectorizer.nationality_vocab.lookup_index(index)\n",
        "            results.append({'nationality': nationality, 'probability': probability})\n",
        "\n",
        "        return results"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9zlIp2uJcYHM",
        "colab_type": "code",
        "outputId": "844f4e9a-c565-48ab-f159-ccf487e4c348",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Load the model\n",
        "dataset = SurnameDataset.load_dataset_and_load_vectorizer(\n",
        "    args.split_data_file,args.vectorizer_file)\n",
        "vectorizer = dataset.vectorizer\n",
        "model = SurnameModel(input_dim=len(vectorizer.surname_vocab), \n",
        "                     hidden_dim=args.hidden_dim, \n",
        "                     output_dim=len(vectorizer.nationality_vocab),\n",
        "                     dropout_p=args.dropout_p)\n",
        "model.load_state_dict(torch.load(args.model_state_file))\n",
        "model = model.to(args.device)\n",
        "print (model.named_modules)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reloading!\n",
            "<bound method Module.named_modules of SurnameModel(\n",
            "  (fc1): Linear(in_features=28, out_features=300, bias=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc2): Linear(in_features=300, out_features=18, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7YND9nlvjjtK",
        "colab_type": "code",
        "outputId": "ce62853e-623d-47cb-b951-43f0b11ebb86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Inference\n",
        "inference = Inference(model=model, vectorizer=vectorizer)\n",
        "surname = input(\"Enter a surname to classify: \")\n",
        "prediction = inference.predict_nationality(surname)\n",
        "print(\"{}: {} → p={:0.2f})\".format(surname, prediction['nationality'], \n",
        "                                   prediction['probability']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Enter a surname to classify: Goku\n",
            "Goku: Korean → p=0.46)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8JXPtHs_pQEC",
        "colab_type": "code",
        "outputId": "f7060ecc-7846-4846-b93b-3f269eb4c0c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Top-k inference\n",
        "top_k = inference.predict_top_k(surname, k=3)\n",
        "print (\"{}: \".format(surname))\n",
        "for result in top_k:\n",
        "    print (\"{} → (p={:0.2f})\".format(result['nationality'], \n",
        "                                     result['probability']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Goku: \n",
            "Korean → (p=0.46)\n",
            "Japanese → (p=0.25)\n",
            "Chinese → (p=0.15)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "5uL1j7a8gJ_h",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "Lh2nWwDhgLXD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- tqdm notebook"
      ]
    }
  ]
}