seq2seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Implements: Sequence to Sequence Learning with Neural Networks**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this implementation we will build and train a lightweight sequence to sequence translation model proposed in this seminal <a href=\"https://arxiv.org/abs/1409.3215\">paper</a>\n",
    "\n",
    "The seq2seq model introduced in the paper uses two multilayered Long Short-Term Memory (LSTMs) models - one encodes the input sequence to a fixed-length vector and the other decodes the output sequence from this vector. We try to repoduce a small scale version of the architecture described in the paper.\n",
    "\n",
    "<img src=\"assets/model.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model will be trained to translate one language into another, in our case German to English. We will use a small chunk of the German-English corpus of <a href=\"bentrevett/multi30k\">Multi30K dataset</a>.\n",
    "\n",
    "<img src=\"assets/data.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cpu\n",
      "\n",
      "MODEL INFO\n",
      "Architecture: Seq2Seq with LSTM\n",
      "Size: 9.45M\n",
      "Hidden dim: 512\n",
      "Embedding dim: 256\n",
      "Num layers: 2\n",
      "Total: 9,451,255\n",
      "Trainable: 9,451,255\n",
      "\n",
      "DATA INFO\n",
      "Multi30k Dataset (German-English)\n",
      "Train samples: 4000\n",
      "Validation samples: 1000\n",
      "Source vocabulary size: 2058\n",
      "Target vocabulary size: 2039\n",
      "\n",
      "epoch: 1/50, train_loss: 4.831, batch: 63/63, mfu: 0.35%\n",
      "epoch: 1/50, train_loss: 5.342, val_loss: 4.683\n",
      "epoch: 2/50, train_loss: 4.646, batch: 63/63, mfu: 0.35%\n",
      "epoch: 2/50, train_loss: 4.870, val_loss: 4.592\n",
      "epoch: 3/50, train_loss: 4.903, batch: 63/63, mfu: 0.35%\n",
      "epoch: 3/50, train_loss: 4.683, val_loss: 4.602\n",
      "epoch: 4/50, train_loss: 4.506, batch: 63/63, mfu: 0.35%\n",
      "epoch: 4/50, train_loss: 4.587, val_loss: 4.620\n",
      "epoch: 5/50, train_loss: 4.274, batch: 63/63, mfu: 0.35%\n",
      "epoch: 5/50, train_loss: 4.482, val_loss: 4.690\n",
      "epoch: 6/50, train_loss: 4.292, batch: 63/63, mfu: 0.35%\n",
      "epoch: 6/50, train_loss: 4.382, val_loss: 4.653\n",
      "epoch: 7/50, train_loss: 4.385, batch: 63/63, mfu: 0.35%\n",
      "epoch: 7/50, train_loss: 4.313, val_loss: 4.625\n",
      "epoch: 8/50, train_loss: 4.220, batch: 63/63, mfu: 0.35%\n",
      "epoch: 8/50, train_loss: 4.302, val_loss: 4.666\n",
      "epoch: 9/50, train_loss: 4.162, batch: 63/63, mfu: 0.35%\n",
      "epoch: 9/50, train_loss: 4.237, val_loss: 4.636\n",
      "epoch: 10/50, train_loss: 4.282, batch: 63/63, mfu: 0.35%\n",
      "epoch: 10/50, train_loss: 4.249, val_loss: 4.705\n",
      "epoch: 11/50, train_loss: 4.077, batch: 63/63, mfu: 0.35%\n",
      "epoch: 11/50, train_loss: 4.179, val_loss: 4.719\n",
      "epoch: 12/50, train_loss: 4.550, batch: 63/63, mfu: 0.35%\n",
      "epoch: 12/50, train_loss: 4.161, val_loss: 4.720\n",
      "epoch: 13/50, train_loss: 4.604, batch: 63/63, mfu: 0.35%\n",
      "epoch: 13/50, train_loss: 4.141, val_loss: 4.685\n",
      "epoch: 14/50, train_loss: 4.377, batch: 63/63, mfu: 0.35%\n",
      "epoch: 14/50, train_loss: 4.086, val_loss: 4.638\n",
      "epoch: 15/50, train_loss: 4.096, batch: 63/63, mfu: 0.35%\n",
      "epoch: 15/50, train_loss: 4.052, val_loss: 4.637\n",
      "epoch: 16/50, train_loss: 3.965, batch: 63/63, mfu: 0.35%\n",
      "epoch: 16/50, train_loss: 4.045, val_loss: 4.576\n",
      "epoch: 17/50, train_loss: 4.422, batch: 63/63, mfu: 0.35%\n",
      "epoch: 17/50, train_loss: 3.950, val_loss: 4.613\n",
      "epoch: 18/50, train_loss: 3.979, batch: 63/63, mfu: 0.35%\n",
      "epoch: 18/50, train_loss: 3.968, val_loss: 4.616\n",
      "epoch: 19/50, train_loss: 4.065, batch: 63/63, mfu: 0.35%\n",
      "epoch: 19/50, train_loss: 3.956, val_loss: 4.463\n",
      "epoch: 20/50, train_loss: 3.944, batch: 63/63, mfu: 0.35%\n",
      "epoch: 20/50, train_loss: 3.886, val_loss: 4.519\n",
      "epoch: 21/50, train_loss: 3.983, batch: 63/63, mfu: 0.35%\n",
      "epoch: 21/50, train_loss: 3.820, val_loss: 4.545\n",
      "epoch: 22/50, train_loss: 3.408, batch: 63/63, mfu: 0.35%\n",
      "epoch: 22/50, train_loss: 3.802, val_loss: 4.506\n",
      "epoch: 23/50, train_loss: 3.439, batch: 63/63, mfu: 0.35%\n",
      "epoch: 23/50, train_loss: 3.779, val_loss: 4.485\n",
      "epoch: 24/50, train_loss: 3.769, batch: 63/63, mfu: 0.35%\n",
      "epoch: 24/50, train_loss: 3.722, val_loss: 4.447\n",
      "epoch: 25/50, train_loss: 3.700, batch: 63/63, mfu: 0.35%\n",
      "epoch: 25/50, train_loss: 3.680, val_loss: 4.374\n",
      "epoch: 26/50, train_loss: 3.435, batch: 63/63, mfu: 0.35%\n",
      "epoch: 26/50, train_loss: 3.675, val_loss: 4.407\n",
      "epoch: 27/50, train_loss: 3.638, batch: 63/63, mfu: 0.35%\n",
      "epoch: 27/50, train_loss: 3.631, val_loss: 4.334\n",
      "epoch: 28/50, train_loss: 3.297, batch: 63/63, mfu: 0.35%\n",
      "epoch: 28/50, train_loss: 3.578, val_loss: 4.384\n",
      "epoch: 29/50, train_loss: 3.918, batch: 63/63, mfu: 0.35%\n",
      "epoch: 29/50, train_loss: 3.580, val_loss: 4.340\n",
      "epoch: 30/50, train_loss: 3.275, batch: 63/63, mfu: 0.35%\n",
      "epoch: 30/50, train_loss: 3.483, val_loss: 4.387\n",
      "epoch: 31/50, train_loss: 3.321, batch: 63/63, mfu: 0.35%\n",
      "epoch: 31/50, train_loss: 3.464, val_loss: 4.332\n",
      "epoch: 32/50, train_loss: 3.353, batch: 63/63, mfu: 0.35%\n",
      "epoch: 32/50, train_loss: 3.429, val_loss: 4.431\n",
      "epoch: 33/50, train_loss: 3.446, batch: 63/63, mfu: 0.35%\n",
      "epoch: 33/50, train_loss: 3.424, val_loss: 4.313\n",
      "epoch: 34/50, train_loss: 3.425, batch: 63/63, mfu: 0.35%\n",
      "epoch: 34/50, train_loss: 3.369, val_loss: 4.330\n",
      "epoch: 35/50, train_loss: 3.404, batch: 63/63, mfu: 0.35%\n",
      "epoch: 35/50, train_loss: 3.353, val_loss: 4.340\n",
      "epoch: 36/50, train_loss: 3.443, batch: 63/63, mfu: 0.35%\n",
      "epoch: 36/50, train_loss: 3.336, val_loss: 4.319\n",
      "epoch: 37/50, train_loss: 3.101, batch: 63/63, mfu: 0.35%\n",
      "epoch: 37/50, train_loss: 3.284, val_loss: 4.361\n",
      "epoch: 38/50, train_loss: 3.235, batch: 63/63, mfu: 0.35%\n",
      "epoch: 38/50, train_loss: 3.234, val_loss: 4.335\n",
      "epoch: 39/50, train_loss: 3.321, batch: 63/63, mfu: 0.35%\n",
      "epoch: 39/50, train_loss: 3.194, val_loss: 4.341\n",
      "epoch: 40/50, train_loss: 3.309, batch: 63/63, mfu: 0.35%\n",
      "epoch: 40/50, train_loss: 3.188, val_loss: 4.392\n",
      "epoch: 41/50, train_loss: 3.245, batch: 63/63, mfu: 0.35%\n",
      "epoch: 41/50, train_loss: 3.132, val_loss: 4.344\n",
      "epoch: 42/50, train_loss: 3.384, batch: 63/63, mfu: 0.35%\n",
      "epoch: 42/50, train_loss: 3.127, val_loss: 4.306\n",
      "epoch: 43/50, train_loss: 2.786, batch: 63/63, mfu: 0.35%\n",
      "epoch: 43/50, train_loss: 3.069, val_loss: 4.357\n",
      "epoch: 44/50, train_loss: 3.151, batch: 63/63, mfu: 0.35%\n",
      "epoch: 44/50, train_loss: 3.037, val_loss: 4.312\n",
      "epoch: 45/50, train_loss: 3.090, batch: 63/63, mfu: 0.35%\n",
      "epoch: 45/50, train_loss: 3.000, val_loss: 4.297\n",
      "epoch: 46/50, train_loss: 3.195, batch: 63/63, mfu: 0.35%\n",
      "epoch: 46/50, train_loss: 2.970, val_loss: 4.345\n",
      "epoch: 47/50, train_loss: 3.292, batch: 63/63, mfu: 0.35%\n",
      "epoch: 47/50, train_loss: 2.945, val_loss: 4.342\n",
      "epoch: 48/50, train_loss: 3.103, batch: 63/63, mfu: 0.35%\n",
      "epoch: 48/50, train_loss: 2.910, val_loss: 4.357\n",
      "epoch: 49/50, train_loss: 2.978, batch: 63/63, mfu: 0.35%\n",
      "epoch: 49/50, train_loss: 2.872, val_loss: 4.356\n",
      "epoch: 50/50, train_loss: 2.787, batch: 63/63, mfu: 0.35%\n",
      "epoch: 50/50, train_loss: 2.787, val_loss: 4.420\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA04AAAIjCAYAAAA0vUuxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABsg0lEQVR4nO3dd3wUdeLG8Wc32Wx6D+n0AKGEXkJXRIqigGdBFFSUU7Hr/Tw8C1a8s6GiiKhgQxRPURREUCnSu6H3JJCQECCdJJvs/v4AohwlkDab5PN+vfaV7Ozs7DPwNeZhZr5jcjgcDgEAAAAAzstsdAAAAAAAcHYUJwAAAAAoA8UJAAAAAMpAcQIAAACAMlCcAAAAAKAMFCcAAAAAKAPFCQAAAADKQHECAAAAgDJQnAAAAACgDBQnAIDTuO2229SwYUOjYzilGTNmyGQyad26dUZHAYA6ieIEACiTyWS6qMfixYuNjnpeHTt21L333nve108Xk/M9Vq1aVY1pAQDOxtXoAAAA5/fpp5+e8fyTTz7RwoULz1oeGxtboc+ZNm2a7HZ7hbZxLqmpqdq4caOee+65Mtd97rnn1KhRo7OWN23atNJzAQBqDooTAKBMt9xyyxnPV61apYULF561/H/l5+fL09Pzoj/HYrGUK19Z5s+fL3d3d11++eVlrjto0CB16tSpSnIAAGouTtUDAFSKvn37qnXr1lq/fr169+4tT09PPfHEE5Kk7777TldddZUiIiJktVrVpEkTPf/88yopKTljG/97jdOBAwdkMpn06quv6v3331eTJk1ktVrVuXNnrV279qKz/fjjj7rsssvk4eFR4f38a6Y33nhDDRo0kIeHh/r06aMtW7actf6vv/6qXr16ycvLS/7+/rr22mu1ffv2s9Y7dOiQxowZU/pn1KhRI91zzz0qKio6Y73CwkI98sgjCgkJkZeXl4YNG6YjR45UeL8AABfGEScAQKU5evSoBg0apJtuukm33HKLQkNDJZ28fsjb21uPPPKIvL299euvv+rpp59Wdna2XnnllTK3O3PmTOXk5Ojvf/+7TCaT/vOf/2j48OHat29fmUepbDabFi1apJdeeumi9iErK0sZGRlnLDOZTAoKCjpj2SeffKKcnByNGzdOBQUFevPNN3X55ZcrISGhdL8XLVqkQYMGqXHjxpowYYJOnDiht99+Wz169NCGDRtKS2JKSoq6dOmizMxMjR07Vi1atNChQ4f09ddfKz8/X25ubqWfe//99ysgIEDPPPOMDhw4oEmTJum+++7Tl19+eVH7BwAoJwcAAJdo3Lhxjv/9X0ifPn0ckhzvvffeWevn5+eftezvf/+7w9PT01FQUFC6bPTo0Y4GDRqUPt+/f79DkiMoKMhx7Nix0uXfffedQ5Jj7ty5ZWb95ZdfHJIc+/fvv+B606dPd0g658NqtZ6VycPDw3Hw4MHS5atXr3ZIcjz88MOly9q1a+eoV6+e4+jRo6XLNm/e7DCbzY5Ro0aVLhs1apTDbDY71q5de1Yuu91+Rr4rrriidJnD4XA8/PDDDhcXF0dmZmaZfxYAgPLjVD0AQKWxWq26/fbbz1r+11PkcnJylJGRoV69eik/P187duwoc7s33nijAgICSp/36tVLkrRv374y3ztv3jy1bNnyoqc5f+edd7Rw4cIzHvPnzz9rvaFDhyoyMrL0eZcuXdS1a1fNmzdP0skJKTZt2qTbbrtNgYGBpevFxcWpf//+pevZ7XbNmTNHQ4YMOee1VSaT6YznY8eOPWNZr169VFJSosTExIvaPwBA+XCqHgCg0kRGRp5xWtlpW7du1ZNPPqlff/1V2dnZZ7yWlZVV5nbr169/xvPTJer48eNlvvfHH3/UkCFDylzvtC5dulzU5BAxMTFnLWvWrJm++uorSSotMs2bNz9rvdjYWC1YsEB5eXnKzc1Vdna2WrdufVH5KvJnAQAoP4oTAKDSnGvyhczMTPXp00e+vr567rnn1KRJE7m7u2vDhg16/PHHL2r6cRcXl3MudzgcF3zf/v37tWPHDk2ZMuXidqAGKO+fBQCgYihOAIAqtXjxYh09elTffPONevfuXbp8//79Vf7ZP/74o/z8/NSzZ89K3/bu3bvPWrZr167SUwIbNGggSdq5c+dZ6+3YsUPBwcHy8vKSh4eHfH19zzkjHwDAeXCNEwCgSp0+QvLXIyJFRUV69913q/yz582bpyuvvFKurpX/74Rz5szRoUOHSp+vWbNGq1ev1qBBgyRJ4eHhateunT7++GNlZmaWrrdlyxb9/PPPGjx4sCTJbDZr6NChmjt3rtatW3fW53AkCQCcA0ecAABVqnv37goICNDo0aP1wAMPyGQy6dNPP63yQnDixAn99ttveu+99y7pffPnzz/nhBXdu3dX48aNS583bdpUPXv21D333KPCwkJNmjRJQUFB+r//+7/SdV555RUNGjRI8fHxGjNmTOl05H5+fpowYULpei+99JJ+/vln9enTR2PHjlVsbKxSU1M1e/Zs/f777/L397/k/QcAVC6KEwCgSgUFBemHH37Qo48+qieffFIBAQG65ZZb1K9fPw0YMKDKPvfXX39VYWFh6RGgi/X000+fc/n06dPPKE6jRo2S2WzWpEmTlJ6eri5dumjy5MkKDw8vXeeKK67QTz/9pGeeeUZPP/20LBaL+vTpo3//+99q1KhR6XqRkZFavXq1nnrqKX3++efKzs5WZGSkBg0aJE9Pz0vccwBAVTA5OAcAAFAL3XvvvVq3bp3WrFlTqds9cOCAGjVqpFdeeUWPPfZYpW4bAOC8OOIEAKiV2rVrd0nTkAMAcCEUJwBArTR27FijIwAAahFm1QMAAACAMnCNEwAAAACUgSNOAAAAAFAGihMAAAAAlKHOTQ5ht9uVkpIiHx8fmUwmo+MAAAAAMIjD4VBOTo4iIiJkNl/4mFKdK04pKSmKjo42OgYAAAAAJ5GcnKyoqKgLrlPnipOPj4+kk384vr6+lbJNm82mn3/+WVdeeaUsFkulbBN1B+MHFcH4QXkxdlARjB9UhDONn+zsbEVHR5d2hAupc8Xp9Ol5vr6+lVqcPD095evra/hfPmoexg8qgvGD8mLsoCIYP6gIZxw/F3MJD5NDAAAAAEAZKE4AAAAAUAaKEwAAAACUoc5d4wQAAIDao6SkRDabzegYuAQ2m02urq4qKChQSUlJlX6Wi4uLXF1dK+U2RBQnAAAA1Ei5ubk6ePCgHA6H0VFwCRwOh8LCwpScnFwt91X19PRUeHi43NzcKrQdihMAAABqnJKSEh08eFCenp4KCQmpll/AUTnsdrtyc3Pl7e1d5k1nK8LhcKioqEhHjhzR/v37FRMTU6HPozgBAACgxrHZbHI4HAoJCZGHh4fRcXAJ7Ha7ioqK5O7uXqXFSZI8PDxksViUmJhY+pnlxeQQAAAAqLE40oSyVFY5ozgBAAAAQBkoTgAAAABQBooTAAAAUIM1bNhQkyZNuuj1Fy9eLJPJpMzMzCrLVBtRnAAAAIBqYDKZLviYMGFCuba7du1ajR079qLX7969u1JTU+Xn51euz7tYta2gMaseAAAAUA1SU1NLv//yyy/19NNPa+fOnaXLvL29S793OBwqKSmRq2vZv66HhIRcUg43NzeFhYVd0nvAEScAAADUAg6HQ/lFxYY8LvYGvGFhYaUPPz8/mUym0uc7duyQj4+P5s+fr44dO8pqter333/X3r17de211yo0NFTe3t7q3LmzFi1adMZ2//dUPZPJpA8++EDDhg2Tp6enYmJi9P3335e+/r9HgmbMmCF/f38tWLBAsbGx8vb21sCBA88oesXFxXrggQfk7++voKAgPf744xo9erSGDh1a7r+z48ePa9SoUQoICJCnp6cGDRqk3bt3l76emJioIUOGKCAgQF5eXmrVqpXmzZtX+t6RI0eWTkcfExOj6dOnlzvLxeCIEwAAAGq8E7YStXx6gSGfve25AfJ0q5xfq//5z3/q1VdfVePGjRUQEKDk5GQNHjxYL774oqxWqz755BMNGTJEO3fuVP369c+7nWeffVb/+c9/9Morr+jtt9/WyJEjlZiYqMDAwHOun5+fr1dffVWffvqpzGazbrnlFj322GP6/PPPJUn//ve/9fnnn2v69OmKjY3Vm2++qTlz5uiyyy4r977efvvt2rNnj77//nv5+vrq8ccf1+DBg7Vt2zZZLBaNGzdORUVFWrp0qby8vLRt27bSo3JPPfWUtm3bpvnz5ys4OFh79uzRiRMnyp3lYlCcAAAAACfx3HPPqX///qXPAwMD1bZt29Lnzz//vL799lt9//33uu+++867ndtuu00jRoyQJL300kt66623tGbNGg0cOPCc69tsNr333ntq0qSJJOm+++7Tc889V/r622+/rfHjx2vYsGGSpMmTJ5ce/SmPvXv3au7cuVq+fLm6d+8uSfr8888VHR2tOXPm6Prrr1dSUpKuu+46tWnTRpLUuHHj0vcnJSWpffv26tSpk6STR92qGsXJQHvSc/THwSx1bhio6EBPo+MAAADUWB4WF217boBhn11ZTheB03JzczVhwgT9+OOPSk1NVXFxsU6cOKGkpKQLbicuLq70ey8vL/n6+io9Pf2863t6epaWJkkKDw8vXT8rK0tpaWnq0qVL6esuLi7q2LGj7Hb7Je3faTt37pSrq6u6du1auiwoKEjNmzfX9u3bJUkPPPCA7rnnHv3888+64oordN1115Xu1z333KPrrrtOGzZs0JVXXqmhQ4eWFrCqwjVOBnp27jY98tVmLdudYXQUAACAGs1kMsnTzdWQh8lkqrT98PLyOuP5Y489pm+//VYvvfSSli1bpk2bNqlNmzYqKiq64HYsFstZfz4XKjnnWv9ir92qKnfeeaf27dunW2+9VQkJCerUqZPefvttSdKgQYOUmJiohx9+WCkpKerXr58ee+yxKs1DcTJQywhfSdL21GyDkwAAAMAZLV++XLfddpuGDRumNm3aKCwsTAcOHKjWDH5+fgoNDdXatWtLl5WUlGjDhg3l3mbz5s1VXFys1atXly47evSodu7cqZYtW5Yui46O1t13361vvvlGjz76qKZNm1b6WkhIiEaPHq3PPvtMkyZN0vvvv1/uPBeDU/UM1DL8ZHHaRnECAADAOcTExOibb77RkCFDZDKZ9NRTT5X79LiKuP/++zVx4kQ1bdpULVq00Ntvv63jx49f1NG2hIQE+fj4lD53OBxq0qSJrrnmGt11112aOnWqfHx89M9//lORkZG69tprJUkPPfSQBg0apGbNmun48eP67bffFBsbK0l6+umn1bFjR7Vq1UqFhYX64YcfSl+rKhQnA50uTjtSs2W3O2Q2V95hXgAAANR8r7/+uu644w51795dwcHBevzxx5WdXf3/6P7444/r8OHDGjVqlFxcXDR27FgNGDBALi5lX9/Vu3fvM567uLgoIyNDH330kR5++GFdffXVKioqUu/evTVv3rzS0wZLSko0btw4HTx4UL6+vho4cKDeeOMNSSfvRTV+/HgdOHBAHh4e6tWrl2bNmlX5O/4XJofRJy9Ws+zsbPn5+SkrK0u+vr6Vsk2bzaZ58+Zp8ODBZ50feiHFJXa1fGaBiortWvKPvmoQ5FX2m1DrlHf8ABLjB+XH2EFFOMP4KSgo0P79+9WoUSO5u7sbkqEus9vtio2N1Q033KDnn3/+kt+bnZ0tX19fmc1Vf+XQhcbKpXQDrnEykKuLWc1DTx623JbC6XoAAABwTomJiZo2bZp27dqlhIQE3XPPPdq/f79uvvlmo6NVG4qTwU6frscEEQAAAHBWZrNZM2bMUOfOndWjRw8lJCRo0aJFVX5dkTPhGieDxYafOuJEcQIAAICTio6O1vLly42OYSiOOBmsZYSfJGl7ao7BSQAAAACcD8XJYC1OHXE6lHlCmfkXvpEZAAAAzlTH5jlDOVTWGKE4GczX3aKoAA9JHHUCAAC4WKenwS4q4h+ecWH5+fmSVOEZILnGyQm0DPfVweMntD01W/FNgoyOAwAA4PRcXV3l6empI0eOyGKxVMu01qgcdrtdRUVFKigoqNK/N4fDofz8fKWnp8vf3/+i7jl1IRQnJxAb7quft6UxQQQAAMBFMplMCg8P1/79+5WYmGh0HFwCh8OhEydOyMPDQyaTqco/z9/fX2FhYRXeDsXJCbSMYEpyAACAS+Xm5qaYmBhO16thbDabli5dqt69e1f5DZQtFkuFjzSdRnFyAqfv5bQ7LVe2ErssLhxqBgAAuBhms1nu7u5Gx8AlcHFxUXFxsdzd3au8OFUmfkN3AlEBHvKxuqqoxK69R3KNjgMAAADgf1CcnIDJZFJsOKfrAQAAAM6K4uQkYk/dz2lbCsUJAAAAcDYUJyfx5wQR3MsJAAAAcDYUJydx+lS9banZ3AEbAAAAcDIUJyfRLNRHLmaTjuUVKT2n0Og4AAAAAP6C4uQk3C0uahzsJYnrnAAAAABnQ3FyIn89XQ8AAACA86A4OZE/J4igOAEAAADOhOLkRDjiBAAAADgnipMTaXmqOO3PyFN+UbHBaQAAAACcRnFyIiE+VgV7W+VwSDsPcz8nAAAAwFlQnJxMbLiPJG6ECwAAADgTipOTYYIIAAAAwPlQnJxMSyaIAAAAAJwOxcnJnC5OO1KzZbc7DE4DAAAAQKI4OZ1GwV5yczUrr6hEScfyjY4DAAAAQBQnp+PqYlbz0NMTRHC6HgAAAOAMKE5OiOucAAAAAOdCcXJCf05JTnECAAAAnAHFyQm1jPCTxL2cAAAAAGdBcXJCLU4dcTqUeUKZ+UUGpwEAAABAcXJCvu4WRQV4SOKoEwAAAOAMKE5OigkiAAAAAOdBcXJSsaeKExNEAAAAAMajODmplhEUJwAAAMBZUJyc1OlT9Xan5cpWYjc4DQAAAFC3UZycVFSAh3ysrioqsWvvkVyj4wAAAAB1GsXJSZlMptLrnLalcLoeAAAAYCSKkxOLPXU/J65zAgAAAIxFcXJipyeIYEpyAAAAwFgUJyf255TkOXI4HAanAQAAAOouipMTaxbqIxezScfyipSeU2h0HAAAAKDOojg5MXeLixoHe0ligggAAADASBQnJ1c6sx7XOQEAAACGoTg5OSaIAAAAAIxHcXJyf04QQXECAAAAjEJxcnItTxWn/Rl5yi8qNjgNAAAAUDcZWpwmTJggk8l0xqNFixYXfM/s2bPVokULubu7q02bNpo3b141pTVGiI9Vwd5WORzSzsM5RscBAAAA6iTDjzi1atVKqamppY/ff//9vOuuWLFCI0aM0JgxY7Rx40YNHTpUQ4cO1ZYtW6oxcfWLDfeRdPJ+TgAAAACqn+HFydXVVWFhYaWP4ODg86775ptvauDAgfrHP/6h2NhYPf/88+rQoYMmT55cjYmr358TRGQZnAQAAACom1yNDrB7925FRETI3d1d8fHxmjhxourXr3/OdVeuXKlHHnnkjGUDBgzQnDlzzrv9wsJCFRb+efPY7OyTkyzYbDbZbLaK78Cpbf31a2VrFvLnvZyq6jNgnKoeP6jdGD8oL8YOKoLxg4pwpvFzKRkMLU5du3bVjBkz1Lx5c6WmpurZZ59Vr169tGXLFvn4+Jy1/uHDhxUaGnrGstDQUB0+fPi8nzFx4kQ9++yzZy3/+eef5enpWfGd+IuFCxdW6vZOy8iXJFdtPXhcP/w4T2ZTlXwMDFZV4wd1A+MH5cXYQUUwflARzjB+8vPzL3pdQ4vToEGDSr+Pi4tT165d1aBBA3311VcaM2ZMpXzG+PHjzzhKlZ2drejoaF155ZXy9fWtlM+w2WxauHCh+vfvL4vFUinb/KviErte2/qrCovtatOtrxoEVW7hg7GqevygdmP8oLwYO6gIxg8qwpnGz+mz0S6G4afq/ZW/v7+aNWumPXv2nPP1sLAwpaWlnbEsLS1NYWFh592m1WqV1Wo9a7nFYqn0v6iq2ObJ7Uotwnz0x8Es7T6Sr6ZhfpX+GTBeVY0f1A2MH5QXYwcVwfhBRTjD+LmUzzd8coi/ys3N1d69exUeHn7O1+Pj4/XLL7+csWzhwoWKj4+vjniGig07PUEEN8IFAAAAqpuhxemxxx7TkiVLdODAAa1YsULDhg2Ti4uLRowYIUkaNWqUxo8fX7r+gw8+qJ9++kmvvfaaduzYoQkTJmjdunW67777jNqFavPnlOQUJwAAAKC6GXqq3sGDBzVixAgdPXpUISEh6tmzp1atWqWQkBBJUlJSkszmP7td9+7dNXPmTD355JN64oknFBMTozlz5qh169ZG7UK1aRlx8vS8bSkUJwAAAKC6GVqcZs2adcHXFy9efNay66+/Xtdff30VJXJeLU4dcUrJKlBmfpH8Pd0MTgQAAADUHU51jRPOz9fdouhAD0nS9tQcg9MAAAAAdQvFqQZhgggAAADAGBSnGiQ2/GRxYoIIAAAAoHpRnGqQlhGnjjgxQQQAAABQrShONUjLU0ec9qTnylZiNzgNAAAAUHdQnGqQqAAP+VhdVVRi194juUbHAQAAAOoMilMNYjKZSq9z4nQ9AAAAoPpQnGqY2FP3c2KCCAAAAKD6UJxqmNIJIihOAAAAQLWhONUwf05JniOHw2FwGgAAAKBuoDjVMM1CfeRiNulYXpHSsguNjgMAAADUCRSnGsbd4qLGwV6SuM4JAAAAqC4UpxqI65wAAACA6kVxqoFKpySnOAEAAADVguJUA8VF+kmSlu06otzCYoPTAAAAALUfxakG6to4SI2DvZRdUKxZa5KMjgMAAADUehSnGsjFbNLf+zSWJH2wbL+Kiu0GJwIAAABqN4pTDTW0faRCfa06nF2gOZsOGR0HAAAAqNUoTjWU1dVFd/Y8edTpvSV7ZbdzM1wAAACgqlCcarARXevL191V+47k6edtaUbHAQAAAGotilMN5m111aj4hpKkKUv2yuHgqBMAAABQFShONdxtPRrK6mrW5uRMrdp3zOg4AAAAQK1Ecarhgr2turFztKSTR50AAAAAVD6KUy1wV6/GcjGbtHTXEW05lGV0HAAAAKDWoTjVAtGBnro6LlySNHXpPoPTAAAAALUPxamWuLtPE0nSj3+kKPFonsFpAAAAgNqF4lRLxIb7qm/zENkd0vscdQIAAAAqFcWpFrnn1FGn2esPKj2nwOA0AAAAQO1BcapFujQKVIf6/ioqtmvG8gNGxwEAAABqDYpTLWIymXRP36aSpE9XJiq7wGZwIgAAAKB2oDjVMv1a1FNMPW/lFBZr5uoko+MAAAAAtQLFqZYxm036+6lrnT78fb8KbCUGJwIAAABqPopTLXRN2whF+LnrSE6hvt14yOg4AAAAQI1HcaqF3FzNGtOrsSRp6pK9KrE7DE4EAAAA1GwUp1rqps7R8ve06MDRfP205bDRcQAAAIAajeJUS3lZXTU6vqEk6b0le+VwcNQJAAAAKC+KUy02untDeVhclHAoS8v3HDU6DgAAAFBjUZxqsUAvN93YOVqSNGXJHoPTAAAAADUXxamWu7NXI7maTVq+56j+OJhpdBwAAACgRqI41XJRAZ66pl2EpJPXOgEAAAC4dBSnOuDuUzfEnb/lsPYdyTU4DQAAAFDzUJzqgGahProitp4cDun9pfuMjgMAAADUOBSnOuKeviePOn2z4ZDSsgsMTgMAAADULBSnOqJjg0B1aRioohK7Pvp9v9FxAAAAgBqF4lSH3N23sSTps1WJOpR5wuA0AAAAQM1BcapDLmteT3FRfsorKtGtH67WsbwioyMBAAAANQLFqQ4xmUx675aOivBz174jebp9+hrlFhYbHQsAAABwehSnOibC30OfjOmqAE+LNh/M0t2frldhcYnRsQAAAACnRnGqg5rW89b027vI081Fv+/J0KNfbVaJ3WF0LAAAAMBpUZzqqHbR/pp6a0dZXEz64Y9UTfh+qxwOyhMAAABwLhSnOqxXTIjeuLGdTCbp01WJevOX3UZHAgAAAJwSxamOuzouQs9d00qSNGnRbn268oCxgQAAAAAnRHGCbo1vqIeuiJEkPf39Vs3dnGJwIgAAAMC5UJwgSXqwX4xGxTeQwyE98tUmLd11xOhIAAAAgNOgOEHSyXs8TRjSSlfHhctW4tDdn63XpuRMo2MBAAAAToHihFJms0mv39BOvWKClV9Uotunr9Ge9FyjYwEAAACGozjhDG6uZr13S0e1jfbX8XybRn24WimZJ4yOBQAAABiK4oSzeFldNf22zmoS4qWUrAKN+miNjucVGR0LAAAAMAzFCecU6OWmT8Z0Vbifu/ak5+r2GWuVV1hsdCwAAADAEBQnnFekv4c+uaOL/D0t2pScqXs+36CiYrvRsQAAAIBqR3HCBcWE+uij2zrLw+KipbuO6IlvE+RwOIyOBQAAAFQrihPK1KF+gKbc0kEuZpO+Xn9QH/6+3+hIAAAAQLWiOOGi9G1eT09eFStJemnedv22I93gRAAAAED1oTjhot3WvaFGdImW3SE98MVG7U7LMToSAAAAUC0oTrhoJpNJz17TWl0aBSqnsFh3frKOacoBAABQJ1CccElO3yA3KsBDiUfzNW7mBtlKmGkPAAAAtRvFCZcs0MtNH4zuJC83F63Ye1TPzd1mdCQAAACgSlGcUC4twnw16ab2MpmkT1cl6tOVB4yOBAAAAFQZihPKrX/LUP3fgBaSpAlzt2nFngyDEwEAAABVg+KECrm7T2MNax+pErtD93y+QQcy8oyOBAAAAFQ6ihMqxGQyaeLwNmob7a+sEzbd+ck6ZRfYjI4FAAAAVCqnKU4vv/yyTCaTHnroofOuM2PGDJlMpjMe7u7u1RcS5+RucdG0WzsqzNdde9Jz9cAXG1VidxgdCwAAAKg0TlGc1q5dq6lTpyouLq7MdX19fZWamlr6SExMrIaEKEs9X3dNG9VJ7hazFu88opfnbzc6EgAAAFBpDC9Oubm5GjlypKZNm6aAgIAy1zeZTAoLCyt9hIaGVkNKXIw2UX569fq2kqRpy/Zr9rpkgxMBAAAAlcPV6ADjxo3TVVddpSuuuEIvvPBCmevn5uaqQYMGstvt6tChg1566SW1atXqvOsXFhaqsLCw9Hl2drYkyWazyWarnGtxTm+nsrZXkw2IDdF9fRtr8uJ9euLbBEX7W9WxQdmFuC5j/KAiGD8oL8YOKoLxg4pwpvFzKRlMDofDsItRZs2apRdffFFr166Vu7u7+vbtq3bt2mnSpEnnXH/lypXavXu34uLilJWVpVdffVVLly7V1q1bFRUVdc73TJgwQc8+++xZy2fOnClPT8/K3B2cYndIM3aZtfmYWd4Whx5tU6JAq9GpAAAAgDPl5+fr5ptvVlZWlnx9fS+4rmHFKTk5WZ06ddLChQtLr20qqzj9L5vNptjYWI0YMULPP//8Odc51xGn6OhoZWRklPmHc7FsNpsWLlyo/v37y2KxVMo2a7r8omLdNG2tth/OUYswH826s7O8rIYf4HRKjB9UBOMH5cXYQUUwflARzjR+srOzFRwcfFHFybDfZNevX6/09HR16NChdFlJSYmWLl2qyZMnq7CwUC4uLhfchsViUfv27bVnz57zrmO1WmW1nn24w2KxVPpfVFVss6bys1j0wW2dde3k37XjcI7+75utmnJLR7mYTUZHc1qMH1QE4wflxdhBRTB+UBHOMH4u5fMNmxyiX79+SkhI0KZNm0ofnTp10siRI7Vp06YyS5N0smglJCQoPDy8GhLjUkX6e2jqrZ3k5mLWz9vS9PwP22TgmaEAAABAuRl2xMnHx0etW7c+Y5mXl5eCgoJKl48aNUqRkZGaOHGiJOm5555Tt27d1LRpU2VmZuqVV15RYmKi7rzzzmrPj4vTsUGAXruhre7/YqNmrDigqAAP3dmrsdGxAAAAgEvi1BedJCUlyWz+86DY8ePHddddd+nw4cMKCAhQx44dtWLFCrVs2dLAlCjLkLYROpxVoBfnbdcLP25XqK+7hrSNMDoWAAAAcNGcqjgtXrz4gs/feOMNvfHGG9UXCJXmzl6NdCjzhGasOKBHv9qsej5WdW0cZHQsAAAA4KIYfgNc1A0mk0lPXd1SA1qFqqjErrs+WafdaTlGxwIAAAAuCsUJ1cbFbNKbN7VXh/r+yi4o1m3T1yotu8DoWAAAAECZKE6oVu4WF30wurMaB3vpUOYJ3T59rXILi42OBQAAAFwQxQnVLtDLTTNu76JgbzdtS83WPZ+tl63EbnQsAAAA4LwoTjBE/SBPfXRbZ3lYXLRsd4bGf5PAPZ4AAADgtChOMExclL/eHdlBLmaTvl5/UG8s2m10JAAAAOCcKE4w1GUt6umFoSdvePzWL7s1a02SwYkAAACAs1GcYLgRXerr/subSpL+NWeLftuRbnAiAAAA4EwUJziFR/o30/AOkSqxOzRu5gYlHMwyOhIAAABQiuIEp2AymfTy8Dj1iglWflGJbp+xVsnH8o2OBQAAAEiiOMGJuLma9e7IDooN91VGbqFGT1+j43lFRscCAAAAKE5wLj7uFs24vbMi/Ny170ie7vpkHTfIBQAAgOEoTnA6ob7umnFHF/m6u2pd4nENe2e59h3JNToWAAAA6jCKE5xSs1AffTKmq+r5WLU7PVfXTl6uhdvSjI4FAACAOoriBKfVLtpfPzzQU50bBiinsFh3fbJOr/+8U3a7w+hoAAAAqGMoTnBq9Xzc9fmd3XRb94aSpLd+3aMxH69VVr7N2GAAAACoUyhOcHpurmZNuKaVXr+hrayuZv2284iGTP5d21OzjY4GAACAOoLihBpjeIco/fee7ooK8FDSsXwNf3eFvt+cYnQsAAAA1AEUJ9QorSP9NPe+nuoVE6wTthI98MVGvfDDNhWX2I2OBgAAgFqM4oQaJ8DLTTNu76J7+jaRJH3w+37d8uFqZeQWGpwMAAAAtRXFCTWSi9mkxwe20JSRHeTl5qJV+45pyNu/a1NyptHRAAAAUAtRnFCjDWoTru/u66HGIV5KzSrQDe+t1Jdrk4yOBQAAgFqG4oQar2k9H303rof6twxVUYldj/83QeO/SVB6ToEcDu75BAAAgIpzNToAUBl83C2aektHvbt4j15buEtfrEnSF2uSZHU1KyrAQ9GBnooO8Cz9PirAQ9EBnvL3tMhkMhkdHwAAAE6O4oRaw2w26b7LY9Q60k/P/bBNBzLyVFhs194jedp7JO+c7/G2uioqwENRfylVXRoGqk2UXzWnBwAAgDOjOKHW6du8nvo2r6eiYrtSs07o4PETSj6Wr+Tj+X/5/oSO5BQqt7BYOw7naMfhnDO2cW/fJnqkfzO5unA2KwAAAChOqMXcXM1qEOSlBkFe53y9wFZyskidKlQHj+VrZ1qOFu88oncX79XaA8f01oj2CvfzqObkAAAAcDYUJ9RZ7hYXNa3nrab1vM9YPndzisZ/k6C1B45r8JvL9PqN7XRZ83oGpQQAAIAz4Dwk4H8MaRuhH+7vqVYRvjqeb9Pt09dq4vztspXYjY4GAAAAg1CcgHNoGOyl/97TXaPiG0iSpi7Zp5veX6VDmScMTgYAAAAjUJyA83C3uOi5a1vr3ZEd5GN11frE47rqrWVatC3N6GgAAACoZhQnoAyD24Trxwd6KS7KT5n5Nt35yTq98MM2FRVz6h4AAEBdQXECLkL9IE/Nvjtet/doKEn64Pf9umHqSh08nm9sMAAAAFQLZtUDLpLV1UXPDGmlbo2D9I/Zm7UpOVOD31ymV69vqytbhV3UNhwOh47mFelARp4OHM3XgYw8HTyWJ+88kwY5HFW8BwAAACgvihNwiQa0ClPLcF/d/8VGbUrO1NhP1+v2Hg01flCs3FzNcjgcOp5v0/6MvFMFKU/7M/KUeKoo5RQWn2OrLir6bpueH9ZGVleXat8nAAAAXBjFCSiH6EBPffX3eL2yYIemLduv6csPaNnuDHm6uWh/Rp5yCs5Vjk4ymaQIPw81DPZUwyAvmeXQZ6uT9NX6Q9p9JE/v3dJRob7u1bg3AAAAKAvFCSgnN1ez/nVVS3VtFKRHZ2/WnvTcM16P8HNXw2Cvk4+gkyWpUbCXogM95W7586iSzWaTV/YBzdxv1cakTF399u9675YO6tggsLp3CQAAAOdBcQIq6IqWoVrwUG/9tjNdgV5uahTspfr/U47KEuvv0Df3dNO4mZu1My1HN72/Ss9e01o3d61fhckBAABwsZhVD6gEYX7uGtGlvga0ClOzUJ9LKk2nNQj01Df3dtfgNmGylTj0xLcJeuLbBKY9BwAAcAIUJ8CJeFld9c7NHfSPAc1lMkkzVydpxLRVSs8uMDoaAABAnUZxApyMyWTSuMua6qPbOsvH3VXrE49ryOTftSHpuNHRAAAA6iyKE+CkLmteT9/f11Mx9byVll2om6au0pdrk4yOBQAAUCdRnAAn1ijYS9+O66EBrUJVVGLX4/9N0FNztnDdEwAAQDWjOAFOztvqqikjO+rR/s1kMkmfrkrUyA9W6UhOodHRAAAA6gyKE1ADmM0m3d8vRh+M6iQfq6vWHjiuIW9z3RMAAEB14T5OQA3SLzZUc+7robGfrNPeI3ka/u4K1fOxKi7KX22j/BQX7a+4SD8FeLkZHRUAAKBWoTgBNUyTEG/NGddD479J0Pwth5WeU6hF29O0aHta6TrRgR5/lqkof7WJ9JOXlf/cAQAAyovfpIAayMfdosk3d9CJohJtTcnS5oNZ+uNgpv44mKX9GXlKPnZCycdO6Mc/UiVJJpPUNMT7ZJmKPlmmWob7ys2Vs3UBAAAuBsUJqME83FzUqWGgOjUMLF2WlW9TwqEsbT6YWVqmUrMKtDs9V7vTc/XfDQclSVZXs+Ki/NShQYA61g9QhwYBCva2GrUrAAAATo3iBNQyfp4W9YwJVs+Y4NJl6TkFSjj455GpzcmZOp5v09oDx7X2wJ8TTDQM8jxZpE49Yur5yMVsMmI3AAAAnArFCagD6vm4q1+su/rFhkqSHA6H9mfkaX3icW1IOq71ice1Ky1XB47m68DRfH2z4ZAkycfqqnb1/UuLVLtof/m4W4zcFQAAAENQnIA6yGQyqXGItxqHeOv6TtGSTp7itzH5uDYkHtf6pOPalJSpnMJiLdudoWW7M069T+oVE6Jnr2mlRsFeRu4CAABAtaI4AZB08hS/vs3rqW/zepKk4hK7dqblnCxSp8pU8rETWrrriAZMWqoH+8VobO/GsrgwwQQAAKj9ylWckpOTZTKZFBUVJUlas2aNZs6cqZYtW2rs2LGVGhCAMVxdzGoV4adWEX66Nb6hJGl/Rp6e/m6Llu3O0CsLdmru5hS9fF2c2kX7G5oVAACgqpXrn4pvvvlm/fbbb5Kkw4cPq3///lqzZo3+9a9/6bnnnqvUgACcR6NgL31yRxe9fkNbBXhatONwjoa/u1zPzd2mvMJio+MBAABUmXIVpy1btqhLly6SpK+++kqtW7fWihUr9Pnnn2vGjBmVmQ+AkzGZTBreIUqLHumjoe0iZHdIHy3fryvfWKrfdqYbHQ8AAKBKlKs42Ww2Wa0n7/eyaNEiXXPNNZKkFi1aKDU1tfLSAXBaQd5WTbqpvT6+o4si/T10KPOEbp++Vg/O2qiM3EKj4wEAAFSqchWnVq1a6b333tOyZcu0cOFCDRw4UJKUkpKioKCgSg0IwLn1aRaihY/01p09G8lskr7blKIrXl+ir9cflMPhMDoeAABApShXcfr3v/+tqVOnqm/fvhoxYoTatm0rSfr+++9LT+EDUHd4urnqyatbas64HooN91Vmvk2Pzd6sWz9co8SjeUbHAwAAqLByzarXt29fZWRkKDs7WwEBAaXLx44dK09Pz0oLB6BmiYvy1/f39dAHy/Zr0qJd+n1PhgZMWqqHrmimO3s2kitTlwMAgBqqXL/FnDhxQoWFhaWlKTExUZMmTdLOnTtVr169Sg0IoGaxuJh1T98mWvBQb3VvEqQCm10vz9+hIZOX65sNB3WiqMToiAAAAJesXMXp2muv1SeffCJJyszMVNeuXfXaa69p6NChmjJlSqUGBFAzNQz20ud3dtUrf4uTn4dF21Oz9chXm9XlxUV6ck6CthzKMjoiAADARStXcdqwYYN69eolSfr6668VGhqqxMREffLJJ3rrrbcqNSCAmstkMun6TtH69dE+euzKZooO9FBOYbE+W5Wkq9/+XYPfXKZPVh5QVr7N6KgAAAAXVK7ilJ+fLx8fH0nSzz//rOHDh8tsNqtbt25KTEys1IAAar4gb6vuuzxGSx67TJ/f2VXXtI2Qm4tZ21Kz9fR3W9XlpUV6aNZGrdibIbudmfgAAIDzKdfkEE2bNtWcOXM0bNgwLViwQA8//LAkKT09Xb6+vpUaEEDtYTab1KNpsHo0DVZmfpHmbDykWWuTteNwjuZsStGcTSlqEOSpGzpF628doxTq6250ZAAAAEnlPOL09NNP67HHHlPDhg3VpUsXxcfHSzp59Kl9+/aVGhBA7eTv6abbejTS/Ad76btxPXRz1/rytroq8Wi+XlmwU91f/lV3frxWP289rLTsAhUWM6kEAAAwTrmOOP3tb39Tz549lZqaWnoPJ0nq16+fhg0bVmnhANR+JpNJbaP91TbaX09eFat5CYf15dokrT1wXIu2p2vR9vTSdb2trgr0clOAl5sCPS2nvp567uWmAM+TXwO9LAr0ssrfwyKz2WTg3gEAgNqiXMVJksLCwhQWFqaDBw9KkqKiorj5LYAK8XRz1d86RulvHaO0Jz1Xs9cla+7mFB3OLpDdIeUWFiu3sFhJx/IvantN63lrxu2dFRXA/eUAAEDFlKs42e12vfDCC3rttdeUm5srSfLx8dGjjz6qf/3rXzKbucklgIppWs9b4wfHavzgWNntDmUX2HQsr0jH84t0LM+m43lFOpZfdPJr6fIiHc8/uV7WCZv2pOdq1IdrNPvueAV5W43eJQAAUIOVqzj961//0ocffqiXX35ZPXr0kCT9/vvvmjBhggoKCvTiiy9WakgAdZvZbJK/p5v8Pd0u+j2pWSf0tykrtS8jT7dNX6uZd3WVj7ulClMCAIDarFyHhj7++GN98MEHuueeexQXF6e4uDjde++9mjZtmmbMmFHJEQHg0oX7eejTMV0U5OWmhENZGvvJehXYmGACAACUT7mK07Fjx9SiRYuzlrdo0ULHjh0rV5CXX35ZJpNJDz300AXXmz17tlq0aCF3d3e1adNG8+bNK9fnAaj9God46+M7usjb6qqV+47qgS82qrjEbnQsAABQA5WrOLVt21aTJ08+a/nkyZMVFxd3ydtbu3atpk6dWuZ7V6xYoREjRmjMmDHauHGjhg4dqqFDh2rLli2X/JkA6obWkX6aNqqT3FzN+nlbmsZ/kyCHg5vsAgCAS1Ou4vSf//xHH330kVq2bKkxY8ZozJgxatmypWbMmKFXX331kraVm5urkSNHatq0aQoICLjgum+++aYGDhyof/zjH4qNjdXzzz+vDh06nLPEAcBp8U2CNHlEe5lN0uz1B/Xy/B1GRwIAADVMuSaH6NOnj3bt2qV33nlHO3ac/AVk+PDhGjt2rF544QX16tXrorc1btw4XXXVVbriiiv0wgsvXHDdlStX6pFHHjlj2YABAzRnzpzzvqewsFCFhYWlz7OzsyVJNptNNpvtonNeyOntVNb2ULcwfqrHZc2C9OLQVhr/7VZNXbpPPlYX/b13I6NjVRjjB+XF2EFFMH5QEc40fi4lQ7nv4xQREXHW7HmbN2/Whx9+qPfff/+itjFr1ixt2LBBa9euvaj1Dx8+rNDQ0DOWhYaG6vDhw+d9z8SJE/Xss8+etfznn3+Wp2fl3ttl4cKFlbo91C2Mn6rnKenaBiZ9l+iiVxfuVvLeHeoeWjtO22P8oLwYO6gIxg8qwhnGT37+xd0bUqpAcaqo5ORkPfjgg1q4cKHc3d2r7HPGjx9/xlGq7OxsRUdH68orr5Svr2+lfIbNZtPChQvVv39/WSxMd4xLw/ipXoMlhf28W1OX7dfs/S7q2aWtBrYKLfN9zorxg/Ji7KAiGD+oCGcaP6fPRrsYhhWn9evXKz09XR06dChdVlJSoqVLl2ry5MkqLCyUi4vLGe8JCwtTWlraGcvS0tIUFhZ23s+xWq2yWs++8aXFYqn0v6iq2CbqDsZP9fnn4FhlFxbrizXJenR2ggK93dWjabDRsSqE8YPyYuygIhg/qAhnGD+X8vnlmhyiMvTr108JCQnatGlT6aNTp04aOXKkNm3adFZpkqT4+Hj98ssvZyxbuHCh4uPjqys2gFrAZDLphaFtNKh1mIpK7Br7yTptTs40OhYAAHBil3TEafjw4Rd8PTMz86K35ePjo9atW5+xzMvLS0FBQaXLR40apcjISE2cOFGS9OCDD6pPnz567bXXdNVVV2nWrFlat27dRV9TBQCnuZhNmnRTO2XPWKvle47qtulrNPvueDWt53PJ2yqwlWhrSra2HMqS1dWsBkFeahjsqVAfd5nNpipIDwAAqtslFSc/P78yXx81alSFAv1VUlKSzOY/D4p1795dM2fO1JNPPqknnnhCMTExmjNnzlkFDAAuhtXVRVNv7aSR01Zp88Es3frhGn19T3dF+nuc9z0ldof2pOdqc3KmNh3M1ObkTO08nKNi+9mTTJwsUZ5qEOSlRsFeahDkqYZBJ7+G+3nIhVIFAECNcUnFafr06VWVQ5K0ePHiCz6XpOuvv17XX399leYAUHd4W101/fYuumHqSu1Jz9WtH67W7L/HK8jbKofDoUOZJ7Q5OUubT5WkhENZyi8qOWs7wd5uiovyV4ndocSjeUo+fkKFxXbtSsvVrrTcs9Z3czErOtDjVJHyUttoPw2Ji+AIFQAATsqwySEAwFkEernp0zFd9LcpK7XvSJ5unrZaUQEe2nwwUxm5RWet7+XmotaRfmoX7a+2px4Rfu4ymf4sPbYSu1IyT+jA0XwlHs3TgYx8HTiapwNH85R8LF9FJXbtPZKnvUfySt8zd3OqXr+xrXzdudAaAABnQ3ECAEnhfh76ZEwXXf/eSu1My9HOtBxJkqvZpBbhPmobdbIgtYv2V5MQ7zJPs7O4nLzWqUGQl6SQM14rsTuUknlCiUdPlqk96bmauSZJi7anaejk5Xp/VMdyXWsFAACqDsUJAE5pEuKtmXd11WerEtU42Ftto/3VKsJX7pazZ/msCBezSdGBnooO9FTPmJPToA/vEKm/f7pe+zLydO3k5XrthnYa2Pr8t1oAAADVy7DpyAHAGbUI89ULQ9vojp6N1LFBQKWXpvOJi/LX3Pt7qmujQOUVlejuz9brtZ93quQck04AAIDqR3ECACcR7G3VZ3d21R09GkmS3v51j8Z8vFZZ+TaDkwEAAIoTADgRi4tZTw9pqTdubCurq1mLdx7RNe/8rp2Hc4yOBgBAnUZxAgAnNKx9lP576p5SiUfzNezd5frxj1SjYwEAUGdRnADASbWO9NPc+3uqR9Mg5ReVaNzMDZo4fzvXPQEAYACKEwA4sUAvN318exf9vXdjSdLUJft02/Q1Op539v2lAABA1aE4AYCTc3Uxa/zgWL01or08LC5atjtDQyb/rq0pWUZHAwCgzqA4AUANcU3bCH1zb3fVD/TUweMndN2UFfp+M9c9AQBQHShOAFCDxIb76vv7eqh3sxAV2Ox69OsEfbPfrKJiu9HRAACo1ShOAFDD+Hu6afptnXVv3yaSpCWHzRr50VqlZJ4wOBkAALUXxQkAaiAXs0n/N7CF3ru5nTxcHNqUnKWr3lqmJbuOGB0NAIBaieIEADVYv9h6eiyuRK0ifHQ836bbpq/R6wt3MWU5AACVjOIEADVcsLv05Z1ddHPX+nI4pLd+2a3RH61RRm6h0dEAAKg1KE4AUAtYLS56aVgbvXFjW3lYXPT7ngxd9dYyrTtwzOhoAADUChQnAKhFhrWP0vf39VCTEC+lZRfqxvdXadrSfXI4OHUPAICKoDgBQC0TE+qj7+/rqWvaRqjE7tCL87br7s/WK+uEzehoAADUWBQnAKiFvKyuevOmdnp+aGu5uZi1YGuarpn8u7amZBkdDQCAGoniBAC1lMlk0q3dGujre+IV6e+hxKP5GvbuCs1ak8SpewAAXCKKEwDUcnFR/vrxgZ7q16Keiort+uc3CXps9h86UVRidDQAAGoMV6MDAACqnr+nm6aN6qT3lu7Vqwt26r8bDmrR9jS1ifRT60g/tY70VZtIP9UP9JTJZDI6LgAATofiBAB1hNls0r19m6p9dIAe+nKj0rIL9fueDP2+J6N0HR93V7WK8P1LofJToyAvmc2UKQBA3UZxAoA6Jr5JkJb93+XaeThHW1KylHAoS1sPZWn74RzlFBRr1b5jWrXvz/s/ebm5qFWEn1qdOioVF+WnJiHeHJkCANQpFCcAqIPcXM1qE+WnNlF+GnFqma3Ert1pudpyKKu0UG1PzVZeUYnWHDimNX+5mW49H6t6xgSrV0ywejQNVj0fd2N2BACAakJxAgBIkiwuZrWM8FXLCF/doGhJUnGJXXuP5JWWqS2HsvTHwSyl5xTqmw2H9M2GQ5KkFmE+6hUTrF4xIerSKFDuFhcjdwUAgEpHcQIAnJeri1nNw3zUPMxH13WMkiQV2Eq0PvG4lu3O0LLdR7Q1JVs7Dudox+EcTVu2X26uZnVpGFh6RCo2zJdrpAAANR7FCQBwSdwtLurR9OQpev8c1EJHcwu1fO9RLdt1RL/vyVBqVkHppBMvz5eCvNzUo+nJEjWgdZh83S1G7wIAAJeM4gQAqJAgb6uuaRuha9pGyOFwaO+R3FNHozK0at9RHc0r0vebU/T95hT9Z8FOPX9taw1sHWZ0bAAALgnFCQBQaUwmk5rW81HTej66vUcjFRXbtTHp5Gl9c/9IUeLRfN392Xpd1SZcE65ppRAfq9GRAQC4KGajAwAAai83V7O6Ng7SYwOaa8FDvTXusiZyMZv0Y0Kqrnh9ib5ef1AOh8PomAAAlIniBACoFu4WF/1jQAt9f18PtYrwVdYJmx6bvVmjp6/VweP5RscDAOCCKE4AgGrVKsJP343roccHtpCbq1lLdx3RlW8s1ccrDshu5+gTAMA5UZwAANXO1cWse/o20fwHe6lLw0DlF5Xome+36oapK7UnPdfoeAAAnIXiBAAwTJMQb80a203PD20tLzcXrUs8rsFvLtM7v+2RrcRudDwAAEpRnAAAhjKbTbq1WwP9/Egf9W0eoqISu15ZsFPXTF6uLYeyjI4HAIAkihMAwElE+nto+m2d9caNbRXgadH21Gxd+85yvTx/hwpsJUbHAwDUcRQnAIDTMJlMGtY+Sgsf6aMhbSNUYnfovSV7deUbSzVrTZKKijl9DwBgDIoTAMDpBHtb9faI9po2qpNCfa1KOpavf36ToD6v/Kbpy/frRBFHoAAA1YviBABwWv1bhurXR/vqyatiVc/HqtSsAj07d5t6/vtXvfPbHmUX2IyOCACoIyhOAACn5mV11Z29GmvZ45fpxWGtFR3ooaN5RXplwU71ePlXvbpgp47mFhodEwBQy1GcAAA1gtXVRSO7NtBvj/bVGze2VUw9b+UUFGvyb3vU89+/6bm525SadcLomACAWoriBACoUVxdzBrWPkoLHuqt927pqDaRfjphK9FHy/er939+0/hv/tCBjDyjYwIAahlXowMAAFAeZrNJA1uHaUCrUC3bnaHJv+3Rmv3H9MWaZH25NllD2kbo772bqFmot1xd+HdCAEDFUJwAADWayWRS72Yh6t0sRGsPHNM7v+3R4p1H9N2mFH23KUWS5OnmIh93V/m4W8746nvqe99zvBYb7is/D4vBewcAcBYUJwBArdG5YaBm3N5FWw5l6d3Fe7Rga5pK7A7lF5Uov6hEadkXP4mEt9VVt/doqDE9G8nf060KUwMAagKKEwCg1mkd6ad3R3aUrcSunIJi5RTYlFNQrOxTX/+6LKfApuwTxcopPL1OsY5kFyglq0Bv/7pHM5YfOFWgGsvPkyNQAFBXUZwAALWWxcWsQC83BXpd2hEju92hn7eladKiXdpxOEdv/bpH05cf0O09G2lMj0YUKACog7haFgCA/3F64ol5D/TSe7d0UIswH+UUFuutX3ar539+1RsLdynrBDffBYC6hOIEAMB5nCxQ4Zr3QC9NGdlBzUN9lFNQrDd/2a2e//5VkxbtUnYBBQoA6gKKEwAAZTCbTRrUJlzzH+yld27uoGahJ2++O2nRbvV8+Ve9uWg3BQoAajmKEwAAF8lsNumquHD99GBvTb65vWLqeSu7oFhvLNqlni//qrd+2a0cChQA1EoUJwAALpHZbNLVcRFa8FBvvT2ivZqeKlCvL9ylXv/5TZ+vTlSJ3WF0TABAJaI4AQBQTmazSUPanixQb41oryYhXsrMt+lf327R8HeX64+DmUZHBABUEooTAAAV5GI26ZpTBeqZIS3lY3XV5oNZuvad5frXtwnKzC8yOiIAoIIoTgAAVBJXF7Nu79FIvzzWR8PaR8rhkD5fnaTLX1uir9Yly87pewBQY1GcAACoZPV83PXGje00a2w3xdTz1rG8Iv3f13/o+qkrtS0l2+h4AIByoDgBAFBFujUO0rwHe+mJwS3k6eai9YnHdfXby/Ts3K1MXw4ANQzFCQCAKmRxMWts7yb65dE+uiouXHaHNH35AfV7bYnmbDwkh4PT9wCgJqA4AQBQDcL9PPTOzR306ZguahzspSM5hXroy00aMW2VdqflGB0PAFAGV6MDAABQl/SKCdH8h3rpg2X79favu7Vq3zENenOZ7ujZSO2j/VVUYldxiUO2ErtsdodsxXYV2+2ylThU9JfvbSV22U6t27VxoIa1jzJ61wCgVqM4AQBQzayuLhp3WVNd0zZCz/2wTQu3pen9pfvKvb1Za5N1NLdId/ZqXIkpAQB/RXECAMAg0YGemjaqk37dkabpyw+owFYii4tZri5mubmY5Go2y+JqlsVsOrX85Fc3V7NcTy07nFWgL9cl64UftyvY26qh7SON3i0AqJUoTgAAGOzyFqG6vEVoud7rcDjkZXXVR8v367HZmxXo5abezUIqOSEAgMkhAACowUwmk568KlbXtI1Qsd2huz9br83JmUbHAoBah+IEAEANZzab9Or1bdWzabDyi0p0+4y12p+RZ3QsAKhVKE4AANQCbq5mvXdrR7WO9NWxvCKN+mi10nMKjI4FALUGxQkAgFrC2+qq6bd1UYMgTyUfO6HbPlqrnAKb0bEAoFagOAEAUIuE+Fj1yR1dFOztpm2p2fr7p+tVWFxidCwAqPEoTgAA1DINgrw04/Yu8nJz0Yq9R/XIV5tltzuMjgUANRrFCQCAWqh1pJ+m3tpJFheTfvwjVc/O3SqHg/IEAOVlaHGaMmWK4uLi5OvrK19fX8XHx2v+/PnnXX/GjBkymUxnPNzd3asxMQAANUfPmGC9dkM7SdLHKxP17uK9xgYCgBrM0BvgRkVF6eWXX1ZMTIwcDoc+/vhjXXvttdq4caNatWp1zvf4+vpq586dpc9NJlN1xQUAoMa5pm2EjuYW6tm52/TKgp0K8bbqhs7RRscCgBrH0OI0ZMiQM56/+OKLmjJlilatWnXe4mQymRQWFlYd8QAAqBVu79FI6TmFmrJ4r8Z/m6Agbzf1bhpodCwAqFEMLU5/VVJSotmzZysvL0/x8fHnXS83N1cNGjSQ3W5Xhw4d9NJLL523ZElSYWGhCgsLS59nZ2dLkmw2m2y2ypmi9fR2Kmt7qFsYP6gIxg8u1sOXN1Za1gl9szFF42Zu0Ee3tJPE2EH58LMHFeFM4+dSMpgcBl8pmpCQoPj4eBUUFMjb21szZ87U4MGDz7nuypUrtXv3bsXFxSkrK0uvvvqqli5dqq1btyoqKuqc75kwYYKeffbZs5bPnDlTnp6elbovAAA4sxK79MFOs7ZlmuXp6tCDrUoUxv8KAdRh+fn5uvnmm5WVlSVfX98Lrmt4cSoqKlJSUpKysrL09ddf64MPPtCSJUvUsmXLMt9rs9kUGxurESNG6Pnnnz/nOuc64hQdHa2MjIwy/3Auls1m08KFC9W/f39ZLJZK2SbqDsYPKoLxg0uVX1SsUdPXa/PBLPm7OXRbzybq0CBArSP85OPuNCeiwMnxswcV4UzjJzs7W8HBwRdVnAz/Cenm5qamTZtKkjp27Ki1a9fqzTff1NSpU8t8r8ViUfv27bVnz57zrmO1WmW1Ws/53sr+i6qKbaLuYPygIhg/uFh+Fotm3N5F101ZoX0ZeZr06z5JkskkNQ3xVrtof7Wr76920f5qHuojVxfuXILz42cPKsIZxs+lfL7hxel/2e32M44QXUhJSYkSEhLOe2ofAAA4W4CXm2bd1VkvzfxFRT4R2nwwW4cyT2h3eq52p+dq9vqDkiR3i1ltIv1OlqnoALWr768IP3dmtAVQJxlanMaPH69Bgwapfv36ysnJ0cyZM7V48WItWLBAkjRq1ChFRkZq4sSJkqTnnntO3bp1U9OmTZWZmalXXnlFiYmJuvPOO43cDQAAapwATzddFuHQ4MFtZbFYdCSnUJuTM7Xp1GNzcqZyCou19sBxrT1wXNJ+SVKIj1Vto/zVv2U9DWkbIU83p/s3WACoEob+tEtPT9eoUaOUmpoqPz8/xcXFacGCBerfv78kKSkpSWbzn6cIHD9+XHfddZcOHz6sgIAAdezYUStWrLio66EAAMD5hfhYdUXLUF3RMlSSZLc7tC8jVxuT/ixTOw7n6EhOoRZtT9Oi7Wl64YftGt4hUjd3baDmYT4G7wEAVC1Di9OHH354wdcXL158xvM33nhDb7zxRhUmAgAAkmQ2m9S0no+a1vPR9Z1O3jD3RFGJtqZkafX+Y/pqXbISj+br45WJ+nhlojo3DNDNXetrUOtwuVtcDE4PAJWP4+sAAOCieLi5qFPDQHVqGKh7+jTR8r0Z+nxVkhZuTys9pe/Zudt0fccojehSX41DvI2ODACVhuIEAAAumdlsUq+YEPWKCVFadoG+WpusL9YkKSWrQNOW7de0ZfvVvUmQRnZtoP4tQ+Xmyux8AGo2ihMAAKiQUF933d8vRvde1lSLd6br89VJ+m1nulbsPaoVe48q2NtNN3SK1ogu9RUdyB13AdRMFCcAAFApXMwm9YsNVb/YUB08nq8v1yZr1tpkHckp1LuL92rKkr2Ki/RTgJebfN0t8nF3lc+pr75/+f7Pr6e+t7rKbGYKdADGojgBAIBKFxXgqUevbK4H+sXol+1p+nx1kpbtztDmg1nl2l6gl5tGxzfU3X0by+rK5BMAqh/FCQAAVBmLi1kDW4drYOtwJR3N19aULOUUFCu7wKacguJTj1PfF578mn3iz9eKSuySpGN5RXpj0S59v/mQJg6PU5dGgQbvGYC6huIEAACqRf0gT9UPurRrnApsJcopKNaKvRl6/odt2nskTzdMXambOkdr/KBY+XlaqigtAJyJKW4AAIDTcre4KMTHqmvbReqXR/pqRJeT95SatTZZ/V5fou83p8jhcBicEkBdQHECAAA1gp+nRROHx+mrv8erSYiXMnIL9cAXG3X7jLVKPpZvdDwAtRzFCQAA1ChdGgVq3oO99PAVzeTmYtbinUd05RtL9f7SvSo+dU0UAFQ2ihMAAKhxrK4uevCKGM1/qJe6NgrUCVuJXpq3Q9e+s1x/HMw0Oh6AWojiBAAAaqwmId6aNbab/nNdnPw8LNqakq2h7yzXs3O3Krew2Oh4AGoRihMAAKjRTCaTbugcrV8e7aNr20XI7pCmLz+gK19fokXb0oyOB6CWYDpyAABQKwR7W/XmTe01vEOUnpyToORjJ3TnJ+vUJtJPLcN91SLcR83DfNQizFeBXm5GxwVQw1CcAABArdKnWYh+fqiPJv2ySx8s26+EQ1lKOJR1xjohPla1CPNRizAfNQ/zVYswHzWt5y13i4tBqQE4O4oTAACodTzcXDR+UKxGxTfUxqTj2nk4RzsO52jn4RwlHcvXkZxCHckp1LLdGaXvMZukhsFepwrVyTLVOtJP4X7uMplMBu4NAGdAcQIAALVWpL+HIv09dHXcn8vyCou1K+3PIrXjcLZ2Hs7R8Xyb9h3J074jeZqXcLh0/SAvN7WK9FPrCF+1ifRT60g/RQV4UKaAOobiBAAA6hQvq6va1w9Q+/oBpcscDoeO5BRqx6kiteNwjralZGtPeq6O5hVp6a4jWrrrSOn6fh4WtY70VetIP7WOOFmmGgR6ymymTAG1FcUJAADUeSaTSfV83VXP1129m4WULi+wlWjn4RxtScnSlkNZ2nIoWzsOZyvrhE3L9xzV8j1HS9f1sbqq5amjUtd3ilbzMB8jdgVAFaE4AQAAnIe7xUVto/3VNtq/dFlRsV270nJOFqmULCUcytb21GzlFBZr9f5jWr3/mKavOKBR8Q30cP9m8nW3GLcDACoNxQkAAOASuLmaT56iF+lXusxWYtee9FxtOZSln7elaeG2NE1ffkBzN6dq/KAWGt4hkmuigBqOG+ACAABUkMXFrNhwX13fKVrTRnXSJ3d0UeNgL2XkFurR2Zt1/XsrtTUlq+wNAXBaFCcAAIBK1rtZiH56qLceH9hCnm4uWpd4XEPe/l1Pf7dFWfk2o+MBKAeKEwAAQBVwczXrnr5N9MujfXRVXLjsDumTlYm6/LXF+mptsux2h9ERAVwCihMAAEAVCvfz0Ds3d9Dnd3ZV03reOppXpP/77x8aPmWFEg5y+h5QU1CcAAAAqkGPpsGa/2Av/WtwrLzcXLQpOVPXvPO7nvg2QcfzioyOB6AMFCcAAIBqYnEx667ejfXrY311bbsIORzSzNVJuuy1xZq5OkklnL4HOC2KEwAAQDUL9XXXmze116yx3dQ81EeZ+TY98W2Chr6zXOsOHDM6HoBzoDgBAAAYpFvjIP3wQE89fXVL+VhdlXAoS397b6Xu/2KjDmWeMDoegL+gOAEAABjI4mLWHT0b6dfH+uqmztEymaS5m1N0+auL9frPO5VfVGx0RACiOAEAADiFEB+rXr4uTnPv66kujQJVWGzXW7/u0eWvLtG3Gw8yfTlgMIoTAACAE2kd6acvx3bTlJEdFBXgocPZBXr4y80aPmWFNiQdNzoeUGdRnAAAAJyMyWTSoDbhWvRIH/1jQPPS6cuHv7tCD83aqNQsrn8CqhvFCQAAwEm5W1w07rKm+u2xvrq+Y5RMJmnOphRd9upiTVq0SyeKSoyOCNQZFCcAAAAnV8/XXa9c31bfj+upzg0DVGCza9Ki3er32mJ9t+mQHA6ufwKqGsUJAACghmgT5aev/h6vyTe3V6S/h1KyCvTgrE26bsoKLdl1hAIFVCGKEwAAQA1iMpl0dVyEfnm0jx7t30weFhdtSMrU6I/WaOCkZfp6/UEVFduNjgnUOhQnAACAGsjd4qL7+8Vo8T/6akzPRvJyc9HOtBw9Nnuzev3nV01ZvFdZJ2xGxwRqDYoTAABADRbq666nrm6pFeP76Z+DWijU16q07EL9+6cd6j7xFz03d5sOHs83OiZQ41GcAAAAagE/D4vu7tNEy/7vcr12fVu1CPNRXlGJPlq+X31eWaz7v9iohINZRscEaixXowMAAACg8ri5mnVdxygN7xCpZbszNG3ZPi3bnaG5m1M0d3OKujUO1NjejdW3WT2ZzSaj4wI1BsUJAACgFjKZTOrdLES9m4VoW0q2Pli2T99vTtGqfce0at8xNa3nrbt6NVLPmBAFeFrkYXGRyUSRAs6H4gQAAFDLtYzw1es3ttM/BjbXjOUHNHN1kvak5+rx/yaUruPmalaAp0UBnm7y8zj5NcDLIn9PNwV4nv568nsvi0mF3HsXdQzFCQAAoI4I9/PQ+MGxuu/ypvpybbJmrk5S8vF82UocKiq2Ky27UGnZhRe1LavZRZlBiRrTq4lcOOUPdQDFCQAAoI7xcbfozl6NdWevxnI4HMovKtHx/CJl5tt0PL9Ix/Ntyswv0vE826nlRco8YStdfjS3SLmFxXpp/k79kHBYE4e3UasIP6N3C6hSFCcAAIA6zGQyycvqKi+rq6ICLu49hYVFevrjnzQvxao/DmbpmsnLdWevRnqoXzN5uLlUbWDAIExHDgAAgEtiNpvUPdShnx7ooavahKvE7tDUJfs0YNJSLdt9xOh4QJWgOAEAAKBc6vlY9c7IDvpgVCeF+7kr6Vi+bv1wjR75cpOO5l7ctVJATUFxAgAAQIVc0TJUCx/po9u6N5TJJH2z8ZCueH2J/rv+oBwOh9HxgEpBcQIAAECFeVtdNeGaVvrmnu5qEeaj4/k2PTp7s279cI0Sj+YZHQ+oMIoTAAAAKk37+gGae39P/d/A5rK6mvX7ngxd+cZSTVm8V7YSu9HxgHKjOAEAAKBSWVzMurdvUy14qLd6NA1SYbFd//5ph66ZvFybkzONjgeUC9ORAwAAoEo0DPbSZ2O66r8bDumFH7dpe2q2rn1nuXzdXeXrYZGvu0W+Hq6nvlrk4/7n9+daJ8Lfg5vtwjAUJwAAAFQZk8mkv3WM0mXNQ/T8D9s0Z1OKsguKlV1QLOnEJW2rQZCnXh4ep/gmQVUTFrgAihMAAACqXJC3VZNuaq8J17TS0bwiZZ+wnSxQJ2zKLrAp+0Txqa//u/zk86x8mxKP5mvEtFUaFd9Ajw9sIS8rv8qi+jDaAAAAUG38Pd3k7+l2ye/LKbDppXk79MWaJH2yMlG/7UzXv6+LU/cmwVWQEjgbk0MAAADA6fm4WzRxeBt9NqarIv09lHzshG6etlpPzdmivMJio+OhDqA4AQAAoMboGROsBQ/31siu9SVJn65K1IBJS7ViT4bByVDbUZwAAABQo3hbXfXisDb6/M6TR58OHj+hmz9YrSfnJCiXo0+oIhQnAAAA1Eg9mp48+nRLt5NHnz5blaQBbyzVco4+oQpQnAAAAFBjeVtd9cLQNpp5Z1dFBXjoUOYJjfxgtZ74lqNPqFwUJwAAANR43ZsGa8FDvXVrtwaSpJmrTx59+n03R59QOShOAAAAqBW8rK56fmhrzbyrq6IDTx59uuXD1Xr86z+0Oy3H6Hio4ShOAAAAqFW6NwnWTw/21uj4k0efvlyXrP5vLNXVby/TB8v2KT27wOCEqIm4AS4AAABqHS+rq569trWuiovQtGX7tHhnurYcytaWQ9l6ad529WgarOEdInVlyzB5WfmVGGVjlAAAAKDW6tIoUF0aBep4XpF+SEjVnI2HtD7xuJbtztCy3RnysGzRgFahGtYhSj2aBMnVhROycG4UJwAAANR6AV5uurVbA93arYESj+ZpzsYUzdl0SPsz8jRnU4rmbEpRsLdV17SN0PAOkWoV4SuTyWR0bDgRihMAAADqlAZBXnrwihg90K+pNh/M0rcbDmruH6nKyC3UR8v366Pl+9W0nreGtY/UdR2iFObnbnRkOAGKEwAAAOokk8mkdtH+ahftryevbqllu4/omw2HtHBbmvak5+qVBTv1xsJdGtA6TKPjG6pzwwCOQtVhFCcAAADUeRYXsy5vEarLW4Qqp8Cmn7Yc1ux1B7XmwDH9+EeqfvwjVbHhvhod30DXtouUh5uL0ZFRzbj6DQAAAPgLH3eLru8Ura/ujtf8B3tpRJdouVvM2p6arX9+k6BuE3/RS/O2K+lovtFRUY0MLU5TpkxRXFycfH195evrq/j4eM2fP/+C75k9e7ZatGghd3d3tWnTRvPmzaumtAAAAKhrYsN9NXF4nFaPv0L/Ghyr6EAPZZ2w6f2l+9Tn1d9058drtXTXEdntDqOjoooZWpyioqL08ssva/369Vq3bp0uv/xyXXvttdq6des511+xYoVGjBihMWPGaOPGjRo6dKiGDh2qLVu2VHNyAAAA1CV+nhbd1buxFj92mT4c3Um9m4XI4ZAWbU/XqI/W6Io3lmjG8v3KKbAZHRVVxNDiNGTIEA0ePFgxMTFq1qyZXnzxRXl7e2vVqlXnXP/NN9/UwIED9Y9//EOxsbF6/vnn1aFDB02ePLmakwMAAKAucjGb1C82VJ/c0UW/PNpHt3VvKG+rq/YdydOEudvU7aVf9PR3W7QrLcfoqKhkTjM5RElJiWbPnq28vDzFx8efc52VK1fqkUceOWPZgAEDNGfOnPNut7CwUIWFhaXPs7OzJUk2m002W+X8i8Dp7VTW9lC3MH5QEYwflBdjBxXB+Dmpvr9V/xrUTA9e3ljfbUrRp6uTtfdInj5ZmahPViaqRZiPrm4TpqvjwhTp72F0XKfhTOPnUjKYHA6HoSdkJiQkKD4+XgUFBfL29tbMmTM1ePDgc67r5uamjz/+WCNGjChd9u677+rZZ59VWlraOd8zYcIEPfvss2ctnzlzpjw9PStnJwAAAFDnORzSrmyTlqWatC3TpBLHn1OXN/JxqEOQXe2CHPJ1MzAkzpCfn6+bb75ZWVlZ8vX1veC6hh9xat68uTZt2qSsrCx9/fXXGj16tJYsWaKWLVtWyvbHjx9/xlGq7OxsRUdH68orryzzD+di2Ww2LVy4UP3795fFYqmUbaLuYPygIhg/KC/GDiqC8XN+V0l6WFJmvk0LtqXphz9StfrAce3PMWl/jou+TZTiGwfp6rgwXRlbT74ede/Pz5nGz+mz0S6G4cXJzc1NTZs2lSR17NhRa9eu1ZtvvqmpU6eetW5YWNhZR5bS0tIUFhZ23u1brVZZrdazllsslkr/i6qKbaLuYPygIhg/KC/GDiqC8XN+IX4W3RLfSLfEN1JadoF++CNV329O0ebkTC3fe1TL9x7VM99vV9/mIbqmXYT6tQitc/eGcobxcymfb3hx+l92u/2Ma5L+Kj4+Xr/88oseeuih0mULFy487zVRAAAAgNFCfd01pmcjjenZSIlH8zR3c4q+25Si3em5+nlbmn7eliYvNxf1bxmqa9pFqHuTYLlb6laJqgkMLU7jx4/XoEGDVL9+feXk5GjmzJlavHixFixYIEkaNWqUIiMjNXHiREnSgw8+qD59+ui1117TVVddpVmzZmndunV6//33jdwNAAAA4KI0CPLSfZfHaNxlTbXjcI6+35yiuZtTdPD4Cc3ZlKI5m1JkdTWrS6NA9WkWol4xIWoW6i2TyVT2xlGlDC1O6enpGjVqlFJTU+Xn56e4uDgtWLBA/fv3lyQlJSXJbP5zxvTu3btr5syZevLJJ/XEE08oJiZGc+bMUevWrY3aBQAAAOCSmUwmxYb7KjbcV/83oLk2JGVq7uYULdh6WKlZBVq2O0PLdmdI2q5QX6t6xYSod7MQ9WwarEAvZpcwgqHF6cMPP7zg64sXLz5r2fXXX6/rr7++ihIBAAAA1ctkMqljgwB1bBCgZ4a01J70XC3dnaGlu45o9f6jSssu1NfrD+rr9QdlMkltIv3U+1SRal/fXxYXQ2/NWmc43TVOAAAAQF1lMpkUE+qjmFAfjenZSAW2Eq07cFxLdx/R0l1HtONwjv44mKU/DmZp8m975G11VXyTIPWOCVb/lmEK83M3ehdqLYoTAAAA4KTcLS7qGROsnjHBemJwrNKyT57Gt3TXEf2+J0PH8oq0cFuaFm5L04S529SvRT2N7NZAvZoGy2zmuqjKRHECAAAAaohQX3f9rWOU/tYxSna7Q1tTsrV09xH9uiNd6xOPl87SVz/QUyO61Nf1naIU7H32rXlw6ShOAAAAQA1kNpvUJspPbaL8NO6yptqdlqPPVyfpvxsOKulYvv790w69vnCnBrYO18iu9dW1USCz81UAxQkAAACoBWJCfTThmlZ6fGALzf0jRZ+vTtLm5JOz9c3dnKImIV4a2bWBrusQJT9Pblx8qShOAAAAQC3i4eaiGzpF64ZO0dpyKEufr07Sd5sOae+RPD33wzb9+6cdGtI2QiO71le7aH+OQl0kihMAAABQS7WO9NPE4W30xOAW+m7TyaNQ21OzS6c3jw331ciu9TW8Q6Q83agGF8Kk7wAAAEAt5+Nu0S3dGmjeAz31zb3d9beOUbK6mrU9NVtPztmibi/9opfmbdfB4/lGR3Va1EoAAACgjjCZTOpQP0Ad6gfoqata6usNB/XpygM6cDRf7y/dpw+W7dOVLcN0e4+G6sJkEmegOAEAAAB1kJ+nRWN6NtLt3Rvqt53pmr78gH7fk6Gfth7WT1sPq1WEr27r3lBD2kbI3eJidFzDUZwAAACAOsxsNqlfbKj6xYZqV1qOpi8/oG83HtTWlGz94+s/9PL8HRrZtb5u6dZA9XzdjY5rGK5xAgAAACBJahbqo4nD22jlP/vp8YEtFOHnrqN5RXrr1z3q8e9f9dCsjdqcnGl0TENwxAkAAADAGQK83HRP3ya6q1cjLdiapunL92td4nHN2ZSiOZtS1KG+v27v0UiD24TLxVw3roOiOAEAAAA4J1cXs66KC9dVceFKOJil6cv3a+4fKdqQlKkNSRv13aYUvTOyvayutf8aKE7VAwAAAFCmNlF+ev3Gdlr+z8v1QL8YubmatWh7msZ+sl4FthKj41U5ihMAAACAi1bPx12P9G+m6bd1lofFRUt2HdEdM9Yqv6jY6GhViuIEAAAA4JL1aBqsj+/oIm+rq1bsPapRH65RToHN6FhVhuIEAAAAoFy6NArUp2O6yMfdVesSj+uWD9coK792lieKEwAAAIBya18/QF/c1U0BnhZtTs7UiGmrdCyvyOhYlY7iBAAAAKBCWkf6adbYeAV7u2lbarZuen+l0nMKjI5VqShOAAAAACqseZiPZo2NV6ivVbvScnXT1FVKzTphdKxKQ3ECAAAAUCma1vPWV3+PV6S/h/Zl5OmGqSuVfCzf6FiVguIEAAAAoNI0CPLSl3/vpgZBnko+dkI3Tl2pAxl5RseqMIoTAAAAgEoVFeCpL8fGq0mIl1KyCnTD1JXak55jdKwKoTgBAAAAqHRhfu6aNTZeLcJ8lJ5TqBunrtL21GyjY5UbxQkAAABAlQjxseqLu7qpdaSvjuYVacS0VdpyqGaWJ4oTAAAAgCoT4OWmz+/spnbR/srMt+nW6eu0vwaetUdxAgAAAFCl/Dws+uzOrurSMFC5hcWass1Fe4/UrAkjKE4AAAAAqpy31VUz7uis7k0C1S7IoUZBnkZHuiQUJwAAAADVwtPNVe+PbK+bmthlNpuMjnNJKE4AAAAAqo3V4qIa1pkkUZwAAAAAoEwUJwAAAAAoA8UJAAAAAMpAcQIAAACAMlCcAAAAAKAMFCcAAAAAKAPFCQAAAADKQHECAAAAgDJQnAAAAACgDBQnAAAAACgDxQkAAAAAykBxAgAAAIAyUJwAAAAAoAwUJwAAAAAoA8UJAAAAAMpAcQIAAACAMlCcAAAAAKAMrkYHqG4Oh0OSlJ2dXWnbtNlsys/PV3Z2tiwWS6VtF3UD4wcVwfhBeTF2UBGMH1SEM42f053gdEe4kDpXnHJyciRJ0dHRBicBAAAA4AxycnLk5+d3wXVMjoupV7WI3W5XSkqKfHx8ZDKZKmWb2dnZio6OVnJysnx9fStlm6g7GD+oCMYPyouxg4pg/KAinGn8OBwO5eTkKCIiQmbzha9iqnNHnMxms6Kioqpk276+vob/5aPmYvygIhg/KC/GDiqC8YOKcJbxU9aRptOYHAIAAAAAykBxAgAAAIAyUJwqgdVq1TPPPCOr1Wp0FNRAjB9UBOMH5cXYQUUwflARNXX81LnJIQAAAADgUnHECQAAAADKQHECAAAAgDJQnAAAAACgDBQnAAAAACgDxakSvPPOO2rYsKHc3d3VtWtXrVmzxuhIcEJLly7VkCFDFBERIZPJpDlz5pzxusPh0NNPP63w8HB5eHjoiiuu0O7du40JC6cyceJEde7cWT4+PqpXr56GDh2qnTt3nrFOQUGBxo0bp6CgIHl7e+u6665TWlqaQYnhTKZMmaK4uLjSG03Gx8dr/vz5pa8zdnCxXn75ZZlMJj300EOlyxg/OJ8JEybIZDKd8WjRokXp6zVx7FCcKujLL7/UI488omeeeUYbNmxQ27ZtNWDAAKWnpxsdDU4mLy9Pbdu21TvvvHPO1//zn//orbfe0nvvvafVq1fLy8tLAwYMUEFBQTUnhbNZsmSJxo0bp1WrVmnhwoWy2Wy68sorlZeXV7rOww8/rLlz52r27NlasmSJUlJSNHz4cANTw1lERUXp5Zdf1vr167Vu3Tpdfvnluvbaa7V161ZJjB1cnLVr12rq1KmKi4s7YznjBxfSqlUrpaamlj5+//330tdq5NhxoEK6dOniGDduXOnzkpISR0REhGPixIkGpoKzk+T49ttvS5/b7XZHWFiY45VXXildlpmZ6bBarY4vvvjCgIRwZunp6Q5JjiVLljgcjpNjxWKxOGbPnl26zvbt2x2SHCtXrjQqJpxYQECA44MPPmDs4KLk5OQ4YmJiHAsXLnT06dPH8eCDDzocDn724MKeeeYZR9u2bc/5Wk0dOxxxqoCioiKtX79eV1xxRekys9msK664QitXrjQwGWqa/fv36/Dhw2eMJT8/P3Xt2pWxhLNkZWVJkgIDAyVJ69evl81mO2P8tGjRQvXr12f84AwlJSWaNWuW8vLyFB8fz9jBRRk3bpyuuuqqM8aJxM8elG337t2KiIhQ48aNNXLkSCUlJUmquWPH1egANVlGRoZKSkoUGhp6xvLQ0FDt2LHDoFSoiQ4fPixJ5xxLp18DJMlut+uhhx5Sjx491Lp1a0knx4+bm5v8/f3PWJfxg9MSEhIUHx+vgoICeXt769tvv1XLli21adMmxg4uaNasWdqwYYPWrl171mv87MGFdO3aVTNmzFDz5s2VmpqqZ599Vr169dKWLVtq7NihOAFADTJu3Dht2bLljPPEgbI0b95cmzZtUlZWlr7++muNHj1aS5YsMToWnFxycrIefPBBLVy4UO7u7kbHQQ0zaNCg0u/j4uLUtWtXNWjQQF999ZU8PDwMTFZ+nKpXAcHBwXJxcTlrBpC0tDSFhYUZlAo10enxwljChdx333364Ycf9NtvvykqKqp0eVhYmIqKipSZmXnG+owfnObm5qamTZuqY8eOmjhxotq2bas333yTsYMLWr9+vdLT09WhQwe5urrK1dVVS5Ys0VtvvSVXV1eFhoYyfnDR/P391axZM+3Zs6fG/uyhOFWAm5ubOnbsqF9++aV0md1u1y+//KL4+HgDk6GmadSokcLCws4YS9nZ2Vq9ejVjCXI4HLrvvvv07bff6tdff1WjRo3OeL1jx46yWCxnjJ+dO3cqKSmJ8YNzstvtKiwsZOzggvr166eEhARt2rSp9NGpUyeNHDmy9HvGDy5Wbm6u9u7dq/Dw8Br7s4dT9SrokUce0ejRo9WpUyd16dJFkyZNUl5enm6//Xajo8HJ5Obmas+ePaXP9+/fr02bNikwMFD169fXQw89pBdeeEExMTFq1KiRnnrqKUVERGjo0KHGhYZTGDdunGbOnKnvvvtOPj4+ped/+/n5ycPDQ35+fhozZoweeeQRBQYGytfXV/fff7/i4+PVrVs3g9PDaOPHj9egQYNUv3595eTkaObMmVq8eLEWLFjA2MEF+fj4lF5LeZqXl5eCgoJKlzN+cD6PPfaYhgwZogYNGiglJUXPPPOMXFxcNGLEiJr7s8foaf1qg7fffttRv359h5ubm6NLly6OVatWGR0JTui3335zSDrrMXr0aIfDcXJK8qeeesoRGhrqsFqtjn79+jl27txpbGg4hXONG0mO6dOnl65z4sQJx7333usICAhweHp6OoYNG+ZITU01LjScxh133OFo0KCBw83NzRESEuLo16+f4+effy59nbGDS/HX6cgdDsYPzu/GG290hIeHO9zc3ByRkZGOG2+80bFnz57S12vi2DE5HA6HQZ0NAAAAAGoErnECAAAAgDJQnAAAAACgDBQnAAAAACgDxQkAAAAAykBxAgAAAIAyUJwAAAAAoAwUJwAAAAAoA8UJAAAAAMpAcQIA4BKYTCbNmTPH6BgAgGpGcQIA1Bi33XabTCbTWY+BAwcaHQ0AUMu5Gh0AAIBLMXDgQE2fPv2MZVar1aA0AIC6giNOAIAaxWq1Kiws7IxHQECApJOn0U2ZMkWDBg2Sh4eHGjdurK+//vqM9yckJOjyyy+Xh4eHgoKCNHbsWOXm5p6xzkcffaRWrVrJarUqPDxc99133xmvZ2RkaNiwYfL09FRMTIy+//77qt1pAIDhKE4AgFrlqaee0nXXXafNmzdr5MiRuummm7R9+3ZJUl5engYMGKCAgACtXbtWs2fP1qJFi84oRlOmTNG4ceM0duxYJSQk6Pvvv1fTpk3P+Ixnn31WN9xwg/744w8NHjxYI0eO1LFjx6p1PwEA1cvkcDgcRocAAOBi3Hbbbfrss8/k7u5+xvInnnhCTzzxhEwmk+6++25NmTKl9LVu3bqpQ4cOevfddzVt2jQ9/vjjSk5OlpeXlyRp3rx5GjJkiFJSUhQaGqrIyEjdfvvteuGFF86ZwWQy6cknn9Tzzz8v6WQZ8/b21vz587nWCgBqMa5xAgDUKJdddtkZxUiSAgMDS7+Pj48/47X4+Hht2rRJkrR9+3a1bdu2tDRJUo8ePWS327Vz506ZTCalpKSoX79+F8wQFxdX+r2Xl5d8fX2Vnp5e3l0CANQAFCcAQI3i5eV11qlzlcXDw+Oi1rNYLGc8N5lMstvtVREJAOAkuMYJAFCrrFq16qznsbGxkqTY2Fht3rxZeXl5pa8vX75cZrNZzZs3l4+Pjxo2bKhffvmlWjMDAJwfR5wAADVKYWGhDh8+fMYyV1dXBQcHS5Jmz56tTp06qWfPnvr888+1Zs0affjhh5KkkSNH6plnntHo0aM1YcIEHTlyRPfff79uvfVWhYaGSpImTJigu+++W/Xq1dOgQYOUk5Oj5cuX6/7776/eHQUAOBWKEwCgRvnpp58UHh5+xrLmzZtrx44dkk7OeDdr1izde++9Cg8P1xdffKGWLVtKkjw9PbVgwQI9+OCD6ty5szw9PXXdddfp9ddfL93W6NGjVVBQoDfeeEOPPfaYgoOD9be//a36dhAA4JSYVQ8AUGuYTCZ9++23Gjp0qNFRAAC1DNc4AQAAAEAZKE4AAAAAUAaucQIA1BqcfQ4AqCoccQIAAACAMlCcAAAAAKAMFCcAAAAAKAPFCQAAAADKQHECAAAAgDJQnAAAAACgDBQnAAAAACgDxQkAAAAAyvD/2btxpbwxkcYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Sample from trained weight:\n",
      "\n",
      "Input (German): . auto einem in jemandem mit <unk> typ ein\n",
      "Target (English): a guy is <unk> with someone in a car .\n",
      "Model Output (English): man in a <unk> <unk> .\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/lv/fs2j5vr90zz71f7mbvytr3s80000gn/T/ipykernel_98069/1523997237.py:402: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  model.load_state_dict(torch.load('best_model.pt'))\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from datasets import load_dataset\n",
    "import os\n",
    "import json\n",
    "import random\n",
    "import spacy\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "\n",
    "\"\"\"Implements: Sequence to Sequence Learning with Neural Networks\"\"\"\n",
    "\n",
    "# Hyperparameters\n",
    "num_layers = 2\n",
    "embedding_dim = 256\n",
    "hidden_dim = 512\n",
    "batch_size = 64\n",
    "dropout = 0.5\n",
    "learning_rate = 0.001\n",
    "clip = 1.0\n",
    "num_epochs = 50\n",
    "\n",
    "# ==============================\n",
    "# Data Preprocessing \n",
    "# ==============================\n",
    "def prep_data():\n",
    "    if not os.path.exists('data/train.jsonl') or not os.path.exists('data/valid.jsonl'):\n",
    "        os.makedirs('data', exist_ok=True)\n",
    "        dataset = load_dataset(\"bentrevett/multi30k\", split=\"train\")\n",
    "        \n",
    "        data = [{\"de\": item[\"de\"], \"en\": item[\"en\"]} for item in dataset]\n",
    "        train_data = data[:4000]\n",
    "        val_data = data[4000:5000]\n",
    "        \n",
    "        def save_data(data, filename):\n",
    "            with open(filename, 'w', encoding='utf-8') as f:\n",
    "                for item in data:\n",
    "                    json.dump(item, f)\n",
    "                    f.write('\\n')\n",
    "        \n",
    "        save_data(train_data, \"data/train.jsonl\")\n",
    "        save_data(val_data, \"data/valid.jsonl\")\n",
    "\n",
    "# ==============================\n",
    "# Tokenize\n",
    "# ==============================\n",
    "\n",
    "# Load spacy tokenizers for German and English (first download)\n",
    "# !python3 -m spacy download en_core_web_sm\n",
    "# !python3 -m spacy download de_core_news_sm\n",
    "spacy_de = spacy.load(\"de_core_news_sm\") # German\n",
    "spacy_en = spacy.load(\"en_core_web_sm\") # English\n",
    "\n",
    "def tokenize_de(text):\n",
    "    \"\"\"Given a string of German text tokenize and return a reversed list of strings (tokens)\"\"\"\n",
    "    return [tok.text.lower() for tok in spacy_de.tokenizer(text)][::-1]\n",
    "\n",
    "def tokenize_en(text):\n",
    "    \"\"\"Tokenizes English text\"\"\"\n",
    "    return [tok.text.lower() for tok in spacy_en.tokenizer(text)]\n",
    "\n",
    "class Vocab(Dataset):\n",
    "    def __init__(self, file_path: str, src_tokenizer, tgt_tokenizer, src_vocab=None, tgt_vocab=None):\n",
    "        self.data = []\n",
    "        self.src_words = Counter()\n",
    "        self.tgt_words = Counter()\n",
    "        \n",
    "        # First pass: collect word frequencies\n",
    "        # print(\"First pass: collecting word frequencies...\")\n",
    "        with open(file_path, 'r', encoding='utf-8') as f:\n",
    "            for line in f:\n",
    "                sample = json.loads(line)\n",
    "                src_tokens = src_tokenizer(sample['de'].strip())\n",
    "                tgt_tokens = tgt_tokenizer(sample['en'].strip())\n",
    "                self.src_words.update(src_tokens)\n",
    "                self.tgt_words.update(tgt_tokens)\n",
    "        \n",
    "        # Build vocab\n",
    "        if src_vocab is None:\n",
    "            self.src_vocab = self._build_vocab(self.src_words)\n",
    "        else:\n",
    "            self.src_vocab = src_vocab\n",
    "            \n",
    "        if tgt_vocab is None:\n",
    "            self.tgt_vocab = self._build_vocab(self.tgt_words)\n",
    "        else:\n",
    "            self.tgt_vocab = tgt_vocab\n",
    "            \n",
    "        self.src_idx2word = {idx: word for word, idx in self.src_vocab.items()}\n",
    "        self.tgt_idx2word = {idx: word for word, idx in self.tgt_vocab.items()}\n",
    "        \n",
    "        # Second pass: create dataset\n",
    "        # print(\"Second pass: creating dataset...\")\n",
    "        with open(file_path, 'r', encoding='utf-8') as f:\n",
    "            for line in f:\n",
    "                sample = json.loads(line)\n",
    "                src_tokens = src_tokenizer(sample['de'].strip())\n",
    "                tgt_tokens = tgt_tokenizer(sample['en'].strip())\n",
    "                \n",
    "                if len(src_tokens) > 0 and len(tgt_tokens) > 0:\n",
    "                    src_indices = self._tokens_to_indices(src_tokens, self.src_vocab)\n",
    "                    tgt_indices = self._tokens_to_indices(tgt_tokens, self.tgt_vocab)\n",
    "                    self.data.append((src_indices, tgt_indices))\n",
    "    \n",
    "    def _build_vocab(self, counter, min_freq=2):\n",
    "        vocab = {'<pad>': 0, '<sos>': 1, '<eos>': 2, '<unk>': 3}\n",
    "        for word, count in counter.items():\n",
    "            if count >= min_freq:\n",
    "                vocab[word] = len(vocab)\n",
    "        return vocab\n",
    "    \n",
    "    def _tokens_to_indices(self, tokens, vocab):\n",
    "        return [vocab.get(token, vocab['<unk>']) for token in tokens]\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        src_indices, tgt_indices = self.data[idx]\n",
    "        src_indices = [self.src_vocab['<sos>']] + src_indices + [self.src_vocab['<eos>']]\n",
    "        tgt_indices = [self.tgt_vocab['<sos>']] + tgt_indices + [self.tgt_vocab['<eos>']]\n",
    "        return torch.tensor(src_indices), torch.tensor(tgt_indices)\n",
    "\n",
    "def collate_fn(batch):\n",
    "    src_batch, tgt_batch = zip(*batch)\n",
    "    src_batch = pad_sequence(src_batch, batch_first=True, padding_value=0)\n",
    "    tgt_batch = pad_sequence(tgt_batch, batch_first=True, padding_value=0)\n",
    "    return src_batch, tgt_batch\n",
    "\n",
    "# ==============================\n",
    "# Seq2Seq Model\n",
    "# ==============================\n",
    "\n",
    "class Encoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, hidden_size, num_layers=2, dropout=0.1):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers,\n",
    "                          dropout=dropout if num_layers > 1 else 0, batch_first=True)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        embedded = self.dropout(self.embedding(x))\n",
    "        outputs, (hidden, cell) = self.lstm(embedded)\n",
    "        return hidden, cell\n",
    "\n",
    "class Decoder(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, hidden_size, num_layers=2, dropout=0.1):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.lstm = nn.LSTM(embed_size, hidden_size, num_layers,\n",
    "                          dropout=dropout if num_layers > 1 else 0, batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_size, vocab_size)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x, hidden, cell):\n",
    "        x = x.unsqueeze(1)\n",
    "        embedded = self.dropout(self.embedding(x))\n",
    "        output, (hidden, cell) = self.lstm(embedded, (hidden, cell))\n",
    "        prediction = self.fc(output.squeeze(1))\n",
    "        return prediction, hidden, cell\n",
    "\n",
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        super().__init__()\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "\n",
    "    def forward(self, src, tgt, teacher_forcing_ratio=0.5):\n",
    "        batch_size = src.size(0)\n",
    "        tgt_len = tgt.size(1)\n",
    "        tgt_vocab_size = self.decoder.fc.out_features\n",
    "\n",
    "        outputs = torch.zeros(batch_size, tgt_len, tgt_vocab_size).to(self.device)\n",
    "        hidden, cell = self.encoder(src)\n",
    "\n",
    "        input = tgt[:, 0]\n",
    "        for t in range(1, tgt_len):\n",
    "            output, hidden, cell = self.decoder(input, hidden, cell)\n",
    "            outputs[:, t] = output\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            input = tgt[:, t] if teacher_force else output.argmax(1)\n",
    "\n",
    "        return outputs\n",
    "    \n",
    "# ==============================\n",
    "# Eval\n",
    "# ==============================\n",
    "\n",
    "def evaluate(model, loader, criterion, device):\n",
    "    model.eval()\n",
    "    total_loss = 0\n",
    "    total_tokens = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for src, tgt in loader:\n",
    "            src, tgt = src.to(device), tgt.to(device)\n",
    "            output = model(src, tgt, 0.0)  # No teacher forcing during eval\n",
    "            \n",
    "            output_dim = output.shape[-1]\n",
    "            output = output[:, 1:].reshape(-1, output_dim)\n",
    "            tgt = tgt[:, 1:].reshape(-1)\n",
    "            \n",
    "            loss = criterion(output, tgt)\n",
    "            batch_tokens = (tgt != 0).sum().item()\n",
    "            total_tokens += batch_tokens\n",
    "            total_loss += loss.item() * batch_tokens\n",
    "    \n",
    "    return total_loss / total_tokens\n",
    "\n",
    "# ==============================\n",
    "# Train\n",
    "# ==============================\n",
    "\n",
    "def train(model, loader, optimizer, criterion, clip, epoch, total_epochs, device):\n",
    "    model.train()\n",
    "    total_loss = 0\n",
    "    total_tokens = 0\n",
    "    \n",
    "    # Calculate total number of batches\n",
    "    num_batches = len(loader)\n",
    "    \n",
    "    # FLOPS calculation\n",
    "    hidden_size = model.encoder.lstm.hidden_size\n",
    "    embed_size = model.encoder.embedding.embedding_dim\n",
    "    num_layers = model.encoder.lstm.num_layers\n",
    "    flops_per_token = 4 * (embed_size + hidden_size) * hidden_size * 2 * num_layers * 2\n",
    "    cpu_peak_flops = 100e9\n",
    "    \n",
    "    for batch_idx, (src, tgt) in enumerate(loader, 1):\n",
    "        src, tgt = src.to(device), tgt.to(device)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        output = model(src, tgt)\n",
    "        \n",
    "        output_dim = output.shape[-1]\n",
    "        output = output[:, 1:].reshape(-1, output_dim)\n",
    "        tgt = tgt[:, 1:].reshape(-1)\n",
    "        \n",
    "        loss = criterion(output, tgt)\n",
    "        loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        optimizer.step()\n",
    "        \n",
    "        batch_tokens = (tgt != 0).sum().item()\n",
    "        total_tokens += batch_tokens\n",
    "        total_loss += loss.item() * batch_tokens\n",
    "        \n",
    "        # Calc mfu\n",
    "        tokens_per_sec = total_tokens / ((batch_idx * len(src)) + 1e-8)\n",
    "        achieved_flops = tokens_per_sec * flops_per_token\n",
    "        mfu = (achieved_flops / cpu_peak_flops) * 100\n",
    "        \n",
    "        print(f\"epoch: {epoch}/{total_epochs}, train_loss: {loss.item():.3f}, \"\n",
    "              f\"batch: {batch_idx}/{num_batches}, mfu: {mfu:.2f}%\", end='\\r')\n",
    "    \n",
    "    return total_loss / total_tokens\n",
    "\n",
    "# ==============================\n",
    "# Sample\n",
    "# ==============================\n",
    "\n",
    "def translate(model, dataset, device, n_samples=5):\n",
    "    model.eval()\n",
    "    print(\"\\nSample from trained weight:\")\n",
    "    \n",
    "    for _ in range(n_samples):\n",
    "        idx = random.randint(0, len(dataset) - 1)\n",
    "        src, tgt = dataset[idx]\n",
    "        src = src.unsqueeze(0).to(device)\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            hidden, cell = model.encoder(src)\n",
    "            input_token = torch.tensor([dataset.tgt_vocab['<sos>']]).to(device)\n",
    "            \n",
    "            translated_tokens = []\n",
    "            for _ in range(100):  # max length of 100 tokens\n",
    "                output, hidden, cell = model.decoder(input_token, hidden, cell)\n",
    "                top_token = output.argmax(1).item()\n",
    "                \n",
    "                if top_token == dataset.tgt_vocab['<eos>']:\n",
    "                    break\n",
    "                    \n",
    "                word = dataset.tgt_idx2word[top_token]\n",
    "                translated_tokens.append(word)\n",
    "                input_token = torch.tensor([top_token]).to(device)\n",
    "        \n",
    "        # Get original source and target text\n",
    "        src_text = ' '.join([dataset.src_idx2word[idx.item()] for idx in src[0]\n",
    "                           if dataset.src_idx2word[idx.item()] not in ['<sos>', '<eos>', '<pad>']])\n",
    "        tgt_text = ' '.join([dataset.tgt_idx2word[idx.item()] for idx in tgt\n",
    "                           if dataset.tgt_idx2word[idx.item()] not in ['<sos>', '<eos>', '<pad>']])\n",
    "        \n",
    "        print(f\"\\nInput (German): {src_text}\")\n",
    "        print(f\"Target (English): {tgt_text}\")\n",
    "        print(f\"Model Output (English): {' '.join(translated_tokens)}\")\n",
    "\n",
    "    \n",
    "# ==============================\n",
    "# Main\n",
    "# ==============================\n",
    "\n",
    "def main():\n",
    "    # Set seed, why not?\n",
    "    SEED = 42\n",
    "    random.seed(SEED)\n",
    "    torch.manual_seed(SEED)\n",
    "    torch.cuda.manual_seed(SEED)\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    \n",
    "    # Prepare data\n",
    "    prep_data()\n",
    "    \n",
    "    # Autodetect device (gpu poor? use cpu)\n",
    "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "    print(f\"Using device: {device}\")\n",
    "    \n",
    "    train_dataset = Vocab('data/train.jsonl', tokenize_de, tokenize_en)\n",
    "    val_dataset = Vocab('data/valid.jsonl', tokenize_de, tokenize_en,\n",
    "                                   train_dataset.src_vocab, train_dataset.tgt_vocab)\n",
    "    \n",
    "    # Load data\n",
    "    train_loader = DataLoader(train_dataset, batch_size=batch_size,\n",
    "                            shuffle=True, collate_fn=collate_fn)\n",
    "    val_loader = DataLoader(val_dataset, batch_size=batch_size,\n",
    "                          shuffle=False, collate_fn=collate_fn)\n",
    "    \n",
    "    # Initialize model\n",
    "    encoder = Encoder(len(train_dataset.src_vocab), embedding_dim,\n",
    "                     hidden_dim, num_layers, dropout)\n",
    "    decoder = Decoder(len(train_dataset.tgt_vocab), embedding_dim,\n",
    "                     hidden_dim, num_layers, dropout)\n",
    "    model = Seq2Seq(encoder, decoder, device).to(device)\n",
    "\n",
    "    total_params = sum(p.numel() for p in model.parameters())\n",
    "    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "    print(\"\\nMODEL INFO\")\n",
    "    print(\"Architecture: Seq2Seq with LSTM\")\n",
    "    print(f\"Size: {trainable_params/1e6:.2f}M\")\n",
    "    print(f\"Hidden dim: {hidden_dim}\")\n",
    "    print(f\"Embedding dim: {embedding_dim}\")\n",
    "    print(f\"Num layers: {num_layers}\")\n",
    "    print(f\"Total: {total_params:,}\")\n",
    "    print(f\"Trainable: {trainable_params:,}\")\n",
    "\n",
    "    print(f\"\\nDATA INFO\")\n",
    "    print(f\"Multi30k Dataset (German-English)\")\n",
    "    print(f\"Train samples: {len(train_dataset)}\")\n",
    "    print(f\"Validation samples: {len(val_dataset)}\")\n",
    "    print(f\"Source vocabulary size: {len(train_dataset.src_vocab)}\")\n",
    "    print(f\"Target vocabulary size: {len(train_dataset.tgt_vocab)}\\n\")\n",
    "    \n",
    "    # Initialize weights\n",
    "    def init_weights(m):\n",
    "        for name, param in m.named_parameters():\n",
    "            if 'weight' in name:\n",
    "                nn.init.normal_(param.data, mean=0, std=0.01)\n",
    "            else:\n",
    "                nn.init.constant_(param.data, 0)\n",
    "    model.apply(init_weights)\n",
    "    \n",
    "    # Set optimizer\n",
    "    optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "    criterion = nn.CrossEntropyLoss(ignore_index=train_dataset.tgt_vocab['<pad>'])\n",
    "    \n",
    "    # Train\n",
    "    train_losses = []\n",
    "    val_losses = []\n",
    "    best_val_loss = float('inf')\n",
    "    \n",
    "    for epoch in range(1, num_epochs + 1):\n",
    "        train_loss = train(model, train_loader, optimizer, criterion,\n",
    "                               clip, epoch, num_epochs, device)\n",
    "        val_loss = evaluate(model, val_loader, criterion, device)\n",
    "        \n",
    "        train_losses.append(train_loss)\n",
    "        val_losses.append(val_loss)\n",
    "        \n",
    "        print(f\"\\nepoch: {epoch}/{num_epochs}, train_loss: {train_loss:.3f}, \"\n",
    "              f\"val_loss: {val_loss:.3f}\")\n",
    "        \n",
    "        if val_loss < best_val_loss:\n",
    "            best_val_loss = val_loss\n",
    "            torch.save(model.state_dict(), 'best_model.pt')\n",
    "            # print(\"Best model saved.\")\n",
    "    \n",
    "    plt.figure(figsize=(10, 6))\n",
    "    plt.plot(range(1, num_epochs + 1), train_losses, label='Training Loss')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.title('Train / Epoch')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "    # Sample\n",
    "    model.load_state_dict(torch.load('best_model.pt'))\n",
    "    translate(model, val_dataset, device)\n",
    "\n",
    "# LFG!!\n",
    "main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cpu\n",
      "\n",
      "Sample from trained weight:\n",
      "\n",
      "Input (German): . gras dem auf <unk> dem mit personen zwei\n",
      "Target (English): two <unk> with top of heads on grass .\n",
      "Model Output (English): two dogs are in the .\n",
      "\n",
      "Input (German): . <unk> hand seiner aus <unk> ein wie , sehen sie <unk> , <unk> sehr sind kinder\n",
      "Target (English): children are very excited to see a <unk> eating food from his hand .\n",
      "Model Output (English): there are a group of people are <unk> a a a a a a . .\n",
      "\n",
      "Input (German): . boot einem in sitzen männer gekleidete blau zwei\n",
      "Target (English): two men in blue clothing sit on a boat .\n",
      "Model Output (English): two men in a <unk> are playing .\n",
      "\n",
      "Input (German): . <unk> <unk> um , dach dem auf kniet person männliche erwachsene eine\n",
      "Target (English): an adult male is kneeling on top of a roof make repairs\n",
      "Model Output (English): a <unk> <unk> <unk> <unk> <unk> a <unk> <unk> .\n",
      "\n",
      "Input (German): . zu hügel <unk> einen auf wiese eine über rennen mädchen ein und junge ein\n",
      "Target (English): a boy and girl run across a grassy field towards a <unk> hill .\n",
      "Model Output (English): a boy in a <unk> <unk> a <unk> <unk> a <unk> .\n",
      "\n",
      "Input (German): . <unk> passenden einem in baby ein trägt mann asiatischer ein\n",
      "Target (English): an asian man is carrying a baby in a matching robe .\n",
      "Model Output (English): an older man in a <unk> shirt is a a .\n",
      "\n",
      "Input (German): . bühne einer auf tanzen und <unk> tragen mädchen drei\n",
      "Target (English): three girls wear straw hats and dance on a stage .\n",
      "Model Output (English): two men in a a a a .\n",
      "\n",
      "Input (German): . macht <unk> einen der , hosen blauen mit mann ein\n",
      "Target (English): man with blue pants flipping in the air .\n",
      "Model Output (English): man in a red dress .\n",
      "\n",
      "Input (German): . schwingt tennisschläger einen der , shorts blauen und hemd weißem mit mann ein\n",
      "Target (English): a man in a white shirt and blue shorts swinging a tennis racket .\n",
      "Model Output (English): man in red shirt and white pants .\n",
      "\n",
      "Input (German): . <unk> eine <unk> kinder\n",
      "Target (English): kids <unk> a rock band .\n",
      "Model Output (English): <unk> <unk> .\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/lv/fs2j5vr90zz71f7mbvytr3s80000gn/T/ipykernel_98069/4009759217.py:19: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  model.load_state_dict(torch.load('best_model.pt'))\n"
     ]
    }
   ],
   "source": [
    "train_dataset = Vocab('data/train.jsonl', tokenize_de, tokenize_en)\n",
    "val_dataset = Vocab('data/valid.jsonl', tokenize_de, tokenize_en,\n",
    "                                train_dataset.src_vocab, train_dataset.tgt_vocab)\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(f\"Using device: {device}\")\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=batch_size,\n",
    "                        shuffle=True, collate_fn=collate_fn)\n",
    "val_loader = DataLoader(val_dataset, batch_size=batch_size,\n",
    "                        shuffle=False, collate_fn=collate_fn)\n",
    "\n",
    "encoder = Encoder(len(train_dataset.src_vocab), embedding_dim,\n",
    "                    hidden_dim, num_layers, dropout)\n",
    "decoder = Decoder(len(train_dataset.tgt_vocab), embedding_dim,\n",
    "                    hidden_dim, num_layers, dropout)\n",
    "model = Seq2Seq(encoder, decoder, device).to(device)\n",
    "\n",
    "model.load_state_dict(torch.load('best_model.pt'))\n",
    "translate(model, val_dataset, device, n_samples=10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}