Models/qa-with-triviaqa-dataset-and-bert-base.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"name":"QA with TriviaQA dataset and BERT-Base.ipynb","provenance":[],"collapsed_sections":[]}},"nbformat_minor":0,"nbformat":4,"cells":[{"cell_type":"markdown","source":["# Question Answering with TriviaQA dataset and BERT-Base\n","\n","In this notebook i train again bert-base-uncased for qa tasks but this time with the [TriviaQA](https://nlp.cs.washington.edu/triviaqa/) dataset. I use the same model parameters with the SQuAD 2.0 model.\n","\n","**Note:** due to limitations on the free version of google colab, the experiments on this assignment were executed on [Kaggle](https://www.kaggle.com/)."],"metadata":{"id":"is-BQ_NxNsO0"}},{"cell_type":"markdown","source":["## First Steps"],"metadata":{"id":"jTfBhfsHN7-a"}},{"cell_type":"markdown","source":["**GPU Info**\n","\n","To get the same results you should have a Tesla P100 GPU. This GPU is available for free usage on [Kaggle](https://www.kaggle.com/)."],"metadata":{"id":"8GvtY-GhN-vi"}},{"cell_type":"code","source":["!nvidia-smi"],"metadata":{"id":"p2mc9E6eNa0R","outputId":"562617ae-c729-438b-97fd-e989fc77de0b","executionInfo":{"status":"ok","timestamp":1646593693702,"user_tz":-120,"elapsed":245,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:03:55.367183Z","iopub.execute_input":"2022-03-07T18:03:55.368121Z","iopub.status.idle":"2022-03-07T18:03:56.086849Z","shell.execute_reply.started":"2022-03-07T18:03:55.368005Z","shell.execute_reply":"2022-03-07T18:03:56.086031Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Mon Mar  7 18:03:55 2022       \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 450.119.04   Driver Version: 450.119.04   CUDA Version: 11.0     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n| N/A   37C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |\n|                               |                      |                  N/A |\n+-------------------------------+----------------------+----------------------+\n                                                                               \n+-----------------------------------------------------------------------------+\n| Processes:                                                                  |\n|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n|        ID   ID                                                   Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n","output_type":"stream"}]},{"cell_type":"markdown","source":["**Install pytorch interface for pre-trained BERT**\n","\n","Then, we will install the interface to use the pre-trained BERT model."],"metadata":{"id":"_pSfqxmEPx88"}},{"cell_type":"code","source":["!pip install transformers"],"metadata":{"id":"0rZyAPuhPySz","outputId":"3fe2671b-41ae-4fd3-f652-03fb21a9c2f4","executionInfo":{"status":"ok","timestamp":1646593712064,"user_tz":-120,"elapsed":16066,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:03:59.044253Z","iopub.execute_input":"2022-03-07T18:03:59.044944Z","iopub.status.idle":"2022-03-07T18:04:07.843884Z","shell.execute_reply.started":"2022-03-07T18:03:59.044909Z","shell.execute_reply":"2022-03-07T18:04:07.843000Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Requirement already satisfied: transformers in /opt/conda/lib/python3.7/site-packages (4.15.0)\nRequirement already satisfied: tokenizers<0.11,>=0.10.1 in /opt/conda/lib/python3.7/site-packages (from transformers) (0.10.3)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from transformers) (6.0)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (2021.11.10)\nRequirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (from transformers) (0.0.47)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers) (4.11.2)\nRequirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from transformers) (0.2.1)\nRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers) (2.26.0)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers) (3.4.2)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (1.20.3)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from transformers) (21.3)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from transformers) (4.62.3)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (4.1.1)\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->transformers) (3.0.6)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers) (3.6.0)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (3.1)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (1.26.7)\nRequirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2.0.9)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2021.10.8)\nRequirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (8.0.3)\nRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.16.0)\nRequirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.1.0)\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n","output_type":"stream"}]},{"cell_type":"markdown","source":["**Import necessary libraries**\n","\n","Now, it's time to import the necessary libraries for this notebook."],"metadata":{"id":"VRmYivZrQPrR"}},{"cell_type":"code","source":["import torch\n","import json\n","import pandas as pd\n","import numpy as np\n","import random\n","import matplotlib.pyplot as plt\n","import re\n","import string\n","import collections\n","from transformers import BertTokenizerFast, BertForQuestionAnswering\n","from transformers.tokenization_utils_base import BatchEncoding\n","from torch.utils.data import Dataset, DataLoader\n","from tqdm import tqdm\n","\n","pd.set_option('max_colwidth', 500)\n","%matplotlib inline"],"metadata":{"id":"V9xaR0zaQP_g","execution":{"iopub.status.busy":"2022-03-07T18:04:10.274523Z","iopub.execute_input":"2022-03-07T18:04:10.274803Z","iopub.status.idle":"2022-03-07T18:04:12.578814Z","shell.execute_reply.started":"2022-03-07T18:04:10.274756Z","shell.execute_reply":"2022-03-07T18:04:12.578087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Enable CUDA**\n","\n","Enable CUDA for GPU utilization by our model. This makes calculations and thus the training of our models faster."],"metadata":{"id":"xUz3L_c8QVXR"}},{"cell_type":"code","source":["device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n","device"],"metadata":{"id":"doKXFdk8QVoD","outputId":"8f33ad68-3224-4e14-bf58-989696447dc4","executionInfo":{"status":"ok","timestamp":1646593723427,"user_tz":-120,"elapsed":249,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:04:15.058401Z","iopub.execute_input":"2022-03-07T18:04:15.058651Z","iopub.status.idle":"2022-03-07T18:04:15.106199Z","shell.execute_reply.started":"2022-03-07T18:04:15.058622Z","shell.execute_reply":"2022-03-07T18:04:15.105243Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":4,"output_type":"execute_result","data":{"text/plain":"device(type='cuda', index=0)"},"metadata":{}}]},{"cell_type":"markdown","source":["**Set random seed**\n","\n","Set a constant random seed in order to get the same (deterministic) outputs every time we run our models.\n","\n"],"metadata":{"id":"HKOXPf5XQX3Z"}},{"cell_type":"code","source":["seed = 17064\n","def reset_seed():\n","  random.seed(seed)\n","  np.random.seed(seed)\n","  torch.manual_seed(seed)\n","  torch.cuda.manual_seed_all(seed)\n","reset_seed()"],"metadata":{"id":"QnruhHSWQcGO","execution":{"iopub.status.busy":"2022-03-07T18:04:17.265824Z","iopub.execute_input":"2022-03-07T18:04:17.266375Z","iopub.status.idle":"2022-03-07T18:04:17.274007Z","shell.execute_reply.started":"2022-03-07T18:04:17.266338Z","shell.execute_reply":"2022-03-07T18:04:17.273168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**BERT model definition**\n","\n","And here, we define the name of the pre-trained bert model from [Hugging Face](https://huggingface.co/models) which we'll use."],"metadata":{"id":"Y-Hb3sovQwHa"}},{"cell_type":"code","source":["BERT_MODEL_NAME = 'bert-base-uncased'"],"metadata":{"id":"wqtitofaQdb3","execution":{"iopub.status.busy":"2022-03-07T18:04:21.226553Z","iopub.execute_input":"2022-03-07T18:04:21.226973Z","iopub.status.idle":"2022-03-07T18:04:21.230850Z","shell.execute_reply.started":"2022-03-07T18:04:21.226939Z","shell.execute_reply":"2022-03-07T18:04:21.229840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Dataset loading\n","\n","Now, let's load our datasets:"],"metadata":{"id":"mtvSPZfoQ5f4"}},{"cell_type":"markdown","source":["**Define json preprocessing function**\n","\n","First, we will define a function that takes a squad json dataset file path as argument and returns a dataframe with the questions, their answers in the form (start_position, end_position, answer_text) and the corpus that contains the answer to the corresponding question."],"metadata":{"id":"Jnby8mS-gXoz"}},{"cell_type":"code","source":["def squad_load_from_json(json_file_path: str):\n","  with open(json_file_path, \"r\") as f:\n","    json_data = json.load(f)['data']\n","    questions = []\n","    answers = []\n","    corpuses = []\n","    for category in json_data:\n","      for paragraph in category['paragraphs']:\n","        context = paragraph['context']\n","        for qa in paragraph['qas']:\n","          corpuses.append(context)\n","          question = qa['question']\n","          questions.append(question)\n","          ans_list = qa['answers']\n","          ans_set = set()\n","          for idx, ans in enumerate(ans_list):\n","            ans_set.add((ans['answer_start'], ans['answer_start']+len(ans['text']), ans['text']))\n","          answers.append(list(ans_set))\n","    return pd.DataFrame(data={'question':questions, 'answer':answers, 'corpus':corpuses})"],"metadata":{"id":"mWhOlZ69gcKr","execution":{"iopub.status.busy":"2022-03-07T18:06:33.000829Z","iopub.execute_input":"2022-03-07T18:06:33.001380Z","iopub.status.idle":"2022-03-07T18:06:33.008695Z","shell.execute_reply.started":"2022-03-07T18:06:33.001347Z","shell.execute_reply":"2022-03-07T18:06:33.007893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Then, we will load the datasets converted to SQuAD json format."],"metadata":{"id":"AcT_n6pP3PFB"}},{"cell_type":"markdown","source":["**Train Dataset**"],"metadata":{"id":"ObZ_q6UXQ8v3"}},{"cell_type":"code","source":["train_dataset = squad_load_from_json(\"../input/triviaqasquadformat/triviaqa_train.json\")\n","train_dataset"],"metadata":{"id":"YIUz68o5g78D","outputId":"69da4e90-8a7f-46c9-e99e-262342eaa377","executionInfo":{"status":"ok","timestamp":1646593731851,"user_tz":-120,"elapsed":1484,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:06:35.177381Z","iopub.execute_input":"2022-03-07T18:06:35.177999Z","iopub.status.idle":"2022-03-07T18:06:44.365381Z","shell.execute_reply.started":"2022-03-07T18:06:35.177948Z","shell.execute_reply":"2022-03-07T18:06:44.364730Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":16,"output_type":"execute_result","data":{"text/plain":"                                                                                                                                               question  \\\n0                                                                                                            Where in England was Dame Judi Dench born?   \n1                                                                                                            Where in England was Dame Judi Dench born?   \n2                                                                                           From which country did Angola achieve independence in 1975?   \n3                                                                                           From which country did Angola achieve independence in 1975?   \n4                                                                                           From which country did Angola achieve independence in 1975?   \n...                                                                                                                                                 ...   \n110642  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110643  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110644  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110645                                              July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?   \n110646                                                                    According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??   \n\n                                                   answer  \\\n0                                    [(1569, 1573, York)]   \n1                                    [(2248, 2252, York)]   \n2                                 [(2899, 2906, Portuga)]   \n3                                                      []   \n4                                   [(199, 206, Portuga)]   \n...                                                   ...   \n110642                                                 []   \n110643                                                 []   \n110644  [(168, 203, 26th President of the United States)]   \n110645                           [(336, 346, Bugs Bunny)]   \n110646                                                 []   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \n0       England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...  \n1       Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...  \n2       A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...  \n3       Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...  \n4       The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...  \n110642  The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...  \n110643  The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...  \n110644  Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...  \n110645  A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...  \n110646  Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...  \n\n[110647 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>[(1569, 1573, York)]</td>\n      <td>England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>[(2248, 2252, York)]</td>\n      <td>Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>[(2899, 2906, Portuga)]</td>\n      <td>A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>[]</td>\n      <td>Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>[(199, 206, Portuga)]</td>\n      <td>The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>110642</th>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>[]</td>\n      <td>The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...</td>\n    </tr>\n    <tr>\n      <th>110643</th>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>[]</td>\n      <td>The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...</td>\n    </tr>\n    <tr>\n      <th>110644</th>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>[(168, 203, 26th President of the United States)]</td>\n      <td>Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...</td>\n    </tr>\n    <tr>\n      <th>110645</th>\n      <td>July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?</td>\n      <td>[(336, 346, Bugs Bunny)]</td>\n      <td>A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...</td>\n    </tr>\n    <tr>\n      <th>110646</th>\n      <td>According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??</td>\n      <td>[]</td>\n      <td>Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...</td>\n    </tr>\n  </tbody>\n</table>\n<p>110647 rows × 3 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Validation Dataset**"],"metadata":{"id":"YnNmSit1RoDl"}},{"cell_type":"code","source":["validation_dataset = squad_load_from_json(\"../input/triviaqasquadformat/triviaqa_dev.json\")\n","validation_dataset"],"metadata":{"id":"UHM-6PXxRr3t","outputId":"0a38841c-33b9-41ef-fff5-356b67a35be8","executionInfo":{"status":"ok","timestamp":1646593734549,"user_tz":-120,"elapsed":252,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:07:12.321522Z","iopub.execute_input":"2022-03-07T18:07:12.321818Z","iopub.status.idle":"2022-03-07T18:07:14.529481Z","shell.execute_reply.started":"2022-03-07T18:07:12.321779Z","shell.execute_reply":"2022-03-07T18:07:14.528649Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":17,"output_type":"execute_result","data":{"text/plain":"                                                                                                                                question  \\\n0                                                                  Which Lloyd Webber musical premiered in the US on 10th December 1993?   \n1                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n2                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n3                                                                                         Who had a 70s No 1 hit with Kiss You All Over?   \n4                                                                                      What claimed the life of singer Kathleen Ferrier?   \n...                                                                                                                                  ...   \n14224                             With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?   \n14225                                                                            Who tried to steal Christmas from the town of Whoville?   \n14226                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14227                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14228  In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?   \n\n                          answer  \\\n0                             []   \n1                             []   \n2                             []   \n3              [(62, 67, Exile)]   \n4         [(2488, 2494, Cancer)]   \n...                          ...   \n14224  [(4, 18, National Guard)]   \n14225       [(161, 167, Grinch)]   \n14226      [(1908, 1913, Brown)]   \n14227      [(2032, 2037, Brown)]   \n14228      [(0, 10, Horseshoes)]   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \n0      Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...  \n1      The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...  \n2      Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...  \n3      `` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...  \n4      Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...  \n14224  The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...  \n14225  Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...  \n14226  `` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...  \n14227  Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...  \n14228  Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...  \n\n[14229 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Which Lloyd Webber musical premiered in the US on 10th December 1993?</td>\n      <td>[]</td>\n      <td>Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>[]</td>\n      <td>The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>[]</td>\n      <td>Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Who had a 70s No 1 hit with Kiss You All Over?</td>\n      <td>[(62, 67, Exile)]</td>\n      <td>`` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>What claimed the life of singer Kathleen Ferrier?</td>\n      <td>[(2488, 2494, Cancer)]</td>\n      <td>Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>14224</th>\n      <td>With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?</td>\n      <td>[(4, 18, National Guard)]</td>\n      <td>The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...</td>\n    </tr>\n    <tr>\n      <th>14225</th>\n      <td>Who tried to steal Christmas from the town of Whoville?</td>\n      <td>[(161, 167, Grinch)]</td>\n      <td>Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...</td>\n    </tr>\n    <tr>\n      <th>14226</th>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>[(1908, 1913, Brown)]</td>\n      <td>`` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...</td>\n    </tr>\n    <tr>\n      <th>14227</th>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>[(2032, 2037, Brown)]</td>\n      <td>Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...</td>\n    </tr>\n    <tr>\n      <th>14228</th>\n      <td>In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?</td>\n      <td>[(0, 10, Horseshoes)]</td>\n      <td>Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...</td>\n    </tr>\n  </tbody>\n</table>\n<p>14229 rows × 3 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Exploding datasets on multiple answers**\n","\n","Now, we will explode the datasets on the answers list field because it will help later."],"metadata":{"id":"ViYC5HkjYtEA"}},{"cell_type":"code","source":["train_dataset = train_dataset.explode('answer').reset_index()\n","train_dataset"],"metadata":{"id":"nvJXORnjY-Ji","executionInfo":{"status":"ok","timestamp":1646593743068,"user_tz":-120,"elapsed":246,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"b9463689-b098-402e-e6b2-4700b277e394","execution":{"iopub.status.busy":"2022-03-07T18:07:22.481086Z","iopub.execute_input":"2022-03-07T18:07:22.481652Z","iopub.status.idle":"2022-03-07T18:07:22.633252Z","shell.execute_reply.started":"2022-03-07T18:07:22.481615Z","shell.execute_reply":"2022-03-07T18:07:22.632573Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":18,"output_type":"execute_result","data":{"text/plain":"         index  \\\n0            0   \n1            1   \n2            2   \n3            3   \n4            4   \n...        ...   \n110642  110642   \n110643  110643   \n110644  110644   \n110645  110645   \n110646  110646   \n\n                                                                                                                                               question  \\\n0                                                                                                            Where in England was Dame Judi Dench born?   \n1                                                                                                            Where in England was Dame Judi Dench born?   \n2                                                                                           From which country did Angola achieve independence in 1975?   \n3                                                                                           From which country did Angola achieve independence in 1975?   \n4                                                                                           From which country did Angola achieve independence in 1975?   \n...                                                                                                                                                 ...   \n110642  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110643  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110644  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110645                                              July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?   \n110646                                                                    According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??   \n\n                                                 answer  \\\n0                                    (1569, 1573, York)   \n1                                    (2248, 2252, York)   \n2                                 (2899, 2906, Portuga)   \n3                                                   NaN   \n4                                   (199, 206, Portuga)   \n...                                                 ...   \n110642                                              NaN   \n110643                                              NaN   \n110644  (168, 203, 26th President of the United States)   \n110645                           (336, 346, Bugs Bunny)   \n110646                                              NaN   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \n0       England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...  \n1       Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...  \n2       A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...  \n3       Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...  \n4       The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...  \n110642  The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...  \n110643  The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...  \n110644  Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...  \n110645  A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...  \n110646  Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...  \n\n[110647 rows x 4 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>(1569, 1573, York)</td>\n      <td>England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>(2248, 2252, York)</td>\n      <td>Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>(2899, 2906, Portuga)</td>\n      <td>A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>NaN</td>\n      <td>Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>(199, 206, Portuga)</td>\n      <td>The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>110642</th>\n      <td>110642</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>NaN</td>\n      <td>The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...</td>\n    </tr>\n    <tr>\n      <th>110643</th>\n      <td>110643</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>NaN</td>\n      <td>The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...</td>\n    </tr>\n    <tr>\n      <th>110644</th>\n      <td>110644</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>(168, 203, 26th President of the United States)</td>\n      <td>Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...</td>\n    </tr>\n    <tr>\n      <th>110645</th>\n      <td>110645</td>\n      <td>July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?</td>\n      <td>(336, 346, Bugs Bunny)</td>\n      <td>A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...</td>\n    </tr>\n    <tr>\n      <th>110646</th>\n      <td>110646</td>\n      <td>According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??</td>\n      <td>NaN</td>\n      <td>Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...</td>\n    </tr>\n  </tbody>\n</table>\n<p>110647 rows × 4 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"code","source":["validation_dataset = validation_dataset.explode('answer').reset_index()\n","validation_dataset"],"metadata":{"id":"8XBWvjFZYkth","executionInfo":{"status":"ok","timestamp":1646593745568,"user_tz":-120,"elapsed":238,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"8bc24a53-1da2-41b9-fd09-901ae1bdc58f","execution":{"iopub.status.busy":"2022-03-07T18:07:26.680545Z","iopub.execute_input":"2022-03-07T18:07:26.681071Z","iopub.status.idle":"2022-03-07T18:07:26.710590Z","shell.execute_reply.started":"2022-03-07T18:07:26.681034Z","shell.execute_reply":"2022-03-07T18:07:26.709927Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":19,"output_type":"execute_result","data":{"text/plain":"       index  \\\n0          0   \n1          1   \n2          2   \n3          3   \n4          4   \n...      ...   \n14224  14224   \n14225  14225   \n14226  14226   \n14227  14227   \n14228  14228   \n\n                                                                                                                                question  \\\n0                                                                  Which Lloyd Webber musical premiered in the US on 10th December 1993?   \n1                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n2                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n3                                                                                         Who had a 70s No 1 hit with Kiss You All Over?   \n4                                                                                      What claimed the life of singer Kathleen Ferrier?   \n...                                                                                                                                  ...   \n14224                             With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?   \n14225                                                                            Who tried to steal Christmas from the town of Whoville?   \n14226                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14227                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14228  In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?   \n\n                        answer  \\\n0                          NaN   \n1                          NaN   \n2                          NaN   \n3              (62, 67, Exile)   \n4         (2488, 2494, Cancer)   \n...                        ...   \n14224  (4, 18, National Guard)   \n14225       (161, 167, Grinch)   \n14226      (1908, 1913, Brown)   \n14227      (2032, 2037, Brown)   \n14228      (0, 10, Horseshoes)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \n0      Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...  \n1      The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...  \n2      Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...  \n3      `` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...  \n4      Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...  \n14224  The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...  \n14225  Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...  \n14226  `` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...  \n14227  Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...  \n14228  Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...  \n\n[14229 rows x 4 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>Which Lloyd Webber musical premiered in the US on 10th December 1993?</td>\n      <td>NaN</td>\n      <td>Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>NaN</td>\n      <td>The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>NaN</td>\n      <td>Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>Who had a 70s No 1 hit with Kiss You All Over?</td>\n      <td>(62, 67, Exile)</td>\n      <td>`` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>What claimed the life of singer Kathleen Ferrier?</td>\n      <td>(2488, 2494, Cancer)</td>\n      <td>Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>14224</th>\n      <td>14224</td>\n      <td>With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?</td>\n      <td>(4, 18, National Guard)</td>\n      <td>The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...</td>\n    </tr>\n    <tr>\n      <th>14225</th>\n      <td>14225</td>\n      <td>Who tried to steal Christmas from the town of Whoville?</td>\n      <td>(161, 167, Grinch)</td>\n      <td>Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...</td>\n    </tr>\n    <tr>\n      <th>14226</th>\n      <td>14226</td>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>(1908, 1913, Brown)</td>\n      <td>`` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...</td>\n    </tr>\n    <tr>\n      <th>14227</th>\n      <td>14227</td>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>(2032, 2037, Brown)</td>\n      <td>Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...</td>\n    </tr>\n    <tr>\n      <th>14228</th>\n      <td>14228</td>\n      <td>In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?</td>\n      <td>(0, 10, Horseshoes)</td>\n      <td>Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...</td>\n    </tr>\n  </tbody>\n</table>\n<p>14229 rows × 4 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["## Tokenization process"],"metadata":{"id":"9bSzq-HQ3gsO"}},{"cell_type":"markdown","source":["**Download BERT tokenizer**\n","\n","After loading the datasets, before we use them for the evaluation process, we must first tokenize them to calculate the gold answer start and end token positions."],"metadata":{"id":"qx0bT8qWC2W2"}},{"cell_type":"code","source":["tokenizer = BertTokenizerFast.from_pretrained(BERT_MODEL_NAME)"],"metadata":{"id":"uwfHVB8fC2w7","executionInfo":{"status":"ok","timestamp":1646593748992,"user_tz":-120,"elapsed":707,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"8349adc2-8c93-4791-9c65-8b9b20bce455","execution":{"iopub.status.busy":"2022-03-07T18:07:39.837301Z","iopub.execute_input":"2022-03-07T18:07:39.837557Z","iopub.status.idle":"2022-03-07T18:07:41.135475Z","shell.execute_reply.started":"2022-03-07T18:07:39.837528Z","shell.execute_reply":"2022-03-07T18:07:41.134752Z"},"trusted":true,"colab":{"referenced_widgets":["b8ce346493654f8dae7b9059c6301f1f","2d48d6c5f0664ce1a650274b50b7c10e","14cf20fc202f453bb39071d2414d59bf","04a0b5f4ecfb4ccebd3a4e59e103fdfe"]}},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b8ce346493654f8dae7b9059c6301f1f"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/455k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"2d48d6c5f0664ce1a650274b50b7c10e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"14cf20fc202f453bb39071d2414d59bf"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"04a0b5f4ecfb4ccebd3a4e59e103fdfe"}},"metadata":{}}]},{"cell_type":"markdown","source":["**Calculate start and end tokens of the answers**"],"metadata":{"id":"EvwlQ6BOh9CS"}},{"cell_type":"code","source":["def calculate_tokenized_ans_indices(dataset: pd.DataFrame):\n","  ans_tok_start = []\n","  ans_tok_end = []\n","  ans_tok_text = []\n","  for idx, ans in enumerate(dataset['answer'].values):\n","    if not pd.isna(ans):\n","      ans_text_start = ans[0]\n","      ans_text_end = ans[1]\n","      ans_text = ans[2]\n","      encoding = tokenizer.encode_plus(text=dataset['corpus'].values[idx], text_pair=dataset['question'].values[idx], max_length=512, padding='max_length', truncation=True)\n","      ans_start = encoding.char_to_token(0, ans_text_start)\n","      ans_end = encoding.char_to_token(0, ans_text_end-1)\n","      # Τruncated answer\n","      if ans_start is None:\n","        ans_start = ans_end = tokenizer.model_max_length\n","      elif ans_end is None:\n","        ans_end = [i for i, inp in enumerate(encoding['input_ids']) if inp == tokenizer.sep_token_id][0]\n","      # Wrong because it gives free score points on truncated answers. Fixed this on the cross evaluation script.\n","      ans_text_tok = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(encoding['input_ids'][ans_start:ans_end+1]))\n","    else:\n","      ans_start = ans_end = tokenizer.model_max_length\n","      ans_text_tok = \"\"\n","    ans_tok_start.append(ans_start)\n","    ans_tok_end.append(ans_end)\n","    ans_tok_text.append(ans_text_tok)\n","  dataset['ans_start_tok'] = ans_tok_start\n","  dataset['ans_end_tok'] = ans_tok_end\n","  dataset['ans_tok_text'] = ans_tok_text\n","  return dataset"],"metadata":{"id":"gBuRHPaUiFp-","execution":{"iopub.status.busy":"2022-03-07T18:07:43.161088Z","iopub.execute_input":"2022-03-07T18:07:43.161347Z","iopub.status.idle":"2022-03-07T18:07:43.170977Z","shell.execute_reply.started":"2022-03-07T18:07:43.161318Z","shell.execute_reply":"2022-03-07T18:07:43.170159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_dataset = calculate_tokenized_ans_indices(train_dataset)\n","train_dataset"],"metadata":{"id":"n_JjLTfglsGv","outputId":"900136de-0c37-47f8-e082-66dbfe2995cb","executionInfo":{"status":"ok","timestamp":1646593824144,"user_tz":-120,"elapsed":68894,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:07:45.360536Z","iopub.execute_input":"2022-03-07T18:07:45.361315Z","iopub.status.idle":"2022-03-07T18:12:09.759970Z","shell.execute_reply.started":"2022-03-07T18:07:45.361268Z","shell.execute_reply":"2022-03-07T18:12:09.759283Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":22,"output_type":"execute_result","data":{"text/plain":"         index  \\\n0            0   \n1            1   \n2            2   \n3            3   \n4            4   \n...        ...   \n110642  110642   \n110643  110643   \n110644  110644   \n110645  110645   \n110646  110646   \n\n                                                                                                                                               question  \\\n0                                                                                                            Where in England was Dame Judi Dench born?   \n1                                                                                                            Where in England was Dame Judi Dench born?   \n2                                                                                           From which country did Angola achieve independence in 1975?   \n3                                                                                           From which country did Angola achieve independence in 1975?   \n4                                                                                           From which country did Angola achieve independence in 1975?   \n...                                                                                                                                                 ...   \n110642  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110643  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110644  On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?   \n110645                                              July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?   \n110646                                                                    According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??   \n\n                                                 answer  \\\n0                                    (1569, 1573, York)   \n1                                    (2248, 2252, York)   \n2                                 (2899, 2906, Portuga)   \n3                                                   NaN   \n4                                   (199, 206, Portuga)   \n...                                                 ...   \n110642                                              NaN   \n110643                                              NaN   \n110644  (168, 203, 26th President of the United States)   \n110645                           (336, 346, Bugs Bunny)   \n110646                                              NaN   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \\\n0       England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...   \n1       Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...   \n2       A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...   \n3       Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...   \n4       The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...   \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...   \n110642  The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...   \n110643  The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...   \n110644  Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...   \n110645  A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...   \n110646  Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...   \n\n        ans_start_tok  ans_end_tok                         ans_tok_text  \n0                 300          300                            yorkshire  \n1                 461          461                            yorkshire  \n2                 512          512                                       \n3                 512          512                                       \n4                  39           39                             portugal  \n...               ...          ...                                  ...  \n110642            512          512                                       \n110643            512          512                                       \n110644             36           41  26th president of the united states  \n110645             76           77                           bugs bunny  \n110646            512          512                                       \n\n[110647 rows x 7 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n      <th>ans_start_tok</th>\n      <th>ans_end_tok</th>\n      <th>ans_tok_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>(1569, 1573, York)</td>\n      <td>England is a country that is part of the United Kingdom . It shares land borders with Scotland to the north and Wales to the west . The Irish Sea lies northwest of England and the Celtic Sea lies to the southwest . England is separated from continental Europe by the North Sea to the east and the English Channel to the south . The country covers much of the central and southern part of the island of Great Britain , which lies in the North Atlantic ; and includes over 100 smaller islands such ...</td>\n      <td>300</td>\n      <td>300</td>\n      <td>yorkshire</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>Where in England was Dame Judi Dench born?</td>\n      <td>(2248, 2252, York)</td>\n      <td>Dame Judith Olivia `` Judi '' Dench , ( born 9 December 1934 ) is an English actress and author . Dench made her professional debut in 1957 with the Old Vic Company . Over the following few years she performed in several of Shakespeare 's plays in such roles as Ophelia in Hamlet , Juliet in Romeo and Juliet and Lady Macbeth in Macbeth . Although most of her work during this period was in theatre , she also branched into film work , and won a BAFTA Award as Most Promising Newcomer . She drew ...</td>\n      <td>461</td>\n      <td>461</td>\n      <td>yorkshire</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>(2899, 2906, Portuga)</td>\n      <td>A nation state is a type of state that conjoins the political entity of a state to the cultural entity of a nation , from which it aims to derive its political legitimacy to rule and potentially its status as a sovereign state if one accepts the declarative theory of statehood as opposed to the constitutive theory.Such a definition is a working one : `` All attempts to develop terminological consensus around `` nation '' resulted in failure '' , concludes . Walker Connor , in [ ] discusses t...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>NaN</td>\n      <td>Angola , officially the Republic of Angola ( ; Kikongo , Kimbundu and Umbundu : Repubilika ya Ngola ) , is a country in Southern Africa . It is the seventh-largest country in Africa , and is bordered by Namibia to the south , the Democratic Republic of the Congo to the north and east , Zambia to the east , and the Atlantic Ocean to west . The exclave province of Cabinda has borders with the Republic of the Congo and the Democratic Republic of the Congo . The capital and largest city of Angol...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>From which country did Angola achieve independence in 1975?</td>\n      <td>(199, 206, Portuga)</td>\n      <td>The Angolan Civil War ( ) was a major civil conflict in Angola , beginning in 1975 and continuing , with some interludes , until 2002 . The war began immediately after Angola became independent from Portugal in November 1975 . Prior to this , a decolonisation conflict , the Angolan War of Independence ( 1961–74 ) , had taken place . The following civil war was essentially a power struggle between two former liberation movements , the People 's Movement for the Liberation of Angola ( MPLA ) a...</td>\n      <td>39</td>\n      <td>39</td>\n      <td>portugal</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>110642</th>\n      <td>110642</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>NaN</td>\n      <td>The President of the United States of America ( POTUS ) is the elected head of state and head of government of the United States . The president leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces . \\n \\n The President of the United States is considered one of the world 's most powerful people , leading the world 's only contemporary superpower . The role includes being the commander-in-chief of the world 's most expensive mili...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>110643</th>\n      <td>110643</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>NaN</td>\n      <td>The Medal of Honor is the United States of America 's highest military honor , awarded for personal acts of valor above and beyond the call of duty . The medal is awarded by the President of the United States in the name of the U.S. Congress to U.S. military personnel only . There are three versions of the medal , one for the Army , one for the Navy , and one for the Air Force . Personnel of the Marine Corps and Coast Guard receive the Navy version . \\n \\n The Medal of Honor was created as a...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>110644</th>\n      <td>110644</td>\n      <td>On January 16, 2001, President Bill Clinton awarded what former president a posthumous Medal of Honor, the only president to have received one?</td>\n      <td>(168, 203, 26th President of the United States)</td>\n      <td>Theodore Roosevelt Jr. ( ; October 27 , 1858 – January 6 , 1919 ) was an American statesman , author , explorer , soldier , naturalist , and reformer who served as the 26th President of the United States from 1901 to 1909 . As a leader of the Republican Party during this time , he became a driving force for the Progressive Era in the United States in the early 20th century . \\n \\n Born a sickly child with debilitating asthma , Roosevelt successfully overcame his health problems by embracing ...</td>\n      <td>36</td>\n      <td>41</td>\n      <td>26th president of the united states</td>\n    </tr>\n    <tr>\n      <th>110645</th>\n      <td>110645</td>\n      <td>July 27, 1940 saw the introduction of what beloved cartoon character in the 8:15 short A Wild Hare?</td>\n      <td>(336, 346, Bugs Bunny)</td>\n      <td>A Wild Hare ( re-released as The Wild Hare ) is a 1940 Warner Bros. Merrie Melodies animated short film . It was produced by Leon Schlesinger Productions , directed by Tex Avery , and written by Rich Hogan . It was originally released on July 27 , 1940 . A Wild Hare is considered by most film historians to be the first `` official '' Bugs Bunny cartoon . \\n The title is a play on `` wild hair '' , the first of many puns between `` hare '' and `` hair '' that would appear in Bugs Bunny titles...</td>\n      <td>76</td>\n      <td>77</td>\n      <td>bugs bunny</td>\n    </tr>\n    <tr>\n      <th>110646</th>\n      <td>110646</td>\n      <td>According to the Bart Simpsons TV ad, Nobody better lay a finger on my what??</td>\n      <td>NaN</td>\n      <td>Bartholomew JoJo `` Bart '' Simpson is a fictional character in the American animated television series The Simpsons and part of the Simpson family . He is voiced by Nancy Cartwright and first appeared on television in The Tracey Ullman Show short `` Good Night '' on April 19 , 1987 . Cartoonist Matt Groening created and designed Bart while waiting in the lobby of James L. Brooks ' office . Groening had been called to pitch a series of shorts based on his comic strip , Life in Hell , but ins...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n  </tbody>\n</table>\n<p>110647 rows × 7 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"code","source":["validation_dataset = calculate_tokenized_ans_indices(validation_dataset)\n","validation_dataset"],"metadata":{"id":"yfp7_F4y7xc-","outputId":"d5ecc67c-146f-4aa3-8025-649a9f0606a4","executionInfo":{"status":"ok","timestamp":1646593852361,"user_tz":-120,"elapsed":9445,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:12:27.029565Z","iopub.execute_input":"2022-03-07T18:12:27.030093Z","iopub.status.idle":"2022-03-07T18:13:02.582177Z","shell.execute_reply.started":"2022-03-07T18:12:27.030054Z","shell.execute_reply":"2022-03-07T18:13:02.581509Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":23,"output_type":"execute_result","data":{"text/plain":"       index  \\\n0          0   \n1          1   \n2          2   \n3          3   \n4          4   \n...      ...   \n14224  14224   \n14225  14225   \n14226  14226   \n14227  14227   \n14228  14228   \n\n                                                                                                                                question  \\\n0                                                                  Which Lloyd Webber musical premiered in the US on 10th December 1993?   \n1                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n2                                                                          Who was the next British Prime Minister after Arthur Balfour?   \n3                                                                                         Who had a 70s No 1 hit with Kiss You All Over?   \n4                                                                                      What claimed the life of singer Kathleen Ferrier?   \n...                                                                                                                                  ...   \n14224                             With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?   \n14225                                                                            Who tried to steal Christmas from the town of Whoville?   \n14226                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14227                                 What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?   \n14228  In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?   \n\n                        answer  \\\n0                          NaN   \n1                          NaN   \n2                          NaN   \n3              (62, 67, Exile)   \n4         (2488, 2494, Cancer)   \n...                        ...   \n14224  (4, 18, National Guard)   \n14225       (161, 167, Grinch)   \n14226      (1908, 1913, Brown)   \n14227      (2032, 2037, Brown)   \n14228      (0, 10, Horseshoes)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \\\n0      Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...   \n1      The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...   \n2      Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...   \n3      `` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...   \n4      Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...   \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...   \n14224  The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...   \n14225  Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...   \n14226  `` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...   \n14227  Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...   \n14228  Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...   \n\n       ans_start_tok  ans_end_tok    ans_tok_text  \n0                512          512                  \n1                512          512                  \n2                512          512                  \n3                 17           17           exile  \n4                486          486          cancer  \n...              ...          ...             ...  \n14224              2            3  national guard  \n14225             39           40          grinch  \n14226            399          399           brown  \n14227            449          449           brown  \n14228              1            2      horseshoes  \n\n[14229 rows x 7 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n      <th>ans_start_tok</th>\n      <th>ans_end_tok</th>\n      <th>ans_tok_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>Which Lloyd Webber musical premiered in the US on 10th December 1993?</td>\n      <td>NaN</td>\n      <td>Andrew Lloyd Webber , Baron Lloyd-Webber ( born 22 March 1948 ) is an English composer and impresario of musical theatre . \\n \\n Several of his musicals have run for more than a decade both in the West End and on Broadway . He has composed 13 musicals , a song cycle , a set of variations , two film scores , and a Latin Requiem Mass . Several of his songs have been widely recorded and were hits outside of their parent musicals , notably `` The Music of the Night '' from The Phantom of the Ope...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>NaN</td>\n      <td>The Prime Minister of the United Kingdom of Great Britain and Northern Ireland is the head of Her Majesty 's Government in the United Kingdom . The prime minister ( informal abbreviation : PM ) and Cabinet ( consisting of all the most senior ministers , most of whom are government department heads ) are collectively accountable for their policies and actions to the Monarch , to Parliament , to their political party and ultimately to the electorate . The , Theresa May , leader of the Conserva...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>Who was the next British Prime Minister after Arthur Balfour?</td>\n      <td>NaN</td>\n      <td>Arthur James Balfour , 1st Earl of Balfour , ( ; 25 July 1848 – 19 March 1930 ) was a British Conservative politician who was the Prime Minister of the United Kingdom from July 1902 to December 1905 , and later Foreign Secretary . \\n \\n Entering Parliament in 1874 , Balfour achieved prominence as Chief Secretary for Ireland , in which position he suppressed agrarian unrest whilst taking measures against absentee landlords . He opposed Irish Home Rule , saying there could be no half-way house...</td>\n      <td>512</td>\n      <td>512</td>\n      <td></td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>Who had a 70s No 1 hit with Kiss You All Over?</td>\n      <td>(62, 67, Exile)</td>\n      <td>`` Kiss You All Over '' is a 1978 song performed by the group Exile . It was written by Mike Chapman and Nicky Chinn . It was included on the band 's album Mixed Emotions , and it featured Jimmy Stokley and guitarist JP Pennington on lead vocals . It was a number one single in the United States , but proved to be Exile 's only big hit in the pop rock market . Billboard ranked it as the No . 5 song for 1978 . In the United Kingdom , the song was released on Mickie Most 's RAK Records , and it...</td>\n      <td>17</td>\n      <td>17</td>\n      <td>exile</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>What claimed the life of singer Kathleen Ferrier?</td>\n      <td>(2488, 2494, Cancer)</td>\n      <td>Kathleen Mary Ferrier , CBE ( 22 April 1912 - 8 October 1953 ) was an English contralto singer who achieved an international reputation as a stage , concert and recording artist , with a repertoire extending from folksong and popular ballads to the classical works of Bach , Brahms , Mahler and Elgar . Her death from cancer , at the height of her fame , was a shock to the musical world and particularly to the general public , which was kept in ignorance of the nature of her illness until afte...</td>\n      <td>486</td>\n      <td>486</td>\n      <td>cancer</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>14224</th>\n      <td>14224</td>\n      <td>With a motto of Always Ready, Always There, what US military branch had it's founding on Dec 14, 1636?</td>\n      <td>(4, 18, National Guard)</td>\n      <td>The National Guard of the United States , part of the reserve components of the United States Armed Forces , is a reserve military force , composed of National Guard military members or units of each state and the territories of Guam , of the Virgin Islands , and of Puerto Rico , as well as of the District of Columbia , for a total of 54 separate organizations . All members of the National Guard of the United States are also members of the militia of the United States as defined by . Nationa...</td>\n      <td>2</td>\n      <td>3</td>\n      <td>national guard</td>\n    </tr>\n    <tr>\n      <th>14225</th>\n      <td>14225</td>\n      <td>Who tried to steal Christmas from the town of Whoville?</td>\n      <td>(161, 167, Grinch)</td>\n      <td>Whoville is a fictional town created by author Theodor Seuss Geisel , under the name Dr. Seuss . Whoville appeared in the books Horton Hears a Who ! and How the Grinch Stole Christmas ! However , there were significant differences between the two renditions . \\n \\n Location \\n \\n The exact location of Whoville seems to vary depending on which book or media is being referenced . \\n According to the book Horton Hears a Who ! , the city of Whoville is located within a floating speck of dust whi...</td>\n      <td>39</td>\n      <td>40</td>\n      <td>grinch</td>\n    </tr>\n    <tr>\n      <th>14226</th>\n      <td>14226</td>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>(1908, 1913, Brown)</td>\n      <td>`` Winter Wonderland '' is a winter song , popularly treated as a Christmastime pop standard , written in 1934 by Felix Bernard ( music ) and Richard B. Smith ( lyricist ) . Through the decades it has been recorded by over 200 different artists . \\n \\n History \\n \\n Dick Smith , a native of Honesdale , Pennsylvania , was reportedly inspired to write the song after seeing Honesdale 's Central Park covered in snow . Smith had written the lyrics while in the West Mountain Sanitarium , being tre...</td>\n      <td>399</td>\n      <td>399</td>\n      <td>brown</td>\n    </tr>\n    <tr>\n      <th>14227</th>\n      <td>14227</td>\n      <td>What is the name of the parson mentioned in the lyrics of the Christmas carol \"Winter Wonderland\"?</td>\n      <td>(2032, 2037, Brown)</td>\n      <td>Frosty 's Winter Wonderland is a 1976 animated Christmas television special produced by Rankin/Bass Productions which originally aired on December 2 , 1976 on ABC . It is the second Frosty special and is a sequel to the 1969 Frosty the Snowman special , also written by Romeo Muller , with narration provided by Andy Griffith . \\n \\n Plot \\n \\n Years have passed since Frosty left for the North Pole , but kept his promise to the children that he would be back again someday . When he hears the n...</td>\n      <td>449</td>\n      <td>449</td>\n      <td>brown</td>\n    </tr>\n    <tr>\n      <th>14228</th>\n      <td>14228</td>\n      <td>In what outdoor sport, sanctioned by the NHPA, do you score 3 points for a ringer, 2 for a leaner, and the closet scores a point?</td>\n      <td>(0, 10, Horseshoes)</td>\n      <td>Horseshoes is an outdoor game played between two people ( or two teams of two people ) using four horseshoes and two throwing targets ( stakes ) set in a sandbox area . The game is played by the players alternating turns tossing horseshoes at stakes in the ground , which are traditionally placed 40 ft apart . Modern games use a more stylized U-shaped bar , about twice the size of an actual horseshoe . \\n \\n Game play \\n \\n Official Rules of the Game of Horseshoes from the NHPA \\n \\n The Nati...</td>\n      <td>1</td>\n      <td>2</td>\n      <td>horseshoes</td>\n    </tr>\n  </tbody>\n</table>\n<p>14229 rows × 7 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Create custom dataset class**\n","\n","Now let's create the custom dataset class which also handle the tokenization part. This way, the tokenization happens during training and validation and we don't consume ram to save all the tokenizations in prior."],"metadata":{"id":"40jHaO2LiFX4"}},{"cell_type":"code","source":["class TriviaQA_Dataset(Dataset):\n","  def __init__(self, data: pd.DataFrame):\n","    self.data = data\n","\n","  def __len__(self):\n","    return len(self.data)\n","\n","  def __getitem__(self, idx):\n","    encoding = tokenizer(text=self.data['corpus'].values[idx], text_pair=self.data['question'].values[idx], max_length=512, padding='max_length', truncation=True, return_tensors='pt').to(device)\n","    return {\n","        'ans_start_tok': torch.tensor(self.data['ans_start_tok'].values[idx], dtype=torch.long, device=device),\n","        'ans_end_tok': torch.tensor(self.data['ans_end_tok'].values[idx], dtype=torch.long, device=device),\n","        'input_ids': encoding['input_ids'][0], \n","        'attention_mask': encoding['attention_mask'][0],\n","        'token_type_ids': encoding['token_type_ids'][0]\n","    }"],"metadata":{"id":"Lg8jIWz0iFmx","execution":{"iopub.status.busy":"2022-03-07T18:13:47.048331Z","iopub.execute_input":"2022-03-07T18:13:47.048591Z","iopub.status.idle":"2022-03-07T18:13:47.056719Z","shell.execute_reply.started":"2022-03-07T18:13:47.048561Z","shell.execute_reply":"2022-03-07T18:13:47.055714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_squad_dataset = TriviaQA_Dataset(data=train_dataset)"],"metadata":{"id":"V1Iwzgz4mgil","execution":{"iopub.status.busy":"2022-03-07T18:13:49.991607Z","iopub.execute_input":"2022-03-07T18:13:49.991868Z","iopub.status.idle":"2022-03-07T18:13:49.995900Z","shell.execute_reply.started":"2022-03-07T18:13:49.991839Z","shell.execute_reply":"2022-03-07T18:13:49.994816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["val_squad_dataset = TriviaQA_Dataset(data=validation_dataset)"],"metadata":{"id":"LC7cg4hGoosl","execution":{"iopub.status.busy":"2022-03-07T18:13:51.398162Z","iopub.execute_input":"2022-03-07T18:13:51.399098Z","iopub.status.idle":"2022-03-07T18:13:51.403643Z","shell.execute_reply.started":"2022-03-07T18:13:51.399051Z","shell.execute_reply":"2022-03-07T18:13:51.402711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Create Dataloaders**"],"metadata":{"id":"EgABQ9XAp6aW"}},{"cell_type":"code","source":["BATCH_SIZE=8"],"metadata":{"id":"cl4-YeWGp_MW","execution":{"iopub.status.busy":"2022-03-07T18:13:52.198054Z","iopub.execute_input":"2022-03-07T18:13:52.198861Z","iopub.status.idle":"2022-03-07T18:13:52.202796Z","shell.execute_reply.started":"2022-03-07T18:13:52.198821Z","shell.execute_reply":"2022-03-07T18:13:52.201711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_dataloader = DataLoader(train_squad_dataset, batch_size=BATCH_SIZE, shuffle=False)"],"metadata":{"id":"FpV37IAZp6p5","execution":{"iopub.status.busy":"2022-03-07T18:13:54.310509Z","iopub.execute_input":"2022-03-07T18:13:54.310820Z","iopub.status.idle":"2022-03-07T18:13:54.319070Z","shell.execute_reply.started":"2022-03-07T18:13:54.310782Z","shell.execute_reply":"2022-03-07T18:13:54.318420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["validation_dataloader = DataLoader(val_squad_dataset, batch_size=1, shuffle=False)"],"metadata":{"id":"Fit_x6PmqHIt","execution":{"iopub.status.busy":"2022-03-07T18:13:55.109910Z","iopub.execute_input":"2022-03-07T18:13:55.110487Z","iopub.status.idle":"2022-03-07T18:13:55.115131Z","shell.execute_reply.started":"2022-03-07T18:13:55.110436Z","shell.execute_reply":"2022-03-07T18:13:55.114110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["On the validation dataloader, i gave 1 as batch size and i'll explain why beyond."],"metadata":{"id":"Hnruurdw3zLZ"}},{"cell_type":"markdown","source":["**Train & Evaluation methods**\n","\n","Now, let's define the training function. It follows the same logic as the previous ones. The only major difference is the way the validation loss is calculated. In particular, for each question in the validation dataset, i take the minimum loss among all the answers because the model's goal is to approach any valid answer as good as possible, and if i consider all answer losses (and even those from answers far from the one that gives the minimum loss), this will add extra noisy error for no reason and confuse the train-validation loss plot. To calculate the validation loss this way, we must take the answers one at a time, so this is the reason i set batch size=1 on the validation dataloader."],"metadata":{"id":"OnxB1D3DqYIM"}},{"cell_type":"code","source":["def train(model: BertForQuestionAnswering, train_dataloader: DataLoader, validation_dataloader: DataLoader, learning_rate: float, epochs: int):\n","  # Define the optimizer\n","  opt = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n","  # Initialize train and validation losses lists\n","  train_losses = []\n","  validation_losses = []\n","  # Train for given # of epochs\n","  for epoch in range(epochs):\n","    model.train()\n","    t_losses = []\n","    for batch in tqdm(train_dataloader):\n","      # Delete previously stored gradients\n","      opt.zero_grad()\n","      # Get loss and outputs from the model\n","      start_positions = batch['ans_start_tok'] # we can do this because we know taht squad trai dataset questions always have 1 question\n","      end_positions = batch['ans_end_tok']\n","      out = model(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'], token_type_ids=batch['token_type_ids'], start_positions=start_positions, end_positions=end_positions)\n","      # Perform backpropagation starting from the loss returned in this epoch\n","      loss = out[0]\n","      loss.backward()\n","      # Update model's weights based on the gradients calculated during backprop\n","      opt.step()\n","      t_losses.append(loss.item())\n","\n","    # Calculate train loss in current epoch\n","    train_loss = np.mean(t_losses) \n","    train_losses.append(train_loss)\n","    with torch.no_grad():\n","      model.eval()\n","      v_losses = []\n","      tmp_losses = []\n","      cur_idx = -1\n","      for val_idx, val_batch in enumerate(tqdm(validation_dataloader)):\n","        start_positions = val_batch['ans_start_tok'] # we can do this because we know taht squad trai dataset questions always have 1 question\n","        end_positions = val_batch['ans_end_tok']\n","        out = model(input_ids=val_batch['input_ids'], attention_mask=val_batch['attention_mask'], token_type_ids=val_batch['token_type_ids'], start_positions=start_positions, end_positions=end_positions)\n","        loss = out[0]\n","        batch_idx = validation_dataset['index'].values[val_idx]\n","        if cur_idx != batch_idx:\n","          cur_idx = batch_idx\n","          if len(tmp_losses) > 0:\n","            v_losses.append(min(tmp_losses))\n","          tmp_losses = []\n","        tmp_losses.append(loss.item())\n","      if len(tmp_losses) > 0:\n","        v_losses.append(min(tmp_losses))\n","      val_loss = np.mean(v_losses) \n","      validation_losses.append(val_loss)\n","\n","    # Print current epoch status\n","    print(f\"Epoch {epoch:3}: Loss = {train_loss:.5f} Val_loss = {val_loss:.5f}\")\n","\n","  return train_losses, validation_losses"],"metadata":{"id":"7xp2O95NSt51","execution":{"iopub.status.busy":"2022-03-07T18:13:57.712867Z","iopub.execute_input":"2022-03-07T18:13:57.713281Z","iopub.status.idle":"2022-03-07T18:13:57.724931Z","shell.execute_reply.started":"2022-03-07T18:13:57.713246Z","shell.execute_reply":"2022-03-07T18:13:57.724246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["And let's also define a function to plot the train and validation losses."],"metadata":{"id":"mWPXXRT-3_aL"}},{"cell_type":"code","source":["def plot_loss_vs_epochs(train_losses, val_losses):\n","  assert(len(train_losses) == len(val_losses))\n","  plt.grid()\n","  plt.plot(list(range(len(train_losses))), train_losses, \"o-\", label=\"Train\")\n","  plt.plot(list(range(len(train_losses))), val_losses, \"o-\", label=\"Validation\")\n","  plt.xlabel(\"Epochs\")\n","  plt.ylabel(\"Loss\")\n","  plt.legend()\n","  plt.show()"],"metadata":{"id":"Hq8DRSvLgevj","execution":{"iopub.status.busy":"2022-03-07T18:19:44.934092Z","iopub.execute_input":"2022-03-07T18:19:44.934356Z","iopub.status.idle":"2022-03-07T18:19:44.940185Z","shell.execute_reply.started":"2022-03-07T18:19:44.934327Z","shell.execute_reply":"2022-03-07T18:19:44.939247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Model Definition**\n","\n","Now let's load the pre-trained bert-base-uncased model."],"metadata":{"id":"4yvh_2pnSn1x"}},{"cell_type":"code","source":["reset_seed()"],"metadata":{"id":"2zGtAdQnSqiV","execution":{"iopub.status.busy":"2022-03-07T18:19:45.739816Z","iopub.execute_input":"2022-03-07T18:19:45.740327Z","iopub.status.idle":"2022-03-07T18:19:45.745565Z","shell.execute_reply.started":"2022-03-07T18:19:45.740292Z","shell.execute_reply":"2022-03-07T18:19:45.744796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["model = BertForQuestionAnswering.from_pretrained(BERT_MODEL_NAME).to(device)\n","model"],"metadata":{"id":"MyKrefB-SsHn","outputId":"09505609-7c10-43b2-84e5-745c119359f5","executionInfo":{"status":"ok","timestamp":1646594801054,"user_tz":-120,"elapsed":1645,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:19:46.419528Z","iopub.execute_input":"2022-03-07T18:19:46.419800Z","iopub.status.idle":"2022-03-07T18:20:00.502070Z","shell.execute_reply.started":"2022-03-07T18:19:46.419752Z","shell.execute_reply":"2022-03-07T18:20:00.501371Z"},"trusted":true,"colab":{"referenced_widgets":["3acda8b7fb4647faa77bd3336d523977"]}},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/420M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"3acda8b7fb4647faa77bd3336d523977"}},"metadata":{}},{"name":"stderr","text":"Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForQuestionAnswering: ['cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias']\n- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n","output_type":"stream"},{"execution_count":34,"output_type":"execute_result","data":{"text/plain":"BertForQuestionAnswering(\n  (bert): BertModel(\n    (embeddings): BertEmbeddings(\n      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n      (position_embeddings): Embedding(512, 768)\n      (token_type_embeddings): Embedding(2, 768)\n      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): BertEncoder(\n      (layer): ModuleList(\n        (0): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (1): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (2): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (3): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (4): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (5): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (6): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (7): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (8): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (9): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (10): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (11): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (qa_outputs): Linear(in_features=768, out_features=2, bias=True)\n)"},"metadata":{}}]},{"cell_type":"markdown","source":["**Training**\n","\n","And now, time for training..."],"metadata":{"id":"bJ6kqc2H4HEo"}},{"cell_type":"code","source":["train_losses, validation_losses = train(model, train_dataloader, validation_dataloader, learning_rate=1e-5, epochs=3)"],"metadata":{"id":"21VOfQ1Pg1zp","outputId":"12c2b610-2ace-4c43-b642-61a9f25f8d10","executionInfo":{"status":"ok","timestamp":1646595124195,"user_tz":-120,"elapsed":320126,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-07T18:20:05.204704Z","iopub.execute_input":"2022-03-07T18:20:05.205292Z","iopub.status.idle":"2022-03-08T00:12:39.851347Z","shell.execute_reply.started":"2022-03-07T18:20:05.205231Z","shell.execute_reply":"2022-03-08T00:12:39.850670Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stderr","text":"100%|██████████| 13831/13831 [1:51:08<00:00,  2.07it/s]\n100%|██████████| 14229/14229 [06:00<00:00, 39.45it/s]\n","output_type":"stream"},{"name":"stdout","text":"Epoch   0: Loss = 2.00640 Val_loss = 0.97488\n","output_type":"stream"},{"name":"stderr","text":"100%|██████████| 13831/13831 [1:51:28<00:00,  2.07it/s]\n100%|██████████| 14229/14229 [06:18<00:00, 37.61it/s]\n","output_type":"stream"},{"name":"stdout","text":"Epoch   1: Loss = 1.25303 Val_loss = 0.91653\n","output_type":"stream"},{"name":"stderr","text":"100%|██████████| 13831/13831 [1:51:32<00:00,  2.07it/s]\n100%|██████████| 14229/14229 [06:06<00:00, 38.85it/s]","output_type":"stream"},{"name":"stdout","text":"Epoch   2: Loss = 0.83732 Val_loss = 1.01230\n","output_type":"stream"},{"name":"stderr","text":"\n","output_type":"stream"}]},{"cell_type":"code","source":["plot_loss_vs_epochs(train_losses, validation_losses)"],"metadata":{"id":"TXx0EVLwmLem","execution":{"iopub.status.busy":"2022-03-08T00:12:39.852905Z","iopub.execute_input":"2022-03-08T00:12:39.856937Z","iopub.status.idle":"2022-03-08T00:12:40.184742Z","shell.execute_reply.started":"2022-03-08T00:12:39.856872Z","shell.execute_reply":"2022-03-08T00:12:40.184096Z"},"trusted":true,"outputId":"fad74e50-5d0c-4e4f-d339-586f8dceab6b"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3pUlEQVR4nO3deVxVdf7H8deHHQFxBRVUXDFTESE3TEVtWmZKMjOtMc1KLdtnbJnfTDWtTjZN47i3aDZOZGZWlrYoiomVuO8b7vsSKAIq8P39ca+KrBfkci7weT4e98G5Z7n3fY/X+7nnfL/ne8UYg1JKqerLzeoASimlrKWFQCmlqjktBEopVc1pIVBKqWpOC4FSSlVzHlYHKK169eqZsLCwMm177tw5/Pz8yjdQOXDVXOC62TRX6Wiu0qmKuVavXn3SGFO/0IXGmEp1i4qKMmWVkJBQ5m2dyVVzGeO62TRX6Wiu0qmKuYBkU8Tnqp4aUkqpak4LgVJKVXNaCJRSqpqrdI3FSqmqQ0TYs2cPWVlZVke5SmBgIFu3brU6RgGO5PLx8SE0NBRPT0+HH1cLgVLKMn5+fgQEBBAWFoaIWB3nsrNnzxIQEGB1jAJKymWM4dSpUxw8eJBmzZo5/LjV4tTQ/LWHiBm3hOGLzhEzbgnz1x6yOpJSCnB3d6du3bouVQQqMxGhbt26pT7CclohEJHGIpIgIltEZLOIPFnIOiIiE0Rkl4hsEJFO5Z1j/tpDvDBvI4dSMwE4lJrJC/M2ajFQykVoEShfZdmfzjwiyAb+ZIxpC3QFxohI23zr3Aq0st9GAlPKO8T477aTeTHnqnmZF3MY/9328n4qpZSqlJxWCIwxR4wxa+zTZ4GtQEi+1foDs+zXO/wM1BKRhuWZ47D9SMDR+Uqp6uPUqVN07NiRjh070qBBA0JCQujYsSMxMTFcuHCh2G2Tk5N54oknKiipc4mpgB+mEZEwIBFoZ4w5k2f+AmCcMeYn+/3FwHPGmOR824/EdsRAcHBwVHx8vMPP/aelGZzKKvgaA73g331c4xLy9PR0/P39rY5RKFfNprlKx1Vz1axZk1atWjm8/jebjvHvhL0cPXOeBjW9eTI2jN+3Cy6XLG+88Qb+/v488cQT5OTk4O7uTnZ2Nh4ertOn5lKukuzatYu0tLSr5sXGxq42xkQXtr7TX6GI+AOfA0/lLQKlYYyZDkwHiI6ONr1793Z4278F2toI8p4eEuBcNmTWDefW9uV6AFImS5cupTSvqSK5ajbNVTqummvt2rUO986Zv/YQf/921+X/y0fOnOfv3+7Cx8eXuMj8JxtKz9vbG29vbx5//HHc3d3ZtGkTMTExDB48mCeffJKsrCx8fX2ZMWMG4eHhLF26lLfffpsFCxbw8ssvs3//flJSUti/fz9PPfWUU44WHO3N5OPjQ2RkpMOP69RCICKe2IrAbGPMvEJWOQQ0znM/1D6v3Fx6g4z/bjuHUjMJqeXLqF7Nmb/2EI/MXsOff9eaMbEttcFKKYv9/evNbDlc9HfFtftTuZCTe9W8zIs5PDt3A5/8ur/Qbdo2qslLt19f6iyHDh0iKSkJd3d3zpw5w/Lly/Hw8ODHH3/kL3/5C59//nmBbbZt20ZCQgJnz54lPDycRx55pFR9+a3ktEIgtk/WD4Ctxph3iljtK+AxEYkHugBpxpgj5Z0lLjKEuMiQq74VDYpuzAvzNvL29zvYdTydcXd1wMez5EMupZQ18heBkuZfi7i4uMunYNLS0hg2bBg7d+5ERLh48WKh2/z+97+/fFQRFBTEsWPHCA0NLfdszuDMI4IYYCiwUUTW2ef9BWgCYIyZCnwL3AbsAjKAB5yY5yo+nu68MyiClkH+jP9uO/tOZzB9aDT1A7wrKoJSKo+SvrnHjFtyuRt4XiG1fPl0VLdyzZJ3qOe//e1vxMbG8sUXX7B3794iT7F5e1/57LjUvlBZOK0Q2BuAiz3fYh8adYyzMpRERBgT25IW9f14+tP19J/4E+8Pu4G2jWpaFUkpVYSxN4cXaO/z9XRn7M3hTn3etLQ0QkJsp5hnzpzp1OeySrW4srgkt7RryGeju5FrYODUJH7YcszqSEqpfOIiQ3hzQHtCavki2I4E3hzQvlwaiovz7LPP8sILLxAZGVmpvuWXhuv0i7JYu5BAvnoshodnJTPy42Seu6UNo3o210ZkpVzIpfY+Z3j55ZcvT589e/bydLdu3dixY8fl+6+99hoAvXv3vnyaKO+2AJs2bXJKRmfRI4I8gmr68Omobvy+fUPGLdzG2LkbOJ+dU/KGSilViekRQT4+nu78Z0gkLYP8effHnew/lcGUP3airr82IiulqiY9IiiEiPBUv9b8Z0gk6w+mEjd5BTuOnS15Q6WUqoS0EBTj9ohGfDqqG1kXcxkwOYmE7cetjqSUUuVOC0EJOjauxVePxdC0bg0enLmKD37aQ0WMz6SUUhVFC4EDGgb68tnobvyubQNeXbCFv3yxkQvZ5X81o1JKWUELgYNqeHkw+b5OjIltwSe/HuD+D38hNaP4YWqVUq4tNjaW77777qp57777Lk8//XSh6/fu3ZvkZNvgyLfddhupqakF1nn55Zd5++23i33e+fPns2XLlsv3X3zxRX788cdSpi8/WghKwc1NGHtzG/51TwRr9qUSN2kFu0+kWx1Lqepjwxz4Vzt4uZbt74Y51/RwQ4YMIf+w9vHx8QwcOLDEbb/99ltq1apVpufNXwheeeUV+vXrV6bHKg9aCMrgzshQPhnZlfTz2cRNWsHynSesjqRU1bdhDnz9BKQdAIzt79dPXFMxGDhwIN98883lH6HZu3cvhw8fZu7cuURHR3P99dfz0ksvFbptWFgYJ0+eBOD111+ndevW9OjRg+3br/z64XvvvccNN9xAREQEd911FxkZGSQlJfHVV18xduxYOnbsyO7duxk+fDhz584FYPHixURGRtK+fXtGjBjB+fPnLz/f66+/TqdOnWjfvj3btm0r8+vOT68jKKOoprWZPyaGhz5KZviMVbx8e1uGdguzOpZSldfC5+HoxqKXH1wFOeevnncxE758DFZ/VPg2DdrDreOKfMg6derQuXNnFi5cSP/+/YmPj2fQoEE8/vjjNG3alJycHPr27cuGDRvo0KFDoY+xevVq4uPjWbduHdnZ2XTq1ImoqCgABgwYwMMPPwzAX//6Vz744AMef/xx7rjjDv7whz8UOPLIyspi+PDhLF68mNatW3P//fczZcoUnnrqKQDq1q3LmjVrmDx5Mm+//Tbvv/9+0furFPSI4BqE1q7B3Ee607t1ff725WZe/HIT2U4YElcpRcEiUNJ8B+U9PRQfH8+QIUP44osv6NSpE5GRkWzevPmq0zj5LV++nDvvvJMaNWpQs2ZN7rjjjsvLNm3axI033kj79u2ZPXs2mzdvLjbL9u3badasGa1btwZg2LBhJCYmXl5+6bGjoqLYu3dvWV9yAXpEcI38vT2Yfn80by3axrTEFPacPMfEezsR6Fs5fpBCKZdRzDd3wNYmkHag4PzAxvDAN2V+2v79+/P000+zZs0aMjIyqFOnDhMmTGD16tXUrl2b4cOHk5WVVabHHj58OPPnzyciIoKZM2eydOnSMueEK0Ndl/cw13pEUA7c3YQXbruOt+7qwM8pp7hz8gr2njxndSylqpa+L4Kn79XzPH1t86+Bv78/sbGxjBgxgiFDhnDmzBn8/PwIDAzk2LFjLFy4sNjte/bsyfz588nMzOTs2bN8/fXXl5edPXuWhg0bcvHiRWbPnn15fkBAwFUD210SHh7O3r172bVrFwAff/wxvXr1uqbX5wgtBOVo0A2N+e+DXfjt3AXiJq9g5e5TVkdSquroMAhun2A7AkBsf2+fYJt/jYYMGcL69esZMmQIERERdOjQgTZt2nDvvfcSExNT7LadOnXinnvuISIigltvvZUbbrjh8rJXX32VLl26EBMTQ5s2bS7PHzx4MOPHjycyMpLdu3dfnu/j48OMGTO4++67ad++PW5ubowePfqaX1+JjDGV6hYVFWXKKiEhoczblsa+k+dMv38uNS1e+MZ88su+EtevqFxl4arZNFfpuGquNWvWWB2hUGfOnLE6QqEczbVly5YC84BkU8Tnqh4ROEGTujX4/NHuxLSsx/PzNvLqgi3k5OqwFEop16SFwElq+njywbBoHogJ44Of9vDQR6s4m1X4j14rpZSVtBA4kYe7Gy/dfj2v39mO5TtPcteUJA6czrA6llIuxeggjuWqLPtTC0EFuK9LU2aN6MyxM+fpP2kFq/aetjqSUi4hJyeHU6dOaTEoJ8YYTp06hY+PT6m20+sIKkj3lvX44tHuPPRRMve99wtvDGjPwKhQq2MpZalz585x9uxZTpxwrWFasrKySv1hWhEcyeXj40NoaOk+W7QQVKDm9f354tEYHv3fav782Xp2HU/n2ZvDrY6llGWMMTRr1szqGAUsXbqUyMhIq2MU4KxcWggqWGANT2Y+0Jm/f72Zqct2s/tEOnc10sNipZR1tBBYwNPdjVf7t6NVUAB//3oz2w640T46k5BaviVvrJRS5cxpjcUi8qGIHBeRTUUsDxSRr0VkvYhsFpEHnJXFFYkIw7qHMeOBzpzMzKX/xBWs2f+b1bGUUtWQM3sNzQRuKWb5GGCLMSYC6A38U0S8nJjHJfVqXZ+/dfWlhpc7g6f/zJfrDlkdSSlVzTitEBhjEoHi+kkaIEBEBPC3r1t+w+lVIo383fhyTAyRjWvxZPw63vl+O7l6JbJSqoKIM/vvikgYsMAY066QZQHAV0AbIAC4xxhT6FiyIjISGAkQHBwclf+n5RyVnp6Ov79/mbZ1pku5snMNs7ZcIPFgNtHB7jzcwRtvd3GJbK5Gc5WO5iqdqpgrNjZ2tTEmutCFRQ1CVB43IAzYVMSygcC/AAFaAnuAmiU9ZmUYdK608ubKzc017yXuNmHPLzB/mLDcHEnNtC6YqRz7zJVortLRXKVzLblw0UHnHgDm2TPusheCNiVsU+WJCA/d2Jz3748m5UQ6d0z8iQ0HU62OpZSqwqwsBPuBvgAiEgyEAykW5nEpfa8L5vNHu+Pp7sagaSv5ZsMRqyMppaooZ3Yf/QRYCYSLyEEReVBERovIpV9ZeBXoLiIbgcXAc8aYk87KUxm1aVCTLx+L4fpGgYz53xomLN6pY7Iopcqd0y4oM8YMKWH5YeB3znr+qqKevzf/e7gLL8zbyDs/7GDX8XTeGtgBH093q6MppaoIvbK4EvD2cOefd0fQMsiftxZtZ//pDKbfH0VQgOsNiqWUqnx0GOpKQkR4tHdLpv4xiu1HzxI3cQWbD6dZHUspVQVoIahkbmnXgM9Gd8MAA6es5LvNR62OpJSq5LQQVELtQgL5ckwMrRsEMPq/q5mydLc2IiulykwLQSUVVNOHT0d25Q8dGvGPRdv402frOZ+dY3UspVQlpI3FlZiPpzsTBnekVZA/7/ywg/2nMpg2NIq6/t5WR1NKVSJ6RFDJiQhP9G3FxHsj2Xgojf6TVrD96FmrYymlKhEtBFXEHzo0Ys6oblzIzmXA5BUs2XbM6khKqUpCC0EVEtG4Fl891oNm9f148KNk3l+eoo3ISqkSaSGoYhoE+jBnVDduub4Br32zlRfmbeRCdq7VsZRSLkwLQRVUw8uDSfd24vE+LYlfdYChH/zCb+cuWB1LKeWitBBUUW5uwp9+F86793Rk7YFU4iavYNfxdKtjKaVckBaCKi4uMoRPHu7KufPZ3Dl5BYk7TlgdSSnlYrQQVANRTWszf0wMIbV8eWDmKj5K2mt1JKWUC9FCUE2E1q7B5490JzY8iJe+2szf5m/iYo42IiultBBUK37eHkwbGsWoXs35+Od9PDBjFWkZF62OpZSymBaCasbdTXjh1usYP7ADv+w5xZ2TV7Dn5DmrYymlLKSFoJq6O7oxsx/qym8ZF4ibtIKk3foroUpVV1oIqrHOzerw5ZgeBAV4c/8Hv/K/X/ZbHUkpZQEtBNVck7o1mPdod3q0qsdfvtjI37/eTLY2IitVrWghUAT4ePLBsBsYEdOMGSv28tCsZM5kaSOyUtWFFgIF2BqRX7y9LW/c2Z6fdp7krslJ7D+VYXUspVQF0EKgrnJvlybMerAzx8+ep/+kn9h+Wn/1TKmqTguBKqB7i3rMHxND7RpevLUqi8+SD1gdSSnlRFoIVKGa1fPji0djaFPHjbFzN/Dmt1vJydXfNlCqKnJaIRCRD0XkuIhsKmad3iKyTkQ2i8gyZ2VRZRNYw5Ono3wY2rUp0xJTGPXxatLPZ1sdSylVzpx5RDATuKWohSJSC5gM3GGMuR6424lZVBl5uAmvxrXjlf7Xk7D9OAOnJHHwN21EVqoqcVohMMYkAqeLWeVeYJ4xZr99/ePOyqKu3f3dwpgx/AYOpWYSN2kFq/f9ZnUkpVQ5EWf+pq2IhAELjDHtCln2LuAJXA8EAP82xswq4nFGAiMBgoODo+Lj48uUJz09HX9//zJt60yumgsKZjucnsu7a7I4nWkY0d6b7o08XCKXq9BcpaO5SudacsXGxq42xkQXutAY47QbEAZsKmLZROBnwA+oB+wEWpf0mFFRUaasEhISyrytM7lqLmMKz3Y6/by5Z1qSafrcAjN+0TaTk5PrErlcgeYqHc1VOteSC0g2RXyuWtlr6CDwnTHmnDHmJJAIRFiYRzmotp8Xs0Z0YfANjZmYsItHZ68h44I2IitVWVlZCL4EeoiIh4jUALoAWy3Mo0rBy8ONNwe056+/v47vtxxl0LSVHEnLtDqWUqoMnNl99BNgJRAuIgdF5EERGS0iowGMMVuBRcAG4FfgfWNMkV1NlesRER66sTnvD4tm78kM+k9cwfoDqVbHUkqVkjN7DQ0xxjQ0xngaY0KNMR8YY6YaY6bmWWe8MaatMaadMeZdZ2VRztWnTTCfP9IdLw83Bk1byYINh62OpJQqBb2yWJWL8AYBfDkmhvYhgTz2v7W8++OOS50ClFIuTguBKjd1/b2Z/XAXBnQK4d0fd/JE/DqyLuqgdUq5Oms6gasqy9vDnX/eHUHr4AD+sWgb+0+d4737owmq6WN1NKVUEfSIQJU7EWF0rxZM/WMUO4+n03/SCjYdSrM6llKqCFoIlNPcfH0DPhvdDQHunrqSRZuOWh1JKVUILQTKqa5vFMj8x2IIbxDA6P+uZlLCLm1EVsrFaCFQThcU4EP8yK7cEdGI8d9t509z1nM+WxuRlXIV2lisKoSPpzv/HtyRVkH+/POHHew7ncG0oVHU8/e2OppS1Z4eEagKIyI83rcVk+/rxObDafSfuIJtR89YHUupak8Lgapwt7VvyJxR3cjOzeWuyUks3nrM6khKVWtaCJQlOoTW4ssxPWhe35+HZiXzXmKKNiIrZREtBMoyDQJ9mDOqG7e2a8Dr327luc83cCE71+pYSlU7WgiUpXy93Jk4pBNP9GnJnOSD/PGDXzh97oLVsZSqVrQQKMu5uQnP/C6cfw/uyLoDqcRNWsHOY2etjqVUtaGFQLmM/h1D+HRkVzIu5DBgchJLtx+3OpJS1YIWAuVSIpvU5svHYgitU4MRM1cxc8UebURWyskcKgQi4icibvbp1iJyh4h4Ojeaqq5Cavkyd3Q3+l4XzMtfb+Gv8zdxMUcbkZVyFkePCBIBHxEJAb4HhgIznRVKKT9vD6b9MYrRvVow+5f9DPvwV1IztBFZKWdwtBCIMSYDGABMNsbcDVzvvFhK2RqRn7+1DW/fHcGqvae5c3ISKSfSrY6lVJXjcCEQkW7AfcA39nnuzomk1NUGRoXyv4e7kpZ5kbhJK1ix66TVkZSqUhwtBE8BLwBfGGM2i0hzIMFpqZTK54awOnw5JoYGgT7c/+GvJOy/aHUkpaoMhwqBMWaZMeYOY8w/7I3GJ40xTzg5m1JXaVynBp8/0p2ererx0ZYLvPzVZrK1EVmpa+Zor6H/iUhNEfEDNgFbRGSsc6MpVVCAjyfvD7uBm8M8mJm0lxEfJXMmS48OlLoWjp4aamuMOQPEAQuBZth6DilV4dzdhCFtvBk3oD1Ju04yYHIS+06dszqWUpWWo4XA037dQBzwlTHmIqBX+ShLDe7chI8f7MLJ9PP0n7SCn1NOWR1JqUrJ0UIwDdgL+AGJItIUKPYXRUTkQxE5LiKbSljvBhHJFpGBDmZR6rJuLeoy/9EY6vp5MfSDX5iz6oDVkZSqdBxtLJ5gjAkxxtxmbPYBsSVsNhO4pbgVRMQd+Ae2i9SUKpOwen7MezSGrs3r8uznG3jj263k5OoBq1KOcrSxOFBE3hGRZPvtn9iODopkjEkETpfw0I8DnwM6upi6JoG+nswYfgPDujVlemIKI2clk34+2+pYSlUK4siAXiLyObbeQh/ZZw0FIowxA0rYLgxYYIxpV8iyEOB/2I4sPrSvN7eIxxkJjAQIDg6Oio+PLzFzYdLT0/H39y/Tts7kqrnAdbMVl2vx/ovM3nqBRn7Ck518qF+j4sZWrIz7y0qaq3SuJVdsbOxqY0x0oQuNMSXegHWOzCtknTBgUxHLPgO62qdnAgMdyRIVFWXKKiEhoczbOpOr5jLGdbOVlCtxx3HT7qVFptMr35tVe05VTChTefeXVTRX6VxLLiDZFPG56uhXpUwR6XHpjojEAJllKEp5RQPxIrIXGAhMFpG4a3xMpQC4sVV9vng0hgAfD+597xfmrTlodSSlXJaHg+uNBmaJSKD9/m/AsGt5YmNMs0vTIjIT26mh+dfymErl1TLIn/ljYnjkv2t4Zs56dh1P58+/C8fNTayOppRLcbTX0HpjTATQAehgjIkE+hS3jYh8AqwEwkXkoIg8KCKjRWT0NadWykG1angx68HODOncmMlLd/PI7NVkXNBGZKXycvSIAABju7r4kmeAd4tZd0gpHnd4aXIoVRqe7m68cWd7WgUF8No3Wxg4ZSXvD4umUS1fq6Mp5RKupTuFHl+rSkNEGNGjGR8Mv4H9pzPoP2kF6w6kWh1LKZdwLYVAr9hRlU5seBDzHu2Oj6cb90xbyVfrD1sdSSnLFVsIROSsiJwp5HYWaFRBGZUqV62DA5j/aAwdQgN54pO1vPPDDnL1SmRVjRVbCIwxAcaYmoXcAowxpWpfUMqV1PX35r8PdWFgVCgTFu/k8fi1ZF7IsTqWUpbQD3NVbXl7uDN+YAdaBfkzbtE2Dp7OYPr90QTX9LE6mlIVquKuvVfKBYkIo3q1YPrQaHYeT6f/xBVsOpRmdSylKpQWAqWAm9oGM3d0d9wEBk5NYuHGI1ZHUqrCaCFQyq5to5rMfyyG6xrW5JHZa5i4ZOelcbGUqtK0ECiVR1CAD5883JW4jo14+/sdPP3pOrIuaiOyqtq0sVipfHw83fnXPR1pGeTP29/vYN/pDKYPjaZ+gLfV0ZRyCj0iUKoQIsJjfVox5b5ObD1yhrhJK9h6pNhfZ1Wq0tJCoFQxbm3fkLmju5OTa7hrShI/bDlmdSSlyp0WAqVK0C4kkC8fi6FlkD8jP05m2rLd2oisqhQtBEo5ILimD5+O7MZt7Rry5sJtjJ27gfPZ2oisqgZtLFbKQb5e7vxnSCQtg/z59+Kd7D+VwZQ/dqKuvzYiq8pNjwiUKgU3N+Hpm1ozYUgk6w6mEjd5BTuOnbU6llLXRI8IlCqDOyIa0bi2LyM/Xs1dk5MY0qUx32w4yqHUTEJ+XsLYm8OJiwyxOqZSDtEjAqXKKLJJbb4cE0OAjwfTE/dwKDUTgEOpmbwwbyPz1x6yOKFSjtFCoNQ1aFTLt9BfaMq8mMP477ZXeB6lykILgVLX6GhaVqHzD6Vm6vAUqlLQQqDUNWpUy7fIZT3+kcCkhF2kZVyswERKlY4WAqWu0dibw/H1dL9qnq+nG4/GtqBto5qM/2473cct5rUFWzhsb0dQypVoryGlrtGl3kHjv9tu6zVUy/eqXkNbDp9heuJuZiTtZWbSXu7o2IhRPVsQ3iDAythKXaaFQKlyEBcZQlxkCEuXLqV3795XLWvbqCbvDo7kzzeH88FPe4j/9QDz1hyiT5sgRvVsTudmdRARa4IrhRNPDYnIhyJyXEQ2FbH8PhHZICIbRSRJRCKclUUpVxBauwYv3X49Sc/34ZmbWrPuQCr3TP+ZOycnsWjTEXJydfwiZQ1nthHMBG4pZvkeoJcxpj3wKjDdiVmUchm1/bx4om8rVjzXh1fj2nH63AVG/3cNN72zjE9+3a89jVSFc1ohMMYkAqeLWZ5kjPnNfvdnINRZWZRyRb5e7gzt2pQlf+rFxHsj8fP24IV5G7nxrQQmL91FWqb2NFIVw1XaCB4EFlodQikreLi78YcOjfh9+4as3H2KqYkpvLVoO5OW7OLeLk0Y0aMZDQOL7qKq1LUSZ46rLiJhwAJjTLti1okFJgM9jDGnilhnJDASIDg4OCo+Pr5MedLT0/H39y/Tts7kqrnAdbNV9Vz7zuSwcM9Ffj2agwBdG3pwWzNPQgLKdhBf1fdXeauKuWJjY1cbY6ILXWiMcdoNCAM2FbO8A7AbaO3oY0ZFRZmySkhIKPO2zuSquYxx3WzVJdf+U+fMS19uMm3+utA0fW6BGTHjV/NLyimTm5traa7yorlK51pyAcmmiM9Vyy4oE5EmwDxgqDFmh1U5lHJljevU4OU7bD2Nnu7XmrUHUhk0bSV3TUniu81HydWeRqocOK2NQEQ+AXoD9UTkIPAS4AlgjJkKvAjUBSbb+1Bnm6IOW5Sq5mr7efFkv1aM7NmcuasPMH15CqM+Xk3z+n6MvLE5d3YKwdvDveQHUqoQTisExpghJSx/CHjIWc+vVFXk6+XO0G5hDOnchIWbjjJ12W6en7eRf/6wgxExzbi3SxMCfT2tjqkqGVfpNaSUKgUPdzduj2jEHzo0JGn3KaYu280/Fm1jUoK9p1FMMxoE+lgdU1USWgiUqsREhJiW9YhpWY9Nh9KYnpjC+8tTmLFiD3EdQxjZszmtgnVMI1U8LQRKVRHtQgKZMCSSsTeH8/7yFD5NPsBnqw/S77ogOtfMobfVAZXL0mGolapiGtepwd/7tyPp+b481a8Vq/f9xhu/ZHHXlCS+155GqhBaCJSqour4efFUv9YkPd+XP17nxbEzWYz8eDX9/rWMT1ft53y2jmmkbLQQKFXF+Xq506+pJ0v/3JsJQyLx9XTnuc83cuM/Epi6bDdnsnRMo+pO2wiUqiY83N24I6IRt3doyE+7TjJtWQrjFm5j4pJd3Gcf0yi4pvY0qo60EChVzYgIN7aqz42t6rPpUBrTElN4b3kKH67Yw52Rtp5GLYO0p1F1oqeGlKrG2oUE8p8hkSwbG8uQzk34av1h+r2TyEMfJbN6X5GjyKsqRguBUorGdWrwSv92rHiuD0/2bUXyvtPcNWUlA6ck8cOWY9rTqIrTQqCUuqyuvzdP39SapOf78PLtbTmSlsXDs5K56V/LmLPqgPY0qqK0ECilCqjh5cHwmGYsG9ubfw/uiLeHO89+voGebyUwTXsaVTlaCJRSRfJwd6N/xxC+eaIHHz/YmZZB/ry5cBsxby7hzYVbOX4my+qIqhxoryGlVIny9jTaeDCNaYm7eS8xhRk/7eXOyBAe7tmclkGu94teyjFaCJRSpdI+NJCJ93Zi/6kM3luewpzkA3yafICb2gYzulcLoprWtjqiKiUtBEqpMmlStwavxrXjqX6t+GjlPmat3MsPW45xQ1htRvVsQZ82Qbi5idUxlQO0jUApdU3q+nvzjL2n0Uu3t+VwahYPzUrm5ncT+Sz5ABeyc62OqEqghUApVS5qeHnwQEwzltp7Gnm4uzF2rq2n0fTE3ZzVnkYuSwuBUqpcedp7Gn37RA8+GtGZ5vX9eOPbbXR/cwnjFm7TnkYuSNsIlFJOISL0al2fXq3rs+FgKtMSU5ieuJsPf9rDgE62nkbKNWghUEo5XYfQWky6txN7T57j/Z9S+Cz5IJ8mH6BTkDs1m/9Gpyba08hKempIKVVhwur58Vpce1Y834fHY1uy7XQOAyYnMWjqShZv1TGNrKJHBEqpClfP35tnfhdOW7fDHPYJ44Of9vDgR8m0DvZnZM8W3BHRCC8P/Z5aUXRPK6Us4+MhjOhh62n07j0dcRPhz5+tp+dbCbyXmEL6+WyrI7qGDXPgX+3otTQO/tXOdr8c6RGBUspynu5uxEWG0L9jI5btOMG0ZSm8/u1WJizZydCuTRkeE0ZQQDX99bQNc+DrJ+BiJgKQdsB2H6DDoHJ5CqcdEYjIhyJyXEQ2FbFcRGSCiOwSkQ0i0slZWZRSlYOI0Ds8iE9GduXLMTH0bFWfqct202NcAi/M20DKiXSrI1YcY+DcKfj+r3Ax8+plFzNh8Svl9lTOPCKYCUwEZhWx/Faglf3WBZhi/6uUUkQ0rsWk+2w9jd5bnsJnqw8Sv+oAN7dtwKhezYmszD2Nsi9A+lE4cwTOHIKzR+DM4St/zxyGs0ch53zRj5F2sNziOK0QGGMSRSSsmFX6A7OMMQb4WURqiUhDY8wRZ2VSSlU+YfX8eP3O9jzVrzWzVu5l1sp9LNp8lM7N6jC6V3Niw4MQcZExjYyB82dsH/Bn7R/ol6fzzDt3ouC2Hj4Q0BBqhkDjzvbpRpA4HjJOFVw/MLTcYovtc9g57IVggTGmXSHLFgDjjDE/2e8vBp4zxiQXsu5IYCRAcHBwVHx8fJnypKen4+/vekPlumoucN1smqt0qlKurGzDsoPZfLf3IqezDKH+wq3NPOnS0AOPchrkrtBcJgevC6l4nz+N9/lTeF04hff5S7fTl6fdcwteOX3RI4Dz3nWvul3wqnPV/WwPfyikoAUdW0b49km45145Oshx82Z7+BiOB/dy+DXFxsauNsZEF7asUjQWG2OmA9MBoqOjTe/evcv0OEuXLqWs2zqTq+YC182muUqnquW6BXglJ5cFGw4zbVkK7208y4L9bjzYoxmDOzfB37uUH20XMq46PbN7/0+0cPe1nbY5c8S27OxRMPl+qtPNw/bNPaAhNIqGgEa2b/E1G9m/0duWeXr64gmUrRT3hg3XweJXMGkHkcBQ3Pu+SNsOg2hbpscryMpCcAhonOd+qH1e+dswBxa/Qq+0g7A2FPq+WG6t7Uopa3i6u3FnZChxHUNYuuME05bt5rVvtjJh8U6GdmvK8O7NqO/vBRmn85ymyX8e3j6dlXrVY7cAOBhg/1BvCPV62f7WbGT/sG9o++tXH9wqoBd+h0HQYRDLnFTQrSwEXwGPiUg8tkbiNKe0D1RA1yullAXsDa5y5gix2YeJbX+Y43X2sG/vLsyKI2QlneaiWyqe5kK+DQX8g2zf2Gs3g6bdr5yPt3/QL1+/ixv73WbJy7KC0wqBiHwC9AbqichB4CXAE8AYMxX4FrgN2AVkAA84JcjiVwrvevXFKFj+T/AJBJ9atr++tfJN25flnfYOKPQ8nlKqHGWdsX9jP1x0w+u54wU2C/LwISigIZmhwezICGXRaV8O59aiYWhzekdH0LpVOPgHg7tnsU+f43HYWa/MJTmz19CQEpYbYIyznv+yorpYmVyo1xqy0mzduE5ss01npQHFNKCLWyHFo7hiUuvqYlLCG1CpKi03x9Zjxn7uvdGhZfDjsnwf+kfgQiHXC/jWuXLuvWGErXfNpW/yl/761gYRfIEIoNHZ83yUtJeJK/fyxtwsujQ7wOjeXvRuXd91ehq5gErRWHxNAkNtp4MKzG8M93xccH5uLlw4C5mptvOGWWn26TTb/bzTl5adOXxlurh+vwCefgUKRpu0LMhcWPDoI39h8fLToxHlui5mFn0O/tJ0vgbX1gC7PcC/ge2DPLgttOxnPx8fclWDK56+pY5UP8CbP98czujeLYj/dT8f/LSHB2asIjw4gFG9mnN7RCM83XWknapfCPq+eLmN4DJPX9v8wrhd+sYfCDQt/fNdzMpXQFKLKCb2+2cOUiv1GKxbbet/XBw3j8JPV5V4Wst+38299K9HKWPyNLgWdgGUfV6+BlcAvAKufJA363Vl2n4+PmnjXrrf1N/pDa7+3h48dGNzhnUP4+v1tp5Gz8xZz9vfbefBG5sz+IbG+JW2p1EVUvVf+aUG4Txdr5zaa8jTBzwbQEADhzf5+VJPgNycK6enijr6yF9YUvdfmZ9bwgBdXgGOtYPkKSxe50/ZiqiHjx6NVEV5r3At8EGfZ16BI928Da5NoWm3gqdpAhqCT81in/7CjrSK6XVj5+nuxoBOodwZGcLS7SeYumw3ry7YYutp1LUpw7qHUT/Au8LyuIqqXwjA6V2vyo2bO9SoY7uVljFwMcOBI5E8heX0nivThZ2TBboDrATcvUrXqJ53Pe+aFfqfXdllnbF/oB+iwZEESFxl/6DPcz7+3AkKtIm5e1/pQRMSDdc1LHg+PqBBpW7vEhFi2wQR2yaItft/Y3piCpOW7mL68hQGRoUS4ZVrdcQKVT0KQXUgYmtD8PKz/WctrZyLtg+Oq05dpbJ9wyrCGwcVLCwZJ+H07iv3819oc3U42zfDEk9r1Sq8sHhUv29oxbrc4JrvPHz+8/F5insbgO3YGlMvXfTUMOJKn/i8H/T2BtfqIrJJbab8MYqUE+m8t3wPc1cf5JPsXJalrmZUzxZENK5ldUSn00KgbNw9wa+u7ZbHkZN1Cb+xd/HbGmP70CmpUT3v9MldV9bLzizyoQHw8C3YwH7mPGR8U3J7SWXr7ntVg2sRA5IVdYWrfwPbh3rQdVcaXO0f9D9vPUDXfnFlanCtLprX9+fNAe15+qZWvPJJIok7T/LtxqN0bV6H0b1a0KsK9zTSQqCunYjtA9c7gKsvFndQ9vkrbSNXFYzfCi8s6UcJTDsKG9bajmJK3d23VjGntWpfvczR0x8lXb2ev8G1qPPxmb8VfGwv/yunZJr1LHgevmYI+NUrtjNA1t6lWgQcFBTgw8DWXowb1uNyT6PhM1bRpoGtp9EfOlS9nkZaCJT1PLxtDY/+QQ5v8svlBvZcW2+r0jSwl7a7b0ntICe2w7rZkHPhytXr8x+BX6eDm2fxDa5+9W0f6LWbQpOuBU/TONDgqpzjUk+j+7vZexol7ubpT9czflHV62lUNV6Fqr7c3Gwfxr61KFt330zHrhW5NJ16ALI22e4X1903NxsOr4HQLlcaXPOfj6/kDa7VhZeHG3dF2Xsa7TjO1GUpl3sa3d/N1tOonn/lbsfSQqCqN09f260U3X0vy8m2FYO3mlPo6ancXBix8JojKtfg5ib0aRNMnzbBrNn/G9OXpTAxYRfTE1O4OzqUh29sTtO6flbHLBMtBEqVlbuHratvkVevl98PhyjX0qlJbaYOjWL3iXTeX57CnFUH+d8v+7m1XUNG9WpOh9BaVkcslarV4qGUFfq+WLAhtrir11WV0aK+P28O6MBPz8UyqlcLEnee4I6JKxgy/WeW7TiBM3/4qzxpIVDqWnUYBLdPgMDGGMQ2jtXtE3SY82okqKYPz93ShqTn+/B/t13HnpPnGPbhr9z67+XMX3uIizmufYGaFgKlykOHQfD0Jpb1ng9Pb9IiUE0F+HjycM/mJD4by9t3R5CTa3jq03X0Hr+UGSv2kHGhhGFgLKKFQCmlypmXhxsDo0L57qmefDAsmpBavvz96y10H7eEd77fzsn0ErotVzBtLFZKKSdxcxP6XhdM3+uCWb3vN6Yt281/EnYxLTGFQdGNeejGZi7R00gLgVJKVYCoprWZfn80u47behp9uuoAs3/Zx63tGzK6ZwvahwZalk0LgVJKVaCWQf6Mu6sDz9zUmg9X7GX2z/v4ZsMRYlrWZVTPFtzYql6Fj2mkbQRKKWWBoJo+PH9rG5Je6MNfbmvDruPp3P/hr9w24Se+XHeI7ArsaaSFQCmlLBTg48nIni1Y/mwf3hrYgYs5uTwZv45e45cy097TaP7aQ8SMW8LwReeIGbeE+WsPlWsGPTWklFIuwMvDjUHRjRnYKZQl244zLXE3L3+9hX8s2sbFHEN2ru3itEOpmbwwbyMAcZEh5fLcekSglFIuxM1N6Nc2mM9Gd+fzR7phDJeLwCWZF3MY/9328nvOcnskpZRS5SqqaR3OZxfeVnA4tYQfdCoFLQRKKeXCGtUq/AeFippfFloIlFLKhY29ORxfz6t/fc7X052xN4eX23M4tRCIyC0isl1EdonI84UsbyIiCSKyVkQ2iMhtzsyjlFKVTVxkCG8OaE+I/QggpJYvbw5oX24NxeDEXkMi4g5MAm4CDgKrROQrY8yWPKv9FZhjjJkiIm2Bb4EwZ2VSSqnKKC4yhLjIEJZe+onWcubMI4LOwC5jTIox5gIQD/TPt44BLv0gayBw2Il5lFJKFUKc9cMJIjIQuMUY85D9/lCgizHmsTzrNAS+B2oDfkA/Y8zqQh5rJDASIDg4OCo+Pr5MmdLT0/H39y/Tts7kqrnAdbNprtLRXKVTFXPFxsauNsZEF7rQGOOUGzAQeD/P/aHAxHzrPAP8yT7dDdgCuBX3uFFRUaasEhISyrytM7lqLmNcN5vmKh3NVTpVMReQbIr4XHXmqaFDQOM890Pt8/J6EJgDYIxZCfgA9ZyYSSmlVD7OLASrgFYi0kxEvIDBwFf51tkP9AUQkeuwFYITTsyklFIqH6e1EQDYu4O+C7gDHxpjXheRV7Adonxl7yn0HuCPreH4WWPM9yU85glgXxkj1QNOlnFbZ3LVXOC62TRX6Wiu0qmKuZoaY+oXtsCphcDViEiyKaqxxEKumgtcN5vmKh3NVTrVLZdeWayUUtWcFgKllKrmqlshmG51gCK4ai5w3Wyaq3Q0V+lUq1zVqo1AKaVUQdXtiEAppVQ+WgiUUqqaqzKFwIEhr71F5FP78l9EJCzPshfs87eLyM0VnOsZEdliH4Z7sYg0zbMsR0TW2W/5L8Zzdq7hInIiz/M/lGfZMBHZab8Nq+Bc/8qTaYeIpOZZ5sz99aGIHBeRTUUsFxGZYM+9QUQ65VnmzP1VUq777Hk2ikiSiETkWbbXPn+diCRXcK7eIpKW59/rxTzLin0PODnX2DyZNtnfU3Xsy5yyv0SksdiG498iIptF5MlC1nHu+6uosScq0w3bBWu7geaAF7AeaJtvnUeBqfbpwcCn9um29vW9gWb2x3GvwFyxQA379COXctnvp1u4v4aTb2wo+/w6QIr9b237dO2KypVv/cexXajo1P1lf+yeQCdgUxHLbwMWAgJ0BX5x9v5yMFf3S88H3Hopl/3+XqCeRfurN7DgWt8D5Z0r37q3A0ucvb+AhkAn+3QAsKOQ/49OfX9VlSMCR4a87g98ZJ+eC/QVEbHPjzfGnDfG7AF22R+vQnIZYxKMMRn2uz9jG5PJ2RzZX0W5GfjBGHPaGPMb8ANwi0W5hgCflNNzF8sYkwicLmaV/sAsY/MzUEtso+s6c3+VmMsYk2R/Xqi495cj+6so1/LeLO9cFfL+MsYcMcassU+fBbYC+X91xqnvr6pSCEKAA3nuH6Tgjry8jjEmG0gD6jq4rTNz5fUgtqp/iY+IJIvIzyISV06ZSpPrLvth6FwRuTSAoEvsL/sptGbAkjyznbW/HFFUdmfur9LK//4ywPcislpsQ71XtG4isl5EForI9fZ5LrG/RKQGtg/Uz/PMdvr+Etsp60jgl3yLnPr+ctovlKnSEZE/AtFArzyzmxpjDolIc2CJiGw0xuyuoEhfA58YY86LyChsR1N9Kui5HTEYmGuMyckzz8r95dJEJBZbIeiRZ3YP+/4KAn4QkW32b8wVYQ22f690sY1JNh9oVUHP7YjbgRXGmLxHD07dXyLij63wPGWMOVNej+uIqnJE4MiQ15fXEREPbL+IdsrBbZ2ZCxHpB/wfcIcx5vyl+caYQ/a/KcBSbN8UKiSXMeZUnizvA1GObuvMXHkMJt9huxP3lyOKyu7M/eUQEemA7d+wvzHm1KX5efbXceALyu+UaImMMWeMMen26W8BTxGphwvsL7vi3l/lvr9ExBNbEZhtjJlXyCrOfX+Vd8OHFTdsRzYp2E4VXGpguj7fOmO4urF4jn36eq5uLE6h/BqLHckVia1xrFW++bUBb/t0PWAn5dRo5mCuhnmm7wR+Nlcap/bY89W2T9epqFz29dpga7iTithfeZ4jjKIbP3/P1Y15vzp7fzmYqwm2dq/u+eb7AQF5ppOw/aJgReVqcOnfD9sH6n77vnPoPeCsXPblgdjaEfwqYn/ZX/cs4N1i1nHq+6vcdq7VN2yt6juwfaj+n33eK9i+ZYPttw4+s/+n+BVonmfb/7Nvtx24tYJz/QgcA9bZb1/Z53cHNtr/I2wEHqzgXG8Cm+3PnwC0ybPtCPt+3AU8UJG57PdfBsbl287Z++sT4AhwEdt52AeB0cBo+3IBJtlzbwSiK2h/lZTrfeC3PO+vZPv85vZ9td7+7/x/FZzrsTzvr5/JU6gKew9UVC77OsOxdSDJu53T9he203UG2JDn3+m2inx/6RATSilVzVWVNgKllFJlpIVAKaWqOS0ESilVzWkhUEqpak4LgVJKVXNaCJSyyzd66bryHPlSRMKKGvFSKavpEBNKXZFpjOlodQilKpoeEShVAvs49G/Zx6L/VURa2ueHicgSufJbEk3s84NF5Av7gGrrRaS7/aHcReQ9+5jz34uIr339J+TKb1LEW/QyVTWmhUCpK3zznRq6J8+yNGNMe2Ai8K593n+Aj4wxHYDZwAT7/AnAMmNMBLax7zfb57cCJhljrgdSgbvs858HIu2PM9o5L02poumVxUrZiUi6Mca/kPl7gT7GmBT74GBHjTF1ReQktjGZLtrnHzHG1BORE0CoyTOAoH144R+MMa3s958DPI0xr4nIIiAd2wic8419MDalKooeESjlGFPEdGmczzOdw5U2ut9jG0emE7DKPjquUhVGC4FSjrknz9+V9ukkbCPZAtwHLLdPL8b2s6OIiLuIBBb1oCLiBjQ2xiQAz2Eb+bLAUYlSzqTfPJS6wldE1uW5v8gYc6kLaW0R2YDtW/0Q+7zHgRkiMhY4ATxgn/8kMF1EHsT2zf8RbCNeFsYd+K+9WAgwwRiTWk6vRymHaBuBUiWwtxFEG2NOWp1FKWfQU0NKKVXN6RGBUkpVc3pEoJRS1ZwWAqWUqua0ECilVDWnhUAppao5LQRKKVXN/T/U32fiiOq3aAAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["**Save the model**\n","\n","After training, i save the model parameters in a file. I make the evaluations in different notebooks."],"metadata":{"id":"WHGtr7vI8hAN"}},{"cell_type":"code","source":["torch.save(model.state_dict(), \"model.pt\")"],"metadata":{"id":"4NYOtGL-mMod","execution":{"iopub.status.busy":"2022-03-08T00:12:40.186161Z","iopub.execute_input":"2022-03-08T00:12:40.186403Z","iopub.status.idle":"2022-03-08T00:12:40.911516Z","shell.execute_reply.started":"2022-03-08T00:12:40.186368Z","shell.execute_reply":"2022-03-08T00:12:40.910815Z"},"trusted":true},"execution_count":null,"outputs":[]}]}