NAACL_2018_Shared_Task_Metaphor_Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NAACL 2018 Shared Task - Metaphor Detection\n",
    "\n",
    "This notebook implements neural network method for Metaphor Detection using Keras, as part of my [bachelor thesis](https://github.com/martialblog/bachelor-thesis-code). It is based on the [NAACL 2018 Shared Task for Metaphor Detection](https://sites.google.com/site/figlangworkshop/shared-task) but did not compete in the task.\n",
    "\n",
    "For further details on the Shared Task and the training data visit: https://github.com/EducationalTestingService/metaphor/tree/master/NAACL-FLP-shared-task\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "- [Prerequisites](#prerequisites)\n",
    "- [Download VUAM Corpus](#vuamc_generation)\n",
    "- [Generate Training and Test Data](#corpus_generation)\n",
    "- [Validate Training and Test Data](#corpus_validation)\n",
    "- [Keras Model Configuration](#model_configuration)\n",
    "- [Load Word Embeddings](#word_embeddings)\n",
    "- [Keras Model Compilation](#model_compilation)\n",
    "- [Model Training](#training)\n",
    "- [Model Evaluation](#evaluation)\n",
    "- [Plot Training Phase](#training_plot)\n",
    "\n",
    "<a id='prerequisites'></a>\n",
    "## Prerequisites \n",
    "\n",
    "Install the Python 3 requirements from the requirements.txt\n",
    "\n",
    "```\n",
    "pip3 install -r requirements.txt\n",
    "```\n",
    "\n",
    "Download the Word Embeddings for encoding lexical items (Gensim KeyedVectors, or pymagnitude) into the *source/* directory. Example for pymagnitude:\n",
    "\n",
    "```\n",
    "cd source/\n",
    "curl -O http://magnitude.plasticity.ai/fasttext+subword/wiki-news-300d-1M.magnitude\n",
    "curl -O http://magnitude.plasticity.ai/word2vec+subword/GoogleNews-vectors-negative300.magnitude\n",
    "```\n",
    "\n",
    "- https://github.com/plasticityai/magnitude\n",
    "- https://code.google.com/archive/p/word2vec/\n",
    "\n",
    "Download the VUAM Corpus as XML (can't be included due to licence restrictions) into the *starterkits/* directory. **Hint**: There is a cell in this Notebook that will do that. See [VUAM Corpus](#vuamc_generation).\n",
    "\n",
    "```\n",
    "cd starterkits/\n",
    "curl -O http://ota.ahds.ac.uk/headers/2541.xml\n",
    "\n",
    "# Or use the Python functions provided in the utils module\n",
    "python3 -i utils.py\n",
    "download_vuamc_xml()\n",
    "```\n",
    "\n",
    "The VUAMC needs to be converted into a CSV file and placed into the *source/* directory. This is done by using the starterkit scripts provided by the NAACL, which are included in the repository, or a Python function.\n",
    "\n",
    "```\n",
    "cd starterkits/\n",
    "python3 vua_xml_parser.py\n",
    "python3 vua_xml_parser_test.py\n",
    "\n",
    "# Or use the Python functions provided in the utils module\n",
    "python3 -i utils.py\n",
    "generate_vuamc_csv()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing custom modules\n",
    "import utils\n",
    "import corpus\n",
    "import evaluate\n",
    "import features\n",
    "\n",
    "# Import general dependencies\n",
    "import numpy\n",
    "import os\n",
    "import collections\n",
    "from keras.utils import to_categorical\n",
    "from keras.layers import TimeDistributed, Bidirectional, LSTM, Input, Masking, Dense\n",
    "from keras.models import Model\n",
    "from keras import backend as kerasbackend\n",
    "from sklearn.model_selection import KFold\n",
    "from keras.preprocessing.text import Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='vuamc_generation'></a>\n",
    "# VUAM Corpus\n",
    "\n",
    "The VUAMC is the training set for this task. However, it could not be included in the repository due its licence. \n",
    "\n",
    "The next cell will check if the VUAMC is downloaded correctly and will take care of it if necessary. It will also generate the CSV files using the converter provided by the NAACL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('source/vuamc_corpus_test.csv') and not os.path.exists('source/vuamc_corpus_train.csv'):\n",
    "    print('VUAMC training and test data not found. Generating...')\n",
    "    utils.download_vuamc_xml()\n",
    "    utils.generate_vuamc_csv()\n",
    "    print('VUAMC CSV generated')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='corpus_generation'></a>\n",
    "# Test and Training Corpus\n",
    "\n",
    "The next cell will convert the CSV files for the training and testing into a *Corpus* object. This is to manage the sentences in the given corpus during runtime and to provide functions such as: list all labels, list all tokens, etc.\n",
    "\n",
    "The validation checks if the tokens in the corpus and the tokens in the training/test files align."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Train Corpus from CSV\n",
    "# c_train = corpus.VUAMC('source/vuamc_corpus_train.csv', 'source/verb_tokens_train_gold_labels.csv', 'source/vuamc_corpus_train_pos.csv')\n",
    "c_train = corpus.VUAMC('source/vuamc_corpus_train.csv', 'source/all_pos_tokens_train_gold_labels.csv', 'source/vuamc_corpus_train_pos.csv')\n",
    "c_train.validate_corpus()\n",
    "print('Loaded and validated training corpus')\n",
    "\n",
    "# Load Test Corpus from CSV\n",
    "# c_test = corpus.VUAMC('source/vuamc_corpus_test.csv', 'source/verb_tokens_test.csv', 'source/vuamc_corpus_test_pos.csv', mode='test')\n",
    "c_test = corpus.VUAMC('source/vuamc_corpus_test.csv', 'source/all_pos_tokens_test.csv', 'source/vuamc_corpus_test_pos.csv', mode='test')\n",
    "c_test.validate_corpus()\n",
    "print('Loaded and validated test corpus')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='corpus_validation'></a>\n",
    "# Corpus Validation\n",
    "\n",
    "For the training of the model we will use a binary classification, using 0 to encode non-metaphor tokens and 1 to encode metaphor tokens. \n",
    "\n",
    "This next cell will demonstrate that the training data is highly imbalanced. The training set includes a significantly higher amount of non-metaphor tokens. A fact that will cause the training to fail, since the model will almost always choose a 0, because this way it is still right almost all the time.\n",
    "\n",
    "To mitigate the imbalance a *weighted_categorical_crossentropy* will be introducted later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_all_labels = len(c_train.label_list)\n",
    "count_of_label_classes = collections.Counter(c_train.label_list)\n",
    "\n",
    "percentage_of_non_metaphor_tokens = round(count_of_label_classes[0] / number_of_all_labels * 100)\n",
    "percentage_of_metaphor_tokens = round(count_of_label_classes[1] / number_of_all_labels * 100)\n",
    "ratio = utils.simplify_ratio(percentage_of_non_metaphor_tokens, percentage_of_metaphor_tokens)\n",
    "assert(percentage_of_non_metaphor_tokens + percentage_of_metaphor_tokens == 100)\n",
    "\n",
    "print('Percentage of metaphor tokens: {}%'.format(percentage_of_metaphor_tokens))\n",
    "print('Percentage of non-metaphor tokens: {}%'.format(percentage_of_non_metaphor_tokens))\n",
    "print('Ratio: {}:{}'.format(ratio[0], ratio[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model_configuration'></a>\n",
    "# Model Configuration\n",
    "\n",
    "The next cell is the primary configuration for the model. Change the parameters here to change the training.\n",
    "\n",
    "## Weighted Categorical Crossentropy\n",
    "\n",
    "As described above, the training set is highly imbalanced. Therefore, we will use a *weighted categorical crossentropy* to calculate the loss during the training.  The weights for the classes can be adjusted using the **WEIGHT_SMOOTHING** constant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SENTENCE_LENGTH = 50\n",
    "WEIGHT_SMOOTHING = 0.0\n",
    "EMBEDDING_DIM = 300\n",
    "KFOLD_SPLIT = 8\n",
    "KERAS_OPTIMIZER = 'rmsprop'\n",
    "KERAS_METRICS = [utils.precision, utils.recall, utils.f1]\n",
    "KERAS_EPOCHS = 5\n",
    "KERAS_BATCH_SIZE = 32\n",
    "KERAS_ACTIVATION = 'softmax'\n",
    "KERAS_DROPOUT = 0.25\n",
    "\n",
    "# help(get_class_weights) for details\n",
    "class_weights =  list(utils.get_class_weights(c_train.label_list, WEIGHT_SMOOTHING).values())\n",
    "print('loss_weight {}'.format(class_weights))\n",
    "KERAS_LOSS = utils.weighted_categorical_crossentropy(class_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='word_embeddings'></a>\n",
    "# Word Embeddings\n",
    "\n",
    "The model uses Word Embeddings to encode lexical items as real number vectors. The next cell will load the Embeddings for the training and test corpus. \n",
    "\n",
    "This is done by using a polymorph class that implements the *Embeddings* interface. That way, changing embeddings is as simple as changing the *Embeddings Object*. Some examples are given in the comments. \n",
    "\n",
    "After both corpora are encoded, the embeddings object is deleted to free up some memory (some embedding libraries use lazy loading, which would not use up memory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to use different Embeddings\n",
    "# embeddings = features.Word2Vec()\n",
    "# embeddings = features.Magnitudes(filepath='customembeddings.magnitude')\n",
    "# embeddings = features.DummyEmbeddings(EMBEDDING_DIM)\n",
    "embeddings = features.Magnitudes()\n",
    "\n",
    "x_input, y_labels, z_postags = features.generate_input_and_labels(c_train.sentences, Vectors=embeddings)\n",
    "x_test, y_test, z_testtags = features.generate_input_and_labels(c_test.sentences, Vectors=embeddings)\n",
    "print('Generated Word Embeddings')\n",
    "\n",
    "# Free up some memory\n",
    "del embeddings\n",
    "print('Deleted Embeddings Object')\n",
    "\n",
    "# POS Tags to numerical sequences\n",
    "pos_tokenizer = Tokenizer()\n",
    "pos_tokenizer.fit_on_texts(z_postags)\n",
    "pos_sequences = pos_tokenizer.texts_to_sequences(z_postags)\n",
    "pos_test_sequences = pos_tokenizer.texts_to_sequences(z_testtags)\n",
    "\n",
    "# Training labels need to be categorical, with 2 classes (0-non-metaphor, 1-metaphor)\n",
    "y_labels = to_categorical(y_labels, 2)\n",
    "z_pos = to_categorical(pos_sequences)\n",
    "z_test = to_categorical(pos_test_sequences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model_compilation'></a>\n",
    "# The Model\n",
    "\n",
    "This cell compiles the model used in the Task.\n",
    "\n",
    " - Input: The input layer will receive the encoded sentences. Shape: Sentence Length * Embedding Dimensions\n",
    " - POS Tags: POS tags can be excluded by removing the Input layer for them\n",
    " - Core: The core of the model is a bidirectionsal LSTM with a recurrent dropout\n",
    " - Output: The output layer is dense time distributed series with predicions for 2 classes (0|1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "postags = Input(shape=(MAX_SENTENCE_LENGTH, 17))\n",
    "sentences = Input(shape=(MAX_SENTENCE_LENGTH, EMBEDDING_DIM))\n",
    "model = Masking(mask_value=[-1] * EMBEDDING_DIM)(sentences)\n",
    "model = Bidirectional(LSTM(100, return_sequences=True, dropout=0, recurrent_dropout=KERAS_DROPOUT))(model)\n",
    "outputs = TimeDistributed(Dense(2, activation=KERAS_ACTIVATION))(model)\n",
    "model = Model(inputs=[sentences, postags], outputs=outputs)\n",
    "# Model with out POS Tags:\n",
    "# model = Model(inputs=[sentences], outputs=outputs)\n",
    "\n",
    "model.compile(optimizer=KERAS_OPTIMIZER, loss=KERAS_LOSS, metrics=KERAS_METRICS)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='training'></a>\n",
    "# Generate Training and Validation split\n",
    "\n",
    "To further optimize the training we will use a k-fold split on the training data. This will split the input data and labels *k* times (into tr) and fit the model each time on the subset.\n",
    "\n",
    "**Hint:** If the model should not use POS tags, the input and validation data needs to be removed here as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=KFOLD_SPLIT, shuffle=True, random_state=1337)\n",
    "histories = []\n",
    "\n",
    "for train, test in kfold.split(x_input, y_labels):\n",
    "    x_train = x_input[train]\n",
    "    x_val = x_input[test]\n",
    "    y_train = y_labels[train]\n",
    "    y_val = y_labels[test]\n",
    "    pos_val = z_pos[test]\n",
    "    pos_train = z_pos[train]\n",
    "\n",
    "    # Fit the model for each split\n",
    "    history = model.fit([x_train, pos_train], y_train,\n",
    "                  batch_size=KERAS_BATCH_SIZE,\n",
    "                  epochs=KERAS_EPOCHS,\n",
    "                  validation_data=([x_val, pos_val], y_val))\n",
    "    \n",
    "    histories.append(history)\n",
    "\n",
    "    # Evaluation after each split\n",
    "    scores = model.evaluate([x_val, pos_val], y_val)\n",
    "    print('Loss: {:.2%}'.format(scores[0]))\n",
    "    print('Precision: {:.2%}'.format(scores[1]))\n",
    "    print('Recall: {:.2%}'.format(scores[2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='evaluation'></a>\n",
    "# Prediction and Evalutation\n",
    "\n",
    "To evalutate the model, we will use the test corpus and generate predictions (labels) for the input sentences. Each sentence will receive a list of binary classes (0|1) for its tokens. \n",
    "\n",
    "The predictions will be saved in a CSV file, which will be similar to the *Gold Labels* from the NAACL. Using both of these files (predicitions and gold-standards) we will evalutate the perfomance of the model. \n",
    "\n",
    "The Performance is measured in Precision, Recall and F1 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get float predictions and turn them into binaries\n",
    "float_predictions = model.predict([x_test, z_test], batch_size=KERAS_BATCH_SIZE)\n",
    "\n",
    "# Without POS tags\n",
    "# float_predictions = model.predict([x_test], batch_size=KERAS_BATCH_SIZE)\n",
    "\n",
    "binary_predictions = kerasbackend.argmax(float_predictions)\n",
    "label_predictions = kerasbackend.eval(binary_predictions)\n",
    "\n",
    "# Write prediction to CSV file\n",
    "predictions_file = 'fasttest_all_predictions_pos.csv'\n",
    "# standard_file = 'source/verb_tokens_test_gold_labels.csv'\n",
    "standard_file = 'source/all_pos_tokens_test_gold_labels.csv'\n",
    "\n",
    "# Write the predictions.csv and compare to gold standard\n",
    "rows = evaluate.corpus_evaluation(c_test, label_predictions, MAX_SENTENCE_LENGTH)\n",
    "evaluate.csv_evalutation(rows, predictions_file)\n",
    "results = evaluate.precision_recall_f1(predictions_file, standard_file)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='training_plot'></a>\n",
    "# Model Training Plot\n",
    "\n",
    "The following plot shows the learning of the model during the training epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly \n",
    "plotly.offline.init_notebook_mode(connected=True)\n",
    "\n",
    "loss_p = plotly.graph_objs.Scatter(\n",
    "    y = [history.history['loss'][0] for history in histories],\n",
    "    mode = 'lines+markers',\n",
    "    name = 'Loss'\n",
    ")\n",
    "\n",
    "val_loss_p = plotly.graph_objs.Scatter(\n",
    "    y = [history.history['val_loss'][0] for history in histories],\n",
    "    mode = 'lines+markers',\n",
    "    name = 'Validation Loss'\n",
    ")\n",
    "\n",
    "acc_p = plotly.graph_objs.Scatter(\n",
    "    y = [history.history['f1'][0] for history in histories],\n",
    "    mode = 'lines+markers',\n",
    "    name = 'Categorical Accuracy'\n",
    ")\n",
    "\n",
    "val_acc_p = plotly.graph_objs.Scatter(\n",
    "    y = [history.history['val_f1'][0] for history in histories],\n",
    "    mode = 'lines+markers',\n",
    "    name = 'Validation Categorical Accuracy'\n",
    ")\n",
    "\n",
    "layout = plotly.graph_objs.Layout(title=\"Training History\",\n",
    "                yaxis=dict(title='Value'),\n",
    "                xaxis=dict(title='Epoch'))\n",
    "\n",
    "data = [loss_p, val_loss_p, acc_p, val_acc_p]\n",
    "fig = plotly.graph_objs.Figure(data=data, layout=layout)\n",
    "\n",
    "plotly.offline.iplot(fig, filename='jupyter-train-history')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}