code/Transformers_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q5I_n9-ds2uW"
   },
   "source": [
    "This notebook is part of the \"Literary Metaphor Detection\n",
    "with LLM Fine-Tuning and Few-Shot Learning\" paper. The corresponding repository can be found on [Github](https://github.com/ma-spie/LLM_metaphor_detection)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eacY3XvvtDTz"
   },
   "source": [
    "# Training with Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QJWspUyDufQB"
   },
   "source": [
    "*goals of this notebook*:\n",
    "\n",
    "* fine-tuning the [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) language model on the metaphor detecton task using the [HuggingFace Transformers](https://huggingface.co/docs/transformers/index) framework with four datasets (PoFO, TroFi, MOH, PoFo_TroFi_MOH).\n",
    "\n",
    "The normalisation and analysis of these datasets can be found in the `Preprocessing_analysis.ipynb`notebook.\n",
    "\n",
    "Fine-tuning a sentence transformer model on the same task with the SetFit framework can be found in the `SetFit_training.ipynb notebook`.\n",
    "\n",
    "This notebook is based on the [HuggingFace Task Guide: Text Classification](https://huggingface.co/docs/transformers/tasks/sequence_classification)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RKnWwDbaxxOO"
   },
   "source": [
    "## installations, imports, loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a79zrMSQx9PS",
    "outputId": "14d5b28b-339a-40e3-ffc6-54fb6a8af285"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 23.3.2 -> 24.0\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n",
      "\n",
      "[notice] A new release of pip is available: 23.3.2 -> 24.0\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n",
      "\n",
      "[notice] A new release of pip is available: 23.3.2 -> 24.0\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "#required installations\n",
    "!pip install transformers datasets evaluate --quiet\n",
    "!pip install accelerate -U --quiet\n",
    "!pip install codecarbon --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "z4vQ4JTcyILn"
   },
   "outputs": [],
   "source": [
    "#general imports\n",
    "from google.colab import files     #only needed if Google colab is used\n",
    "import pandas as pd\n",
    "import evaluate\n",
    "import numpy as np\n",
    "from codecarbon import EmissionsTracker\n",
    "\n",
    "#imports for training\n",
    "from datasets import load_dataset\n",
    "from transformers import AutoTokenizer\n",
    "from transformers import DataCollatorWithPadding\n",
    "from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KMVDOEqIy5yp"
   },
   "source": [
    "upload these files from the `folder preprocessed_datasets` from the repository:\n",
    "\n",
    "\n",
    "*   PoFo_normalised.csv\n",
    "*   TroFi_normalised.cs\n",
    "*   MOH-X_normalised.csv\n",
    "*   PoFo_TroFi_MOH.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 180
    },
    "id": "pKk7yp97y7bf",
    "outputId": "08c59b15-f00c-4e5e-c019-8228eeab804d"
   },
   "outputs": [],
   "source": [
    "uploaded_files = files.upload()     #only needed if Google colab is used, otherwis these files have to be in the same folder as this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2TWTTH_uzFg2"
   },
   "source": [
    "To fine-tune DistilBERT on all four datasets uncomment one dataset at a time and run the script a total of four times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "gJr_4lazzIBG"
   },
   "outputs": [],
   "source": [
    "inputfile= \"PoFo_normalised.csv\"\n",
    "#inputfile = \"TroFi_normalised.csv\"\n",
    "#inputfile = \"MOH-X_normalised.csv\"\n",
    "#inputfile = \"PoFo_TroFi_MOH.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "jIR-L-Ej0re_"
   },
   "outputs": [],
   "source": [
    "#load and split dataset\n",
    "dataset = load_dataset(\"csv\", data_files=inputfile, delimiter= \"\\t\")\n",
    "split_dataset = dataset[\"train\"].train_test_split(test_size=0.2, seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SD6voWnW0uqG",
    "outputId": "109a4ec2-1019-42d4-8b6a-8f358588e479"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['text', 'label'],\n",
      "        num_rows: 441\n",
      "    })\n",
      "    test: Dataset({\n",
      "        features: ['text', 'label'],\n",
      "        num_rows: 111\n",
      "    })\n",
      "})\n",
      "{'text': \"maybe these events are nature 's sleight of hand , and the real\", 'label': 'metaphorical'}\n",
      "{'text': 'where books were trees .', 'label': 'metaphorical'}\n",
      "Train dataset size: 441\n",
      "Test dataset size: 111\n"
     ]
    }
   ],
   "source": [
    "#checking outputs\n",
    "print(split_dataset)\n",
    "print(split_dataset[\"train\"][0])\n",
    "print(split_dataset[\"test\"][0])\n",
    "print(\"Train dataset size:\", len(split_dataset[\"train\"]))\n",
    "print(\"Test dataset size:\", len(split_dataset[\"test\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4rMq-EWi1_CR"
   },
   "source": [
    "## Preprocessing data\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yuRZSkci2OKv"
   },
   "source": [
    "The dataset is tokenized and the string labels are binarized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s48RRTlt2OiV",
    "outputId": "02ad522f-4327-47d4-d106-fdd0d6516217"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenizer: DistilBertTokenizerFast(name_or_path='distilbert-base-uncased', vocab_size=30522, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={\n",
      "\t0: AddedToken(\"[PAD]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t100: AddedToken(\"[UNK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t101: AddedToken(\"[CLS]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t102: AddedToken(\"[SEP]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t103: AddedToken(\"[MASK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "print(f\"Tokenizer: {tokenizer}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "FSQgJrzuXvmv"
   },
   "outputs": [],
   "source": [
    "id2label = {0: \"literal\", 1: \"metaphorical\"}\n",
    "label2id = {\"literal\": 0, \"metaphorical\": 1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "FLVIR-EvX3lG"
   },
   "outputs": [],
   "source": [
    "def preprocess_function(examples):\n",
    "      examples[\"label\"] = [label2id[label] for label in examples[\"label\"]]\n",
    "      return tokenizer(examples[\"text\"], truncation=True, max_length=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_bkQgus9X6FW",
    "outputId": "d9e76a79-8e4f-4608-f706-5e1723297ec3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train IDs: [101, 2672, 2122, 2824, 2024, 3267, 1005, 1055, 22889, 7416, 13900, 1997, 2192, 1010, 1998, 1996, 2613, 102]\n",
      "Test IDs: [101, 2073, 2808, 2020, 3628, 1012, 102]\n"
     ]
    }
   ],
   "source": [
    "tokenized_dataset = split_dataset.map(preprocess_function, batched=True)\n",
    "print(f\"Train IDs: {tokenized_dataset['train']['input_ids'][0]}\")\n",
    "print(f\"Test IDs: {tokenized_dataset['test']['input_ids'][0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mryrmvQoYB-Z",
    "outputId": "06da2231-b5a3-4f97-f393-cc4a63bddb87"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data collator: DataCollatorWithPadding(tokenizer=DistilBertTokenizerFast(name_or_path='distilbert-base-uncased', vocab_size=30522, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={\n",
      "\t0: AddedToken(\"[PAD]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t100: AddedToken(\"[UNK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t101: AddedToken(\"[CLS]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t102: AddedToken(\"[SEP]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "\t103: AddedToken(\"[MASK]\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
      "}, padding=True, max_length=None, pad_to_multiple_of=None, return_tensors='pt')\n"
     ]
    }
   ],
   "source": [
    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
    "print(f\"Data collator: {data_collator}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-ejceCME_fj"
   },
   "source": [
    "## Evaluation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hsUy16k6FQAY"
   },
   "source": [
    "Define accuracy, F1 score, precision and recall as metrics for evalaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "f4N6hFpjFgqr"
   },
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "  metrics = evaluate.combine([\"accuracy\", \"f1\", \"precision\", \"recall\"])\n",
    "  predictions, labels = eval_pred\n",
    "  predictions = np.argmax(predictions, axis=1)\n",
    "  metrics_results = metrics.compute(predictions=predictions, references=labels)\n",
    "  return metrics_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZtekAb3iQ8uW"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AXaVRxvPQ92F",
    "outputId": "c58b6e8e-0e99-4172-9a74-82775f6dc098"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "#initialise model\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "        \"distilbert/distilbert-base-uncased\", num_labels=2, id2label=id2label, label2id=label2id\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CFmxkNIOZPXj",
    "outputId": "a0b789f2-dddb-4619-c896-b707bbb20918"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[codecarbon INFO @ 17:28:27] [setup] RAM Tracking...\n",
      "[codecarbon INFO @ 17:28:27] [setup] GPU Tracking...\n",
      "[codecarbon INFO @ 17:28:27] Tracking Nvidia GPU via pynvml\n",
      "[codecarbon INFO @ 17:28:27] [setup] CPU Tracking...\n",
      "[codecarbon WARNING @ 17:28:27] No CPU tracking mode found. Falling back on CPU constant mode.\n",
      "[codecarbon WARNING @ 17:28:28] We saw that you have a AMD Ryzen 9 7950X 16-Core Processor but we don't know it. Please contact us.\n",
      "[codecarbon INFO @ 17:28:28] CPU Model on constant consumption mode: AMD Ryzen 9 7950X 16-Core Processor\n",
      "[codecarbon INFO @ 17:28:28] >>> Tracker's metadata:\n",
      "[codecarbon INFO @ 17:28:28]   Platform system: Windows-10-10.0.22631-SP0\n",
      "[codecarbon INFO @ 17:28:28]   Python version: 3.11.7\n",
      "[codecarbon INFO @ 17:28:28]   CodeCarbon version: 2.3.5\n",
      "[codecarbon INFO @ 17:28:28]   Available RAM : 31.118 GB\n",
      "[codecarbon INFO @ 17:28:28]   CPU count: 32\n",
      "[codecarbon INFO @ 17:28:28]   CPU model: AMD Ryzen 9 7950X 16-Core Processor\n",
      "[codecarbon INFO @ 17:28:28]   GPU count: 1\n",
      "[codecarbon INFO @ 17:28:28]   GPU model: 1 x NVIDIA GeForce RTX 4090\n"
     ]
    }
   ],
   "source": [
    "# start emissions tracker\n",
    "emissions_tracker = EmissionsTracker(save_to_file=True, output_file=\"emissions.csv\", on_csv_write=\"append\")\n",
    "emissions_tracker.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "3FCkRpyXZS_D",
    "outputId": "e6b8d96d-d876-4706-c582-342afa6904bb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing dataset: PoFo_normalised.csv\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\julia\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\accelerate\\accelerator.py:436: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches']). Please pass an `accelerate.DataLoaderConfiguration` instead: \n",
      "dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False)\n",
      "  warnings.warn(\n",
      "[codecarbon INFO @ 17:28:32] [setup] RAM Tracking...\n",
      "[codecarbon INFO @ 17:28:32] [setup] GPU Tracking...\n",
      "[codecarbon INFO @ 17:28:32] Tracking Nvidia GPU via pynvml\n",
      "[codecarbon INFO @ 17:28:32] [setup] CPU Tracking...\n",
      "[codecarbon WARNING @ 17:28:32] No CPU tracking mode found. Falling back on CPU constant mode.\n",
      "[codecarbon WARNING @ 17:28:33] We saw that you have a AMD Ryzen 9 7950X 16-Core Processor but we don't know it. Please contact us.\n",
      "[codecarbon INFO @ 17:28:33] CPU Model on constant consumption mode: AMD Ryzen 9 7950X 16-Core Processor\n",
      "[codecarbon INFO @ 17:28:33] >>> Tracker's metadata:\n",
      "[codecarbon INFO @ 17:28:33]   Platform system: Windows-10-10.0.22631-SP0\n",
      "[codecarbon INFO @ 17:28:33]   Python version: 3.11.7\n",
      "[codecarbon INFO @ 17:28:33]   CodeCarbon version: 2.3.5\n",
      "[codecarbon INFO @ 17:28:33]   Available RAM : 31.118 GB\n",
      "[codecarbon INFO @ 17:28:33]   CPU count: 32\n",
      "[codecarbon INFO @ 17:28:33]   CPU model: AMD Ryzen 9 7950X 16-Core Processor\n",
      "[codecarbon INFO @ 17:28:33]   GPU count: 1\n",
      "[codecarbon INFO @ 17:28:33]   GPU model: 1 x NVIDIA GeForce RTX 4090\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='70' max='70' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [70/70 00:52, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.690000</td>\n",
       "      <td>0.699233</td>\n",
       "      <td>0.477477</td>\n",
       "      <td>0.646341</td>\n",
       "      <td>0.477477</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.665700</td>\n",
       "      <td>0.688708</td>\n",
       "      <td>0.522523</td>\n",
       "      <td>0.662420</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.981132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.617900</td>\n",
       "      <td>0.677654</td>\n",
       "      <td>0.603604</td>\n",
       "      <td>0.694444</td>\n",
       "      <td>0.549451</td>\n",
       "      <td>0.943396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.551100</td>\n",
       "      <td>0.648937</td>\n",
       "      <td>0.648649</td>\n",
       "      <td>0.688000</td>\n",
       "      <td>0.597222</td>\n",
       "      <td>0.811321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.513400</td>\n",
       "      <td>0.654216</td>\n",
       "      <td>0.639640</td>\n",
       "      <td>0.701493</td>\n",
       "      <td>0.580247</td>\n",
       "      <td>0.886792</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Checkpoint destination directory transformers_training_PoFo_normalised.csv\\checkpoint-14 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "[codecarbon INFO @ 17:28:46] Energy consumed for RAM : 0.000049 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:28:46] Energy consumed for all GPUs : 0.000443 kWh. Total GPU Power : 106.0826712779584 W\n",
      "[codecarbon INFO @ 17:28:46] Energy consumed for all CPUs : 0.000177 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:28:46] 0.000669 kWh of electricity used since the beginning.\n",
      "[codecarbon INFO @ 17:28:51] Energy consumed for RAM : 0.000049 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:28:51] Energy consumed for all GPUs : 0.000591 kWh. Total GPU Power : 141.64158791102088 W\n",
      "[codecarbon INFO @ 17:28:51] Energy consumed for all CPUs : 0.000178 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:28:51] 0.000818 kWh of electricity used since the beginning.\n",
      "Checkpoint destination directory transformers_training_PoFo_normalised.csv\\checkpoint-28 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "[codecarbon INFO @ 17:29:01] Energy consumed for RAM : 0.000097 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:01] Energy consumed for all GPUs : 0.001249 kWh. Total GPU Power : 193.49159556461152 W\n",
      "[codecarbon INFO @ 17:29:01] Energy consumed for all CPUs : 0.000355 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:01] 0.001701 kWh of electricity used since the beginning.\n",
      "[codecarbon INFO @ 17:29:06] Energy consumed for RAM : 0.000097 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:06] Energy consumed for all GPUs : 0.001446 kWh. Total GPU Power : 204.9732736436286 W\n",
      "[codecarbon INFO @ 17:29:06] Energy consumed for all CPUs : 0.000355 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:06] 0.001897 kWh of electricity used since the beginning.\n",
      "Checkpoint destination directory transformers_training_PoFo_normalised.csv\\checkpoint-42 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "Checkpoint destination directory transformers_training_PoFo_normalised.csv\\checkpoint-56 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "[codecarbon INFO @ 17:29:16] Energy consumed for RAM : 0.000146 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:17] Energy consumed for all GPUs : 0.002127 kWh. Total GPU Power : 205.6484504440457 W\n",
      "[codecarbon INFO @ 17:29:17] Energy consumed for all CPUs : 0.000537 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:17] 0.002810 kWh of electricity used since the beginning.\n",
      "[codecarbon INFO @ 17:29:21] Energy consumed for RAM : 0.000146 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:21] Energy consumed for all GPUs : 0.002235 kWh. Total GPU Power : 189.47624638979852 W\n",
      "[codecarbon INFO @ 17:29:21] Energy consumed for all CPUs : 0.000532 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:21] 0.002912 kWh of electricity used since the beginning.\n",
      "Checkpoint destination directory transformers_training_PoFo_normalised.csv\\checkpoint-70 already exists and is non-empty.Saving will proceed but saved results may be invalid.\n",
      "[codecarbon INFO @ 17:29:29] Energy consumed for RAM : 0.000171 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:29] Energy consumed for all GPUs : 0.002591 kWh. Total GPU Power : 162.91682990986328 W\n",
      "[codecarbon INFO @ 17:29:29] Energy consumed for all CPUs : 0.000625 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:29] 0.003387 kWh of electricity used since the beginning.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing of PoFo_normalised.csv completed.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Processing dataset: {inputfile}\")\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "        output_dir=f\"transformers_training_{inputfile}\",\n",
    "        seed=42,\n",
    "        learning_rate=2e-5,\n",
    "        per_device_train_batch_size=32,\n",
    "        per_device_eval_batch_size=32,\n",
    "        num_train_epochs=5,\n",
    "        logging_dir=\"logs\",  \n",
    "        logging_strategy=\"epoch\",\n",
    "        evaluation_strategy=\"epoch\",         #Evaluation is done at the end of each epoch.https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.evaluation_strategy\n",
    "        save_strategy=\"epoch\",               #save_strategy has to be the same as evaluation_strategy https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.evaluation_strategy\n",
    "        load_best_model_at_end=True,\n",
    "    )\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_dataset[\"train\"],\n",
    "    eval_dataset=tokenized_dataset[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=data_collator,\n",
    "    compute_metrics=compute_metrics,\n",
    ")\n",
    "\n",
    "trainer.train()\n",
    "\n",
    "print(f\"Processing of {inputfile} completed.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "O3cXe6nhZd2h",
    "outputId": "7dea38de-f23a-47eb-de2a-eba3488de3c7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[codecarbon INFO @ 17:29:29] Energy consumed for RAM : 0.000186 kWh. RAM Power : 11.66927719116211 W\n",
      "[codecarbon INFO @ 17:29:29] Energy consumed for all GPUs : 0.002717 kWh. Total GPU Power : 172.552037212497 W\n",
      "[codecarbon INFO @ 17:29:29] Energy consumed for all CPUs : 0.000682 kWh. Total CPU Power : 42.5 W\n",
      "[codecarbon INFO @ 17:29:29] 0.003585 kWh of electricity used since the beginning.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Emissions as CO₂-equivalents [CO₂eq] in kg for PoFo_normalised.csv: 0.0013816308201039678\n"
     ]
    }
   ],
   "source": [
    "# calculate emissions data for training\n",
    "emissions: float = emissions_tracker.stop()\n",
    "print(f\"Emissions as CO₂-equivalents [CO₂eq] in kg for {inputfile}: {emissions}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8LJUjobeZze5"
   },
   "source": [
    "## Result documentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 254
    },
    "id": "hcvYEA4SZ0Qf",
    "outputId": "da559981-e071-4ab1-cc6a-59cec8cbb6ce"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='4' max='4' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [4/4 00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation results for PoFo_normalised.csv: {'eval_loss': 0.6489366888999939, 'eval_accuracy': 0.6486486486486487, 'eval_f1': 0.688, 'eval_precision': 0.5972222222222222, 'eval_recall': 0.8113207547169812, 'eval_runtime': 6.5954, 'eval_samples_per_second': 16.83, 'eval_steps_per_second': 0.606, 'epoch': 5.0}\n",
      "                         PoFo_normalised.csv\n",
      "eval_loss                           0.648937\n",
      "eval_accuracy                       0.648649\n",
      "eval_f1                             0.688000\n",
      "eval_precision                      0.597222\n",
      "eval_recall                         0.811321\n",
      "eval_runtime                        6.595400\n",
      "eval_samples_per_second            16.830000\n",
      "eval_steps_per_second               0.606000\n",
      "epoch                               5.000000\n"
     ]
    }
   ],
   "source": [
    "# initialize empty dictionary to store results\n",
    "evaluation_results = {}\n",
    "metrics = trainer.evaluate(tokenized_dataset[\"test\"])\n",
    "evaluation_results[inputfile] = metrics\n",
    "print(f\"Evaluation results for {inputfile}: {metrics}\")\n",
    "\n",
    "# convert evaluation results to a DataFrame for easier access to data in further scripts (for example for visualisaton)\n",
    "evaluation_results_df = pd.DataFrame(evaluation_results)\n",
    "evaluation_results_df.to_csv(f\"DistilBERT_evaluation_results_df_{inputfile}.csv\")\n",
    "print(f\"{evaluation_results_df}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UJn2Uo_JbqZq",
    "outputId": "29b80ff9-bcda-4c7b-faae-927affdb77d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trained model saved as PoFo_normalised.csv_model_DistilBERT\n"
     ]
    }
   ],
   "source": [
    "# save the trained model with a unique name for each dataset\n",
    "model_name = f\"{inputfile}_model_DistilBERT\"\n",
    "trainer.save_model(model_name)\n",
    "print(f\"Trained model saved as {model_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "CT99g-TVbqK4"
   },
   "outputs": [],
   "source": [
    "# create a text file for documenting the results\n",
    "with open(f\"DistilBERT_model_evaluation_results_{inputfile}.txt\", \"a\") as file:\n",
    "    file.write(f\"Model: {inputfile}\\n\")\n",
    "    file.write(f\"Evaluation Results: {metrics}\\n\")\n",
    "    file.write(f\"Emissions as CO2-equivalents [CO2eq] in kg for training this model: {emissions}\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uOcEOVHcZ-bF"
   },
   "source": [
    "As a result we get (for each dataset)\n",
    "\n",
    "*   a model fine-tuned on the metaphor identification task - you can find the model on [Zenodo]( https://doi.org/10.5281/zenodo.11624278)\n",
    "\n",
    "*  the file `DistilBERT_model_evaluation_results_inputfile.txt` with evaluation results\n",
    "\n",
    "*  the emissions information in the `emissions.csv` file\n",
    "\n",
    "*  the evaluation results in a DataFrame as a csv file"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}