AutoML_Batch_Learning_Dataset_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Automated Machine Learning\n",
    "This is the code for the paper entitled \"**[IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective](https://arxiv.org/abs/2209.08018)**\" published in *Engineering Applications of Artificial Intelligence* (Elsevier's Journal, IF:7.8).<br>\n",
    "Authors: Li Yang (lyang339@uwo.ca) and Abdallah Shami (Abdallah.Shami@uwo.ca)<br>\n",
    "Organization: The Optimized Computing and Communications (OC2) Lab, ECE Department, Western University\n",
    "\n",
    "L. Yang and A. Shami, \"IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective\", *Engineering Applications of Artificial Intelligence*, vol. 116, pp. 1-33, 2022, doi: https://doi.org/10.1016/j.engappai.2022.105366."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code Part 1: Automated Offline/Static/Batch Learning\n",
    "Batch learning: Batch learning methods analyze static data in batches and often need access to the entire dataset prior to model training. Traditional ML algorithms can effectively solve batch learning tasks. Although batch learning models often achieve high performance due to their ability to learn diverse data patterns, it is often difficult to update these models once created. Therefore, batch learning faces two significant challenges: model degradation and data unavailability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset 1: CICIDS2017\n",
    "A subset of the network traffic data randomly sampled from the [CICIDS2017 dataset](https://www.unb.ca/cic/datasets/ids-2017.html).  \n",
    "\n",
    "The Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) dataset has the most updated network threats. The CICIDS2017 dataset is close to real-world network data since it has a large amount of network traffic data, a variety of network features, various types of attacks, and highly imbalanced classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from scipy.stats import shapiro\n",
    "from imblearn.over_sampling import SMOTE\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the sampled CICIDS2017 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"Data/cic_0.01km.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Bwd Packet Length Min</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Bwd Header Length</th>\n",
       "      <th>Fwd Packets/s</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Min Packet Length</th>\n",
       "      <th>URG Flag Count</th>\n",
       "      <th>Down/Up Ratio</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>min_seg_size_forward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.083300e+04</td>\n",
       "      <td>50833</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>19.672260</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>319</td>\n",
       "      <td>153</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.900000e+01</td>\n",
       "      <td>49</td>\n",
       "      <td>49</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>40816.326530</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>277</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>306</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>3.060000e+02</td>\n",
       "      <td>306</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>3267.973856</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>63041</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>65.0000</td>\n",
       "      <td>124</td>\n",
       "      <td>124</td>\n",
       "      <td>6.304100e+04</td>\n",
       "      <td>63041</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>15.862693</td>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47682</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43.0000</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "      <td>4.768200e+04</td>\n",
       "      <td>47682</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>20.972275</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4.500000e+01</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>22222.222220</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>349</td>\n",
       "      <td>307</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>114309573</td>\n",
       "      <td>511</td>\n",
       "      <td>427</td>\n",
       "      <td>31.9375</td>\n",
       "      <td>746</td>\n",
       "      <td>0</td>\n",
       "      <td>3.941709e+06</td>\n",
       "      <td>94</td>\n",
       "      <td>165</td>\n",
       "      <td>332</td>\n",
       "      <td>424</td>\n",
       "      <td>0.139971</td>\n",
       "      <td>0.122474</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8192</td>\n",
       "      <td>343</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>48850</td>\n",
       "      <td>80</td>\n",
       "      <td>40</td>\n",
       "      <td>40.0000</td>\n",
       "      <td>72</td>\n",
       "      <td>72</td>\n",
       "      <td>1.628333e+04</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>64</td>\n",
       "      <td>64</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40.941658</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>260</td>\n",
       "      <td>66</td>\n",
       "      <td>33</td>\n",
       "      <td>33.0000</td>\n",
       "      <td>97</td>\n",
       "      <td>97</td>\n",
       "      <td>8.666667e+01</td>\n",
       "      <td>48</td>\n",
       "      <td>48</td>\n",
       "      <td>40</td>\n",
       "      <td>40</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>7692.307692</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>2.500000e+01</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>40000.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11256</td>\n",
       "      <td>35000</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0              50833                            0                      0   \n",
       "1                 49                            0                      0   \n",
       "2                306                            6                      6   \n",
       "3              63041                           65                     65   \n",
       "4              47682                           43                     43   \n",
       "...              ...                          ...                    ...   \n",
       "28298             45                            0                      0   \n",
       "28299      114309573                          511                    427   \n",
       "28300          48850                           80                     40   \n",
       "28301            260                           66                     33   \n",
       "28302             25                            6                      6   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Bwd Packet Length Min  \\\n",
       "0                      0.0000                      0                      0   \n",
       "1                      0.0000                      0                      0   \n",
       "2                      6.0000                      6                      6   \n",
       "3                     65.0000                    124                    124   \n",
       "4                     43.0000                     59                     59   \n",
       "...                       ...                    ...                    ...   \n",
       "28298                  0.0000                      0                      0   \n",
       "28299                 31.9375                    746                      0   \n",
       "28300                 40.0000                     72                     72   \n",
       "28301                 33.0000                     97                     97   \n",
       "28302                  6.0000                      6                      6   \n",
       "\n",
       "       Flow IAT Mean  Flow IAT Min  Fwd IAT Min  Fwd Header Length  \\\n",
       "0       5.083300e+04         50833            0                 32   \n",
       "1       4.900000e+01            49           49                 64   \n",
       "2       3.060000e+02           306            0                 20   \n",
       "3       6.304100e+04         63041            0                 32   \n",
       "4       4.768200e+04         47682            0                 32   \n",
       "...              ...           ...          ...                ...   \n",
       "28298   4.500000e+01            45            0                 32   \n",
       "28299   3.941709e+06            94          165                332   \n",
       "28300   1.628333e+04             1           48                 64   \n",
       "28301   8.666667e+01            48           48                 40   \n",
       "28302   2.500000e+01            25            0                 20   \n",
       "\n",
       "       Bwd Header Length  Fwd Packets/s  Bwd Packets/s  Min Packet Length  \\\n",
       "0                     32      19.672260      19.672260                  0   \n",
       "1                      0   40816.326530       0.000000                  0   \n",
       "2                     20    3267.973856    3267.973856                  6   \n",
       "3                     32      15.862693      15.862693                 65   \n",
       "4                     32      20.972275      20.972275                 43   \n",
       "...                  ...            ...            ...                ...   \n",
       "28298                 32   22222.222220   22222.222220                  0   \n",
       "28299                424       0.139971       0.122474                  0   \n",
       "28300                 64      40.941658      40.941658                 40   \n",
       "28301                 40    7692.307692    7692.307692                 33   \n",
       "28302                 20   40000.000000   40000.000000                  6   \n",
       "\n",
       "       URG Flag Count  Down/Up Ratio  Init_Win_bytes_forward  \\\n",
       "0                   1              1                     319   \n",
       "1                   0              0                     277   \n",
       "2                   0              1                       0   \n",
       "3                   0              1                      -1   \n",
       "4                   0              1                      -1   \n",
       "...               ...            ...                     ...   \n",
       "28298               1              1                     349   \n",
       "28299               0              0                    8192   \n",
       "28300               0              1                      -1   \n",
       "28301               0              1                      -1   \n",
       "28302               1              1                   11256   \n",
       "\n",
       "       Init_Win_bytes_backward  min_seg_size_forward  Label  \n",
       "0                          153                    32      0  \n",
       "1                           -1                    32      0  \n",
       "2                            0                    20      0  \n",
       "3                           -1                    32      0  \n",
       "4                           -1                    32      0  \n",
       "...                        ...                   ...    ...  \n",
       "28298                      307                    32      0  \n",
       "28299                      343                    20      0  \n",
       "28300                       -1                    32      0  \n",
       "28301                       -1                    20      0  \n",
       "28302                    35000                    20      0  \n",
       "\n",
       "[28303 rows x 20 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Automated Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Transformation/Encoding\n",
    "Automatically identify and transform string/text features into numerical features to make the data more readable by ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data encoding function\n",
    "def Auto_Encoding(df):\n",
    "    cat_features=[x for x in df.columns if df[x].dtype==\"object\"] ## Find string/text features\n",
    "    le=LabelEncoder()\n",
    "    for col in cat_features:\n",
    "        if col in df.columns:\n",
    "            i = df.columns.get_loc(col)\n",
    "            # Transform to numerical features\n",
    "            df.iloc[:,i] = df.apply(lambda i:le.fit_transform(i.astype(str)), axis=0, result_type='expand')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Encoding(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Imputation\n",
    "Detect and impute missing values to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the automated data imputation function\n",
    "def Auto_Imputation(df):\n",
    "    if df.isnull().values.any() or np.isinf(df).values.any(): # if there is any empty or infinite values\n",
    "        df.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "        df.fillna(0, inplace = True)  # Replace empty values with zeros; there are other imputation methods discussed in the paper\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=Auto_Imputation(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated normalization\n",
    "Normalize the range of features to a similar scale to improve data quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def Auto_Normalization(df):\n",
    "    stat, p = shapiro(df)\n",
    "    print('Statistics=%.3f, p=%.3f' % (stat, p))\n",
    "    # interpret\n",
    "    alpha = 0.05\n",
    "    numeric_features = df.drop(['Label'],axis = 1).dtypes[df.dtypes != 'object'].index\n",
    "    \n",
    "    # The selection strategy is based on the following article: \n",
    "    # https://medium.com/@kumarvaishnav17/standardization-vs-normalization-in-machine-learning-3e132a19c8bf\n",
    "    # Check if the data distribution follows a Gaussian/normal distribution\n",
    "    # If so, select the Z-score normalization method; otherwise, select the min-max normalization\n",
    "    # Details are in the paper\n",
    "    if p > alpha:\n",
    "        print('Sample looks Gaussian (fail to reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.mean()) / (x.std()))\n",
    "        print('Z-score normalization is automatically chosen and used')\n",
    "    else:\n",
    "        print('Sample does not look Gaussian (reject H0)')\n",
    "        df[numeric_features] = df[numeric_features].apply(\n",
    "            lambda x: (x - x.min()) / (x.max()-x.min()))\n",
    "        print('Min-max normalization is automatically chosen and used')\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Statistics=0.076, p=0.000\n",
      "Sample does not look Gaussian (reject H0)\n",
      "Min-max normalization is automatically chosen and used\n"
     ]
    }
   ],
   "source": [
    "df=Auto_Normalization(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Train-test split\n",
    "Split the dataset into the training and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['Label'],axis=1)\n",
    "y = df['Label']\n",
    "\n",
    "# Here we used the 80%/20% split, it can be changed based on specific tasks\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated data balancing\n",
    "Generate minority class samples to solve class-imbalance and improve data quality.  \n",
    "Synthetic Minority Over-sampling Technique (SMOTE) method is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    18126\n",
       "1     4516\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For binary data (can be modified for multi-class data with the same logic)\n",
    "def Auto_Balancing(X_train, y_train):\n",
    "    number0 = pd.Series(y_train).value_counts().iloc[0]\n",
    "    number1 = pd.Series(y_train).value_counts().iloc[1]\n",
    "    \n",
    "    if number0 > number1:\n",
    "        nlarge = number0\n",
    "    else:\n",
    "        nlarge = number1\n",
    "    \n",
    "    # evaluate whether the incoming dataset is imbalanced (the abnormal/normal ratio is smaller than a threshold (e.g., 50%)) \n",
    "    if (number1/number0 > 1.5) or (number0/number1 > 1.5):\n",
    "        smote=SMOTE(n_jobs=-1,sampling_strategy={0:nlarge, 1:nlarge})\n",
    "        X_train, y_train = smote.fit_resample(X_train, y_train)\n",
    "        \n",
    "    return X_train, y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    18126\n",
       "0    18126\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(y_train).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Model learning (for Comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.788%\n",
      "Precision: 99.37899999999999%\n",
      "Recall: 99.556%\n",
      "F1-score: 99.467%\n",
      "Time: 2.81878\n",
      "Wall time: 338 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lg = lgb.LGBMClassifier(verbose = -1)\n",
    "lg.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = lg.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.77000000000001%\n",
      "Precision: 99.466%\n",
      "Recall: 99.378%\n",
      "F1-score: 99.422%\n",
      "Time: 8.80991\n",
      "Wall time: 2.89 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = rf.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 75.64%\n",
      "Precision: 44.795%\n",
      "Recall: 97.15599999999999%\n",
      "F1-score: 61.318%\n",
      "Time: 1.23198\n",
      "Wall time: 42.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nb = GaussianNB()\n",
    "nb.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = nb.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.604%\n",
      "Precision: 94.854%\n",
      "Recall: 98.311%\n",
      "F1-score: 96.552%\n",
      "Time: 779.11991\n",
      "Wall time: 4.42 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train,y_train)\n",
    "t1=time.time()\n",
    "predictions = knn.predict(X_test)\n",
    "t2=time.time()\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from keras.layers import Input,Dense,Dropout,BatchNormalization,Activation\n",
    "from keras import Model\n",
    "import keras.backend as K\n",
    "import keras.callbacks as kcallbacks\n",
    "from keras import optimizers\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'sgd',neurons=16,batch_size=1024,epochs=80,activation='relu',patience=8,loss='binary_crossentropy'):\n",
    "    K.clear_session()\n",
    "    inputs=Input(shape=(X.shape[1],))\n",
    "    x=Dense(1000)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.3)(x)\n",
    "    x=Dense(256)(inputs)\n",
    "    x=BatchNormalization()(x)\n",
    "    x=Activation('relu')(x)\n",
    "    x=Dropout(0.25)(x)\n",
    "    x=Dense(2,activation='softmax')(x)\n",
    "    model=Model(inputs=inputs,outputs=x,name='base_nlp')\n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
    "#     model.compile(optimizer=Adam(lr = 0.01),loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, pd.get_dummies(y).values,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 94.489%\n",
      "Precision: 80.223%\n",
      "Recall: 95.911%\n",
      "F1-score: 87.368%\n",
      "Time: 779.11991\n",
      "Wall time: 18.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ann = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "ann.fit(X_train,y_train)\n",
    "predictions = ann.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Time: \"+str(round((t2-t1)/len(y_test)*1000000,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Automated Feature Engineering\n",
    "Feature selection method 1: **Information Gain (IG)**, used to remove irrelevant features to improve model efficiency  \n",
    "Feature selection method 2: **Pearson Correlation**, used to remove redundant features to improve model efficiency and accuracy  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove irrelevant features and select important features\n",
    "def Feature_Importance_IG(data):\n",
    "    features = data.drop(['Label'],axis=1).values  # \"Label\" should be changed to the target class variable name if different\n",
    "    labels = data['Label'].values\n",
    "    \n",
    "    # Extract feature names\n",
    "    feature_names = list(data.drop(['Label'],axis=1).columns)\n",
    "\n",
    "    # Empty array for feature importances\n",
    "    feature_importance_values = np.zeros(len(feature_names))\n",
    "    model = lgb.LGBMRegressor(verbose = -1)\n",
    "    model.fit(features, labels)\n",
    "    feature_importances = pd.DataFrame({'feature': feature_names, 'importance': model.feature_importances_})\n",
    "\n",
    "    # Sort features according to importance\n",
    "    feature_importances = feature_importances.sort_values('importance', ascending = False).reset_index(drop = True)\n",
    "\n",
    "    # Normalize the feature importances to add up to one\n",
    "    feature_importances['normalized_importance'] = feature_importances['importance'] / feature_importances['importance'].sum()\n",
    "    feature_importances['cumulative_importance'] = np.cumsum(feature_importances['normalized_importance'])\n",
    "    \n",
    "    cumulative_importance=0.90 # Only keep the important features with cumulative importance scores>=90%. It can be changed.\n",
    "\n",
    "    # Make sure most important features are on top\n",
    "    feature_importances = feature_importances.sort_values('cumulative_importance')\n",
    "\n",
    "    # Identify the features not needed to reach the cumulative_importance\n",
    "    record_low_importance = feature_importances[feature_importances['cumulative_importance'] > cumulative_importance]\n",
    "\n",
    "    to_drop = list(record_low_importance['feature'])\n",
    "#     print(feature_importances.drop(['importance'],axis=1))\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove redundant features\n",
    "def Feature_Redundancy_Pearson(data):\n",
    "    correlation_threshold=0.90 # Only remove features with the redundancy>90%. It can be changed\n",
    "    features = data.drop(['Label'],axis=1)\n",
    "    corr_matrix = features.corr()\n",
    "\n",
    "    # Extract the upper triangle of the correlation matrix\n",
    "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k = 1).astype(np.bool))\n",
    "\n",
    "    # Select the features with correlations above the threshold\n",
    "    # Need to use the absolute value\n",
    "    to_drop = [column for column in upper.columns if any(upper[column].abs() > correlation_threshold)]\n",
    "\n",
    "    # Dataframe to hold correlated pairs\n",
    "    record_collinear = pd.DataFrame(columns = ['drop_feature', 'corr_feature', 'corr_value'])\n",
    "\n",
    "    # Iterate through the columns to drop\n",
    "    for column in to_drop:\n",
    "\n",
    "        # Find the correlated features\n",
    "        corr_features = list(upper.index[upper[column].abs() > correlation_threshold])\n",
    "\n",
    "        # Find the correlated values\n",
    "        corr_values = list(upper[column][upper[column].abs() > correlation_threshold])\n",
    "        drop_features = [column for _ in range(len(corr_features))]    \n",
    "\n",
    "        # Record the information (need a temp df for now)\n",
    "        temp_df = pd.DataFrame.from_dict({'drop_feature': drop_features,\n",
    "                                         'corr_feature': corr_features,\n",
    "                                         'corr_value': corr_values})\n",
    "        record_collinear = record_collinear.append(temp_df, ignore_index = True)\n",
    "#     print(record_collinear)\n",
    "    return to_drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Auto_Feature_Engineering(df):\n",
    "    drop1 = Feature_Importance_IG(df)\n",
    "    dfh1 = df.drop(columns = drop1)\n",
    "    \n",
    "    drop2 = Feature_Redundancy_Pearson(dfh1)\n",
    "    dfh2 = dfh1.drop(columns = drop2)\n",
    "    \n",
    "    return dfh2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow Duration</th>\n",
       "      <th>Total Length of Fwd Packets</th>\n",
       "      <th>Fwd Packet Length Max</th>\n",
       "      <th>Fwd Packet Length Mean</th>\n",
       "      <th>Bwd Packet Length Max</th>\n",
       "      <th>Flow IAT Mean</th>\n",
       "      <th>Flow IAT Min</th>\n",
       "      <th>Fwd IAT Min</th>\n",
       "      <th>Fwd Header Length</th>\n",
       "      <th>Bwd Packets/s</th>\n",
       "      <th>Init_Win_bytes_forward</th>\n",
       "      <th>Init_Win_bytes_backward</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.236419e-04</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>4.707129e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.311484e-05</td>\n",
       "      <td>0.004883</td>\n",
       "      <td>0.002350</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.416669e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.907407e-07</td>\n",
       "      <td>4.083333e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.004242</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.583334e-06</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>2.870370e-06</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>2.178649e-03</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.253752e-04</td>\n",
       "      <td>0.000090</td>\n",
       "      <td>0.002619</td>\n",
       "      <td>0.016856</td>\n",
       "      <td>0.010660</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>5.837500e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.057513e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.973835e-04</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.001732</td>\n",
       "      <td>0.011151</td>\n",
       "      <td>0.005072</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>4.415370e-04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.398152e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28298</th>\n",
       "      <td>4.083335e-07</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>4.537037e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000150</td>\n",
       "      <td>1.481481e-02</td>\n",
       "      <td>0.005341</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28299</th>\n",
       "      <td>9.525802e-01</td>\n",
       "      <td>0.000710</td>\n",
       "      <td>0.017204</td>\n",
       "      <td>0.008282</td>\n",
       "      <td>0.064133</td>\n",
       "      <td>3.649735e-02</td>\n",
       "      <td>9.074074e-07</td>\n",
       "      <td>1.375000e-06</td>\n",
       "      <td>0.001554</td>\n",
       "      <td>8.164962e-08</td>\n",
       "      <td>0.125015</td>\n",
       "      <td>0.005249</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28300</th>\n",
       "      <td>4.071168e-04</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.001612</td>\n",
       "      <td>0.010373</td>\n",
       "      <td>0.006190</td>\n",
       "      <td>1.508086e-04</td>\n",
       "      <td>4.629629e-08</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000299</td>\n",
       "      <td>2.729444e-05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28301</th>\n",
       "      <td>2.200001e-06</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>0.001330</td>\n",
       "      <td>0.008558</td>\n",
       "      <td>0.008339</td>\n",
       "      <td>8.395061e-07</td>\n",
       "      <td>4.814815e-07</td>\n",
       "      <td>4.000000e-07</td>\n",
       "      <td>0.000187</td>\n",
       "      <td>5.128205e-03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28302</th>\n",
       "      <td>2.416668e-07</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000242</td>\n",
       "      <td>0.001556</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>2.685185e-07</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>2.666667e-02</td>\n",
       "      <td>0.171768</td>\n",
       "      <td>0.534073</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28303 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flow Duration  Total Length of Fwd Packets  Fwd Packet Length Max  \\\n",
       "0       4.236419e-04                     0.000000               0.000000   \n",
       "1       4.416669e-07                     0.000000               0.000000   \n",
       "2       2.583334e-06                     0.000008               0.000242   \n",
       "3       5.253752e-04                     0.000090               0.002619   \n",
       "4       3.973835e-04                     0.000060               0.001732   \n",
       "...              ...                          ...                    ...   \n",
       "28298   4.083335e-07                     0.000000               0.000000   \n",
       "28299   9.525802e-01                     0.000710               0.017204   \n",
       "28300   4.071168e-04                     0.000111               0.001612   \n",
       "28301   2.200001e-06                     0.000092               0.001330   \n",
       "28302   2.416668e-07                     0.000008               0.000242   \n",
       "\n",
       "       Fwd Packet Length Mean  Bwd Packet Length Max  Flow IAT Mean  \\\n",
       "0                    0.000000               0.000000   4.707129e-04   \n",
       "1                    0.000000               0.000000   4.907407e-07   \n",
       "2                    0.001556               0.000516   2.870370e-06   \n",
       "3                    0.016856               0.010660   5.837500e-04   \n",
       "4                    0.011151               0.005072   4.415370e-04   \n",
       "...                       ...                    ...            ...   \n",
       "28298                0.000000               0.000000   4.537037e-07   \n",
       "28299                0.008282               0.064133   3.649735e-02   \n",
       "28300                0.010373               0.006190   1.508086e-04   \n",
       "28301                0.008558               0.008339   8.395061e-07   \n",
       "28302                0.001556               0.000516   2.685185e-07   \n",
       "\n",
       "       Flow IAT Min   Fwd IAT Min  Fwd Header Length  Bwd Packets/s  \\\n",
       "0      4.707129e-04  0.000000e+00           0.000150   1.311484e-05   \n",
       "1      4.907407e-07  4.083333e-07           0.000299   0.000000e+00   \n",
       "2      2.870370e-06  0.000000e+00           0.000094   2.178649e-03   \n",
       "3      5.837500e-04  0.000000e+00           0.000150   1.057513e-05   \n",
       "4      4.415370e-04  0.000000e+00           0.000150   1.398152e-05   \n",
       "...             ...           ...                ...            ...   \n",
       "28298  4.537037e-07  0.000000e+00           0.000150   1.481481e-02   \n",
       "28299  9.074074e-07  1.375000e-06           0.001554   8.164962e-08   \n",
       "28300  4.629629e-08  4.000000e-07           0.000299   2.729444e-05   \n",
       "28301  4.814815e-07  4.000000e-07           0.000187   5.128205e-03   \n",
       "28302  2.685185e-07  0.000000e+00           0.000094   2.666667e-02   \n",
       "\n",
       "       Init_Win_bytes_forward  Init_Win_bytes_backward  Label  \n",
       "0                    0.004883                 0.002350      0  \n",
       "1                    0.004242                 0.000000      0  \n",
       "2                    0.000015                 0.000015      0  \n",
       "3                    0.000000                 0.000000      0  \n",
       "4                    0.000000                 0.000000      0  \n",
       "...                       ...                      ...    ...  \n",
       "28298                0.005341                 0.004700      0  \n",
       "28299                0.125015                 0.005249      0  \n",
       "28300                0.000000                 0.000000      0  \n",
       "28301                0.000000                 0.000000      0  \n",
       "28302                0.171768                 0.534073      0  \n",
       "\n",
       "[28303 rows x 13 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfh2 = Auto_Feature_Engineering(df)\n",
    "dfh2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Split & Balancing (After Feature Engineering)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dfh2.drop(['Label'],axis=1)\n",
    "y = dfh2['Label']\n",
    "\n",
    "#X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2, shuffle=False,random_state = 0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y, train_size = 0.8, test_size = 0.2,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = Auto_Balancing(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Automated Model Selection\n",
    "Select the best-performing model among five common machine learning models (Naive Bayes, KNN, random forest, LightGBM, and ANN/MLP) by evaluating their learning performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 1: Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pipeline\n",
    "pipe = Pipeline([('classifier', GaussianNB())])\n",
    "\n",
    "# Create space of candidate learning algorithms and their hyperparameters\n",
    "search_space = [{'classifier': [GaussianNB()]},\n",
    "                {'classifier': [KNeighborsClassifier()]},\n",
    "                {'classifier': [RandomForestClassifier()]},\n",
    "                {'classifier': [lgb.LGBMClassifier(verbose = -1)]},\n",
    "                {'classifier': [KerasClassifier(build_fn=ANN, verbose=0)]},\n",
    "                 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = GridSearchCV(pipe, search_space, cv=5, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, estimator=Pipeline(steps=[('classifier', GaussianNB())]),\n",
       "             param_grid=[{'classifier': [GaussianNB()]},\n",
       "                         {'classifier': [KNeighborsClassifier()]},\n",
       "                         {'classifier': [RandomForestClassifier()]},\n",
       "                         {'classifier': [LGBMClassifier(verbose=-1)]},\n",
       "                         {'classifier': [<keras.wrappers.scikit_learn.KerasClassifier object at 0x0000012538B56358>]}])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model:{'classifier': LGBMClassifier(verbose=-1)}\n",
      "Accuracy:0.9843838600604344\n"
     ]
    }
   ],
   "source": [
    "print(\"Best Model:\"+ str(clf.best_params_))\n",
    "print(\"Accuracy:\"+ str(clf.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([9.57856178e-03, 8.62160683e-02, 1.30356531e+00, 1.47006226e-01,\n",
       "        1.11083066e+01]),\n",
       " 'std_fit_time': array([4.79699025e-04, 5.26681793e-03, 6.38863325e-02, 1.09814059e-02,\n",
       "        5.33423928e-01]),\n",
       " 'mean_score_time': array([0.00200119, 0.23560872, 0.04088306, 0.00937514, 0.17958202]),\n",
       " 'std_score_time': array([1.18880773e-05, 2.28870300e-02, 1.02982763e-03, 1.19632886e-03,\n",
       "        4.75653887e-03]),\n",
       " 'param_classifier': masked_array(data=[GaussianNB(), KNeighborsClassifier(),\n",
       "                    RandomForestClassifier(), LGBMClassifier(verbose=-1),\n",
       "                    <keras.wrappers.scikit_learn.KerasClassifier object at 0x0000012538B56358>],\n",
       "              mask=[False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'classifier': GaussianNB()},\n",
       "  {'classifier': KNeighborsClassifier()},\n",
       "  {'classifier': RandomForestClassifier()},\n",
       "  {'classifier': LGBMClassifier(verbose=-1)},\n",
       "  {'classifier': <keras.wrappers.scikit_learn.KerasClassifier at 0x12538b56358>}],\n",
       " 'split0_test_score': array([0.34764176, 0.96343402, 0.99752694, 0.996997  ,        nan]),\n",
       " 'split1_test_score': array([0.39727963, 0.96679032, 0.99752694, 0.99717364,        nan]),\n",
       " 'split2_test_score': array([0.31867161, 0.88235294, 0.88782901, 0.94258965,        nan]),\n",
       " 'split3_test_score': array([0.36484099, 0.98975265, 0.99805654, 0.99787986,        nan]),\n",
       " 'split4_test_score': array([0.33533569, 0.9770318 , 0.98339223, 0.98727915,        nan]),\n",
       " 'mean_test_score': array([0.35275394, 0.95587235, 0.97286633, 0.98438386,        nan]),\n",
       " 'std_test_score': array([0.02690639, 0.03788689, 0.04287885, 0.02126006,        nan]),\n",
       " 'rank_test_score': array([4, 3, 2, 1, 5])}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.cv_results_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM model is the best performing machine learning model, and the best cross-validation accuracy is 98.438%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 2: Bayesian Optimization with Tree Parzen Estimator (BO-TPE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 10/10 [00:19<00:00,  1.94s/trial, best loss: -0.9980568804098215]\n",
      "Hyperopt estimated optimum {'classifier_type': 3}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    \n",
    "    classifier_type = params['type']\n",
    "    del params['type']\n",
    "    if classifier_type == 'nb':\n",
    "        clf = GaussianNB()\n",
    "    elif classifier_type == 'knn':\n",
    "        clf = KNeighborsClassifier()\n",
    "    elif classifier_type == 'rf':\n",
    "        clf = RandomForestClassifier()\n",
    "    elif classifier_type == 'lgb':\n",
    "        clf = lgb.LGBMClassifier(verbose = -1)\n",
    "    elif classifier_type == 'ann':\n",
    "        clf = KerasClassifier(build_fn=ANN, verbose=0)\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = hp.choice('classifier_type', [{'type': 'nb'},{'type': 'knn'},{'type': 'rf'},{'type': 'lgb'},{'type': 'ann'},])\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=10)\n",
    "print(\"Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classifier type 3 is the LightGBM model, and the best hold-out accuracy is 99.806%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 4. Hyperparameter Optimization\n",
    "Optimize the best performing machine learning model (lightGBM) by tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 20/20 [01:08<00:00,  3.44s/trial, best loss: -0.9849138641863646]\n",
      "LightGBM: Hyperopt estimated optimum {'learning_rate': 0.4765834961973211, 'max_depth': 14.0, 'min_child_samples': 25.0, 'n_estimators': 480.0, 'num_leaves': 600.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate': abs(float(params['learning_rate'])),\n",
    "        \"num_leaves\": int(params['num_leaves']),\n",
    "        \"min_child_samples\": int(params['min_child_samples']),\n",
    "    }\n",
    "    clf = lgb.LGBMClassifier( **params)\n",
    "    score = cross_val_score(clf, X, y, scoring='accuracy', cv=StratifiedKFold(n_splits=5)).mean()\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 50, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"learning_rate\":hp.uniform('learning_rate', 0, 1),\n",
    "    \"num_leaves\":hp.quniform('num_leaves',100,2000,100),\n",
    "    \"min_child_samples\":hp.quniform('min_child_samples',10,50,5),\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"LightGBM: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.477%\n",
      "Precision: 98.985%\n",
      "Recall: 93.24900000000001%\n",
      "F1-score: 95.758%\n",
      "Wall time: 17.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=14, learning_rate=  0.4765834961973211, n_estimators = 480, \n",
    "                         num_leaves = 600, min_child_samples = 25)\n",
    "clf.fit(X,y)\n",
    "scores = cross_val_score(clf, X, y, cv=5,scoring='accuracy')\n",
    "print(\"Accuracy: \"+ str(round(scores.mean(),5)*100)+\"%\")\n",
    "scores = cross_val_score(clf, X, y, cv=5,scoring='precision')\n",
    "print(\"Precision: \"+ str(round(scores.mean(),5)*100)+\"%\")\n",
    "scores = cross_val_score(clf, X, y, cv=5,scoring='recall')\n",
    "print(\"Recall: \"+ str(round(scores.mean(),5)*100)+\"%\")\n",
    "scores = cross_val_score(clf, X, y, cv=5,scoring='f1')\n",
    "print(\"F1-score: \"+ str(round(scores.mean(),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After hyperparameter optimization, the cross-validation accuracy has been improved from 98.438% to 98.477%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold-out validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████| 50/50 [00:56<00:00,  1.14s/trial, best loss: -0.9984101748807631]\n",
      "LightGBM: Hyperopt estimated optimum {'learning_rate': 0.7925617918030913, 'max_depth': 35.0, 'min_child_samples': 25.0, 'n_estimators': 200.0, 'num_leaves': 200.0}\n"
     ]
    }
   ],
   "source": [
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'learning_rate': abs(float(params['learning_rate'])),\n",
    "        \"num_leaves\": int(params['num_leaves']),\n",
    "        \"min_child_samples\": int(params['min_child_samples']),\n",
    "    }\n",
    "    clf = lgb.LGBMClassifier( **params)\n",
    "    clf.fit(X_train,y_train)\n",
    "    predictions = clf.predict(X_test)\n",
    "    score = accuracy_score(y_test,predictions)\n",
    "    return {'loss':-score, 'status': STATUS_OK }\n",
    "\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 50, 500, 20),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"learning_rate\":hp.uniform('learning_rate', 0, 1),\n",
    "    \"num_leaves\":hp.quniform('num_leaves',100,2000,100),\n",
    "    \"min_child_samples\":hp.quniform('min_child_samples',10,50,5),\n",
    "}\n",
    "\n",
    "# Detect the optimal hyperparameter values\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=50)\n",
    "print(\"LightGBM: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.84100000000001%\n",
      "Precision: 99.381%\n",
      "Recall: 99.822%\n",
      "F1-score: 99.601%\n",
      "Wall time: 360 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=35, learning_rate= 0.7925617918030913, n_estimators = 200, \n",
    "                         num_leaves = 200, min_child_samples = 25)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "After hyperparameter optimization, the hold-out accuracy has been improved from 99.806% to 99.841%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Combined Algorithm Selection and Hyperparameter tuning (CASH)\n",
    "CASH is the process of combining the two AutoML procedures: model selection and hyperparameter optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Method: Particle Swarm Optimization (PSO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'algorithm': 'lightgbm', 'n_neighbors': None, 'learning_rate': 0.25474609375, 'max_depth': 23.670605468749997, 'min_child_samples': 15.73671875, 'n_estimators': 418.7626953125, 'num_leaves': 1462.550366596432, 'max_features': None, 'min_samples_leaf': None, 'min_samples_split': None}\n",
      "0.9980568804098215\n"
     ]
    }
   ],
   "source": [
    "import optunity\n",
    "import optunity.metrics\n",
    "\n",
    "search = {'algorithm': {'k-nn': {'n_neighbors': [3, 10]},\n",
    "                        'naive-bayes': None,\n",
    "                        'random-forest': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_features': [5, 12],\n",
    "                                'max_depth': [5,50],\n",
    "                                \"min_samples_split\":[2,11],\n",
    "                                \"min_samples_leaf\":[1,11]},\n",
    "                        'lightgbm': {\n",
    "                                'n_estimators': [50, 500],\n",
    "                                'max_depth': [5, 50],\n",
    "                                'learning_rate': (0, 1),\n",
    "                                \"num_leaves\":[100, 2000],\n",
    "                                \"min_child_samples\":[10, 50],\n",
    "                                    },\n",
    "                        'ann': {\n",
    "                                'neurons': [10, 100],\n",
    "                                'epochs': [20, 50],\n",
    "                                'patience': [3, 20],\n",
    "                                }\n",
    "                        }\n",
    "          \n",
    "         }\n",
    "def performance(\n",
    "                algorithm, n_neighbors=None, \n",
    "    n_estimators=None, max_features=None,max_depth=None,min_samples_split=None,min_samples_leaf=None,\n",
    "    learning_rate=None,num_leaves=None,min_child_samples=None,\n",
    "    neurons=None,epochs=None,patience=None\n",
    "):\n",
    "    # fit the model\n",
    "    if algorithm == 'k-nn':\n",
    "        model = KNeighborsClassifier(n_neighbors=int(n_neighbors))\n",
    "    elif algorithm == 'naive-bayes':\n",
    "        model = GaussianNB()\n",
    "    elif algorithm == 'random-forest':\n",
    "        model = RandomForestClassifier(n_estimators=int(n_estimators),\n",
    "                                       max_features=int(max_features),\n",
    "                                       max_depth=int(max_depth),\n",
    "                                       min_samples_split=int(min_samples_split),\n",
    "                                       min_samples_leaf=int(min_samples_leaf))\n",
    "    elif algorithm == 'lightgbm':\n",
    "        model = lgb.LGBMClassifier(n_estimators=int(n_estimators),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   learning_rate=float(learning_rate),\n",
    "                                   num_leaves=int(num_leaves),\n",
    "                                   min_child_samples=int(min_child_samples),\n",
    "                                  )\n",
    "    elif algorithm == 'ann':\n",
    "        model = KerasClassifier(build_fn=ANN, verbose=0,\n",
    "                               neurons=int(neurons),\n",
    "                                epochs=int(epochs),\n",
    "                                patience=int(patience)\n",
    "                               )\n",
    "    else:\n",
    "        raise ArgumentError('Unknown algorithm: %s' % algorithm)\n",
    "# predict the test set\n",
    "    model.fit(X_train,y_train)\n",
    "    prediction = model.predict(X_test)\n",
    "    score = accuracy_score(y_test,prediction)\n",
    "    return score\n",
    "\n",
    "# Run the CASH process\n",
    "optimal_configuration, info, _ = optunity.maximize_structured(performance, \n",
    "                                                              search_space=search, \n",
    "                                                              num_evals=50)\n",
    "print(optimal_configuration)\n",
    "print(info.optimum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 99.84100000000001%\n",
      "Precision: 99.381%\n",
      "Recall: 99.822%\n",
      "F1-score: 99.601%\n",
      "Wall time: 2.28 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lgb.LGBMClassifier(max_depth=24, learning_rate= 0.25474609375, n_estimators = 419, \n",
    "                         num_leaves = 1463, min_child_samples = 16)\n",
    "clf.fit(X_train,y_train)\n",
    "predictions = clf.predict(X_test)\n",
    "print(\"Accuracy: \"+str(round(accuracy_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Precision: \"+str(round(precision_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"Recall: \"+str(round(recall_score(y_test,predictions),5)*100)+\"%\")\n",
    "print(\"F1-score: \"+str(round(f1_score(y_test,predictions),5)*100)+\"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM with the above hyperparameter values is identified as the optimal model"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}