Neural Network.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SSXrNBp0M5qk"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib as plt\n",
        "import sklearn\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## QUESTION 1"
      ],
      "metadata": {
        "id": "E_cgzxmt6OBQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sigmoid(x):\n",
        "    return 1 / (1 + pow(2.71828182846, -x))\n",
        "\n",
        "def and_gate(input1, input2):\n",
        "    # Define the weights and biases for the hidden layer\n",
        "    hidden_weights = [20, 20]\n",
        "    hidden_bias = -30\n",
        "\n",
        "    # Define the weights and bias for the output layer\n",
        "    output_weights = [20, 20]\n",
        "    output_bias = -10\n",
        "\n",
        "    # Feedforward propagation for the hidden layer\n",
        "    hidden_layer_output = sigmoid(hidden_weights[0] * input1 + hidden_weights[1] * input2 + hidden_bias)\n",
        "\n",
        "    # Feedforward propagation for the output layer\n",
        "    output_layer_output = sigmoid(output_weights[0] * hidden_layer_output + output_weights[1] * hidden_layer_output + output_bias)\n",
        "\n",
        "    # Return the output\n",
        "    return int(output_layer_output >= 0.5)\n",
        "\n",
        "# Default input cases\n",
        "default_inputs = [(0, 0), (0, 1), (1, 0), (1,1)]\n",
        "\n",
        "# Driver test with default inputs\n",
        "for input_pair in default_inputs:\n",
        "    input1, input2 = input_pair\n",
        "    output = and_gate(input1, input2)\n",
        "    print(\"Input:\", input_pair, \"Output:\", output)\n",
        "\n",
        "# User input\n",
        "input1 = int(input(\"Enter the first input (0 or 1): \"))\n",
        "input2 = int(input(\"Enter the second input (0 or 1): \"))\n",
        "\n",
        "output = and_gate(input1, input2)\n",
        "print(\"Output:\", output)\n",
        "\n",
        "###############################################################################\n",
        "###############################################################################\n",
        "####################  ANSWERS OF QUESTIONS ####################################\n",
        "### 1 ####\n",
        "# I used one hidden layer\n",
        "\n",
        "### 2 ####\n",
        "## For the hidden layer:\n",
        "# The Node:\n",
        "#Weight 1: 20\n",
        "#Weight 2: 20\n",
        "#Bias: -30\n",
        "\n",
        "#For the output layer:\n",
        "# The node\n",
        "#Weight 1: 20\n",
        "#Weight 2: 20\n",
        "# Bias: -10\n"
      ],
      "metadata": {
        "id": "GeEsI0xJ6SP_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8e526ca5-fd90-4b07-b2cc-187a3867d9b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input: (0, 0) Output: 0\n",
            "Input: (0, 1) Output: 0\n",
            "Input: (1, 0) Output: 0\n",
            "Input: (1, 1) Output: 1\n",
            "Enter the first input (0 or 1): 1\n",
            "Enter the second input (0 or 1): 1\n",
            "Output: 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## QUESTION 2"
      ],
      "metadata": {
        "id": "sfsJVf3U6ThC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "url = 'https://raw.githubusercontent.com/CombatLawyer/COMP472/main/NNData.csv'\n",
        "df = pd.read_csv(url)"
      ],
      "metadata": {
        "id": "tZZ3YyYM6Vbv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3SWC5zWX9G6R",
        "outputId": "ad0d92b0-87b2-4434-9d6d-285c88057ace"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "    A  B  C  D  E  y\n",
            "0   1  1  1  0  0  1\n",
            "1   1  1  1  0  1  1\n",
            "2   0  0  0  1  0  0\n",
            "3   0  0  0  1  1  0\n",
            "4   0  0  1  0  0  1\n",
            "5   0  0  1  0  1  1\n",
            "6   0  0  1  1  0  1\n",
            "7   0  0  1  1  1  1\n",
            "8   0  1  0  0  0  0\n",
            "9   0  1  0  0  1  0\n",
            "10  0  1  0  1  0  0\n",
            "11  0  1  0  1  1  0\n",
            "12  0  1  1  0  0  1\n",
            "13  0  1  1  0  1  1\n",
            "14  0  1  1  1  0  1\n",
            "15  0  1  1  1  1  1\n",
            "16  1  0  0  0  0  1\n",
            "17  1  0  0  0  1  1\n",
            "18  1  0  0  1  0  1\n",
            "19  1  0  0  1  1  1\n",
            "20  1  0  1  0  0  1\n",
            "21  1  0  1  0  1  1\n",
            "22  1  0  1  1  0  1\n",
            "23  1  0  1  1  1  1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "url = 'https://raw.githubusercontent.com/CombatLawyer/COMP472/main/NNTestSet.csv'\n",
        "test = pd.read_csv(url)"
      ],
      "metadata": {
        "id": "EeymX5t7f2UC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WvnkWDDegorN",
        "outputId": "c4d08a96-a06c-447e-87d9-f857c145cd05"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   A  B  C  D  E  y\n",
            "0  1  1  0  0  0  1\n",
            "1  1  1  0  0  1  1\n",
            "2  0  0  0  0  0  0\n",
            "3  0  0  0  0  1  0\n",
            "4  1  1  0  1  0  1\n",
            "5  1  1  0  1  1  1\n",
            "6  1  1  1  1  0  1\n",
            "7  1  1  1  1  1  1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# MLP classifier initialization, solver lbfgs and relu was used because it converges faster.\n",
        "mlp = MLPClassifier(solver='lbfgs', activation='relu', random_state=1, max_iter=100)\n",
        "\n",
        "# Grid of paramaters to test which amount of layers/amount of nodes is optimal to use. Will search only these configurations.\n",
        "parameter_space = {'hidden_layer_sizes': [(1), (1, 1), (1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1, 1), (2, 2), (2, 2, 2), (2, 2, 2, 2), (2, 2, 2, 2, 2), (3), (3, 3), (3, 3, 3), (3, 3, 3, 3, 3), (4), (4, 4), (4, 4, 4), (4, 4, 4, 4), (4, 4, 4, 4, 4), (5), (5, 5), (5, 5, 5), (5, 5, 5, 5), (5, 5, 5, 5, 5)]}\n",
        "\n",
        "# Provide grid of parameters to the MLP classifier.\n",
        "clf = GridSearchCV(mlp, parameter_space)\n",
        "\n",
        "# Fit data to grid of MLPs to determine the optimal parameters.\n",
        "clf.fit(df.iloc[:,:-1].values, df.iloc[:,-1:].values.reshape(len(df),))"
      ],
      "metadata": {
        "id": "KanYB_EG9Nbv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 118
        },
        "outputId": "d9ab829d-a655-4bec-d527-7dc99f6e9ea7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(estimator=MLPClassifier(max_iter=100, random_state=1,\n",
              "                                     solver='lbfgs'),\n",
              "             param_grid={'hidden_layer_sizes': [1, (1, 1), (1, 1, 1),\n",
              "                                                (1, 1, 1, 1), (1, 1, 1, 1, 1),\n",
              "                                                (2, 2), (2, 2, 2), (2, 2, 2, 2),\n",
              "                                                (2, 2, 2, 2, 2), 3, (3, 3),\n",
              "                                                (3, 3, 3), (3, 3, 3, 3, 3), 4,\n",
              "                                                (4, 4), (4, 4, 4), (4, 4, 4, 4),\n",
              "                                                (4, 4, 4, 4, 4), 5, (5, 5),\n",
              "                                                (5, 5, 5), (5, 5, 5, 5),\n",
              "                                                (5, 5, 5, 5, 5)]})"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(estimator=MLPClassifier(max_iter=100, random_state=1,\n",
              "                                     solver=&#x27;lbfgs&#x27;),\n",
              "             param_grid={&#x27;hidden_layer_sizes&#x27;: [1, (1, 1), (1, 1, 1),\n",
              "                                                (1, 1, 1, 1), (1, 1, 1, 1, 1),\n",
              "                                                (2, 2), (2, 2, 2), (2, 2, 2, 2),\n",
              "                                                (2, 2, 2, 2, 2), 3, (3, 3),\n",
              "                                                (3, 3, 3), (3, 3, 3, 3, 3), 4,\n",
              "                                                (4, 4), (4, 4, 4), (4, 4, 4, 4),\n",
              "                                                (4, 4, 4, 4, 4), 5, (5, 5),\n",
              "                                                (5, 5, 5), (5, 5, 5, 5),\n",
              "                                                (5, 5, 5, 5, 5)]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(estimator=MLPClassifier(max_iter=100, random_state=1,\n",
              "                                     solver=&#x27;lbfgs&#x27;),\n",
              "             param_grid={&#x27;hidden_layer_sizes&#x27;: [1, (1, 1), (1, 1, 1),\n",
              "                                                (1, 1, 1, 1), (1, 1, 1, 1, 1),\n",
              "                                                (2, 2), (2, 2, 2), (2, 2, 2, 2),\n",
              "                                                (2, 2, 2, 2, 2), 3, (3, 3),\n",
              "                                                (3, 3, 3), (3, 3, 3, 3, 3), 4,\n",
              "                                                (4, 4), (4, 4, 4), (4, 4, 4, 4),\n",
              "                                                (4, 4, 4, 4, 4), 5, (5, 5),\n",
              "                                                (5, 5, 5), (5, 5, 5, 5),\n",
              "                                                (5, 5, 5, 5, 5)]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: MLPClassifier</label><div class=\"sk-toggleable__content\"><pre>MLPClassifier(max_iter=100, random_state=1, solver=&#x27;lbfgs&#x27;)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MLPClassifier</label><div class=\"sk-toggleable__content\"><pre>MLPClassifier(max_iter=100, random_state=1, solver=&#x27;lbfgs&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Optimal MLP predictions on the test data.\n",
        "clf.predict(test.iloc[:,:-1].values)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ewJ_JE-9gqak",
        "outputId": "42ae63c0-081e-43af-f1ab-c273a2f74c29"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 0, 1, 1, 1, 1, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Test accuracy of the optimal layers MLP.\n",
        "print(f\"Accuracy: {clf.score(test.iloc[:,:-1].values, test.iloc[:,-1:].values.reshape(len(test),))*100}\")\n",
        "print(f\"Precision: {sklearn.metrics.precision_score(clf.predict(test.iloc[:,:-1].values), test.iloc[:,-1:].values.reshape(len(test),))*100}\")\n",
        "print(f\"Recall: {sklearn.metrics.recall_score(clf.predict(test.iloc[:,:-1].values), test.iloc[:,-1:].values.reshape(len(test),))*100}\")\n",
        "print(f\"F1 score: {sklearn.metrics.f1_score(clf.predict(test.iloc[:,:-1].values), test.iloc[:,-1:].values.reshape(len(test),))*100}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kgUFu5FuhCGw",
        "outputId": "a7c26f32-07a4-40d6-c755-7b51f716b020"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 87.5\n",
            "Precision: 100.0\n",
            "Recall: 85.71428571428571\n",
            "F1 score: 92.3076923076923\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# The best parameters chosen through the grid search.\n",
        "print(clf.best_params_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RzZ094dmrsOV",
        "outputId": "b2e88bde-bfd1-4a56-9acc-d27392610bb0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'hidden_layer_sizes': (5, 5)}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "clf.cv_results_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Cn1N17BOxmdd",
        "outputId": "39fcff98-37f2-40b7-9e47-f7643f64d831"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'mean_fit_time': array([0.01827874, 0.01269579, 0.00960073, 0.00814424, 0.00999093,\n",
              "        0.02089849, 0.03577347, 0.13674664, 0.00863662, 0.020438  ,\n",
              "        0.05799198, 0.04631777, 0.08062429, 0.00547872, 0.00671301,\n",
              "        0.02241335, 0.01796045, 0.02588477, 0.01299734, 0.01507058,\n",
              "        0.02482014, 0.03995261, 0.04178357]),\n",
              " 'std_fit_time': array([0.00369592, 0.00430782, 0.00305906, 0.00304659, 0.00622273,\n",
              "        0.02110079, 0.03066988, 0.053497  , 0.00147912, 0.01331901,\n",
              "        0.02427242, 0.02710214, 0.0260538 , 0.00057423, 0.00033533,\n",
              "        0.0122159 , 0.01342445, 0.00715577, 0.00108592, 0.00259595,\n",
              "        0.01303395, 0.01400135, 0.01257731]),\n",
              " 'mean_score_time': array([0.00244131, 0.00219526, 0.00161633, 0.00103827, 0.00294333,\n",
              "        0.00121727, 0.00665331, 0.00257735, 0.00103364, 0.00291567,\n",
              "        0.00141234, 0.00108972, 0.00128617, 0.00060897, 0.00067811,\n",
              "        0.00076089, 0.00092359, 0.0012413 , 0.00109634, 0.00117564,\n",
              "        0.00110602, 0.00114613, 0.00260344]),\n",
              " 'std_score_time': array([1.69854753e-03, 2.06791273e-03, 1.07483438e-03, 6.41267800e-05,\n",
              "        3.89679411e-03, 4.39867601e-04, 4.89747937e-03, 2.96490202e-03,\n",
              "        6.11814965e-05, 3.06563577e-03, 6.84990567e-04, 2.05436589e-05,\n",
              "        3.74906089e-04, 3.53822811e-05, 3.09762168e-05, 2.61325035e-05,\n",
              "        2.36193305e-04, 4.79420815e-05, 5.94093174e-05, 2.24313241e-04,\n",
              "        4.37639367e-05, 5.07273614e-05, 2.91261427e-03]),\n",
              " 'param_hidden_layer_sizes': masked_array(data=[1, (1, 1), (1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1, 1),\n",
              "                    (2, 2), (2, 2, 2), (2, 2, 2, 2), (2, 2, 2, 2, 2), 3,\n",
              "                    (3, 3), (3, 3, 3), (3, 3, 3, 3, 3), 4, (4, 4),\n",
              "                    (4, 4, 4), (4, 4, 4, 4), (4, 4, 4, 4, 4), 5, (5, 5),\n",
              "                    (5, 5, 5), (5, 5, 5, 5), (5, 5, 5, 5, 5)],\n",
              "              mask=[False, False, False, False, False, False, False, False,\n",
              "                    False, False, False, False, False, False, False, False,\n",
              "                    False, False, False, False, False, False, False],\n",
              "        fill_value='?',\n",
              "             dtype=object),\n",
              " 'params': [{'hidden_layer_sizes': 1},\n",
              "  {'hidden_layer_sizes': (1, 1)},\n",
              "  {'hidden_layer_sizes': (1, 1, 1)},\n",
              "  {'hidden_layer_sizes': (1, 1, 1, 1)},\n",
              "  {'hidden_layer_sizes': (1, 1, 1, 1, 1)},\n",
              "  {'hidden_layer_sizes': (2, 2)},\n",
              "  {'hidden_layer_sizes': (2, 2, 2)},\n",
              "  {'hidden_layer_sizes': (2, 2, 2, 2)},\n",
              "  {'hidden_layer_sizes': (2, 2, 2, 2, 2)},\n",
              "  {'hidden_layer_sizes': 3},\n",
              "  {'hidden_layer_sizes': (3, 3)},\n",
              "  {'hidden_layer_sizes': (3, 3, 3)},\n",
              "  {'hidden_layer_sizes': (3, 3, 3, 3, 3)},\n",
              "  {'hidden_layer_sizes': 4},\n",
              "  {'hidden_layer_sizes': (4, 4)},\n",
              "  {'hidden_layer_sizes': (4, 4, 4)},\n",
              "  {'hidden_layer_sizes': (4, 4, 4, 4)},\n",
              "  {'hidden_layer_sizes': (4, 4, 4, 4, 4)},\n",
              "  {'hidden_layer_sizes': 5},\n",
              "  {'hidden_layer_sizes': (5, 5)},\n",
              "  {'hidden_layer_sizes': (5, 5, 5)},\n",
              "  {'hidden_layer_sizes': (5, 5, 5, 5)},\n",
              "  {'hidden_layer_sizes': (5, 5, 5, 5, 5)}],\n",
              " 'split0_test_score': array([0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 1. , 0.8, 1. , 1. , 1. , 1. ,\n",
              "        1. , 1. , 0.8, 0.8, 1. , 1. , 1. , 1. , 1. , 1. ]),\n",
              " 'split1_test_score': array([0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.6, 0.6, 0.8, 1. , 1. , 1. , 1. ,\n",
              "        1. , 1. , 1. , 0.8, 1. , 0.8, 0.8, 1. , 0.8, 0.8]),\n",
              " 'split2_test_score': array([0.8, 0.8, 0.8, 0.8, 0.8, 1. , 0.6, 0.8, 0.8, 0.6, 1. , 1. , 1. ,\n",
              "        1. , 1. , 0.6, 1. , 1. , 1. , 1. , 1. , 1. , 1. ]),\n",
              " 'split3_test_score': array([0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.8, 0.6, 0.6, 1. , 0.6, 0.6, 0.6,\n",
              "        0.6, 0.6, 1. , 0.8, 0.6, 0.6, 1. , 0.8, 1. , 0.8]),\n",
              " 'split4_test_score': array([0.75, 0.75, 0.75, 0.75, 0.75, 0.75, 1.  , 0.75, 0.75, 1.  , 1.  ,\n",
              "        1.  , 1.  , 1.  , 1.  , 1.  , 1.  , 1.  , 1.  , 1.  , 1.  , 1.  ,\n",
              "        1.  ]),\n",
              " 'mean_test_score': array([0.75, 0.75, 0.75, 0.75, 0.75, 0.79, 0.76, 0.75, 0.75, 0.92, 0.92,\n",
              "        0.92, 0.92, 0.92, 0.92, 0.88, 0.88, 0.92, 0.88, 0.96, 0.96, 0.96,\n",
              "        0.92]),\n",
              " 'std_test_score': array([0.07745967, 0.07745967, 0.07745967, 0.07745967, 0.07745967,\n",
              "        0.12806248, 0.1496663 , 0.14832397, 0.07745967, 0.16      ,\n",
              "        0.16      , 0.16      , 0.16      , 0.16      , 0.16      ,\n",
              "        0.16      , 0.09797959, 0.16      , 0.16      , 0.08      ,\n",
              "        0.08      , 0.08      , 0.09797959]),\n",
              " 'rank_test_score': array([17, 17, 17, 17, 17, 15, 16, 17, 17,  4,  4,  4,  4,  4,  4, 12, 12,\n",
              "         4, 12,  1,  1,  1,  4], dtype=int32)}"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "a) How many hidden layers have you used? And why?"
      ],
      "metadata": {
        "id": "AIMj_qIqpckR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the grid search the best amount of layers determined was 2 hidden layers. The hyperparameter search was done using a grid search using options from 1-5 layers with uniform layer counts to determine which neural network was the most optimal. This was done by doing cross-validation (default is 5-fold) which holds out a portion of the data, trains the model using the rest and determines the accuracy. This is repeated for each fold and then averaged. The best result was (5, 5) which had the highest mean test score. The network of (3, 3, 3, 3) and (2) was removed because it failed to converge during the grid search using the lbfgs solver."
      ],
      "metadata": {
        "id": "PXybYMqxxUwk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "b) How many nodes in each hidden layer and why that number of nodes in particular?"
      ],
      "metadata": {
        "id": "Yq5PxDdLsgmh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the grid search as a hyperparameter search the best amount of layers for that 2 hidden layer was determined as 5 nodes. This was done using the options provided (1-5) nodes in each layer with the same nodes per layer. In the same fashion the grid search was used to determine the optimal amount of layers from the options provided."
      ],
      "metadata": {
        "id": "7ZZ2FFnkxe6m"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "c) What is the activation function that you used and why? Did you use the same activation function in all layers? Why?"
      ],
      "metadata": {
        "id": "xuqVRpM9sjeX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The activation function used was the ReLU (Rectified Linear Unit) function, which was specifically selected because it converges faster than the older options (six times faster than logistical and tanh). The same activation function is used in every layer for simplicity, but does not need to be the case. For specific neural networks like CNNs (Convolutional Neural Network) can use a ReLU layers for activation but then a softmax at the end to get probabilities."
      ],
      "metadata": {
        "id": "QnVfwCuX0SPu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "d) What learning algorithm did you use to train the neural net and why?"
      ],
      "metadata": {
        "id": "WJ9xZpS2slU0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A MLP (Multilayer Perceptrons) was used for simplicity sake. The MLP learns a function, which can then make predictions based on a suitable input. The problem was not very difficult, nor did not have a more specialized scenario that would require a more specialized neural network like CNNs (Convolutional Neural Networks) and images, or LSTMs (Long Short Term Memory Networks) which retain information over time, or GANs (Generative Adversarial Networks) which can generate data, etc."
      ],
      "metadata": {
        "id": "7FAjaQjT2581"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "e) Can you use one hidden layer only to solve this problem? If yes, how many nodes are you going to have in it? And why?"
      ],
      "metadata": {
        "id": "BgzdaALisnHa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Linear test of the impact of the number of nodes on the test accuracy.\n",
        "for i in range(1, 31):\n",
        "\n",
        "  # Create the MLP using the specified number of nodes.\n",
        "  clf = MLPClassifier(activation='relu', solver='lbfgs', random_state=1, hidden_layer_sizes=(i))\n",
        "\n",
        "  # Fit data to the MLP.\n",
        "  clf.fit(df.iloc[:,:-1].values, df.iloc[:,-1:].values.reshape(len(df),))\n",
        "\n",
        "  # Determine accuracy using the test set.\n",
        "  testAcc = clf.score(test.iloc[:,:-1].values, test.iloc[:,-1:].values.reshape(len(test),))\n",
        "  print(f\"A neural network with 1 layer and {i} node(s) has a test accuracy of {testAcc}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-CEAHi8_svVE",
        "outputId": "13942e2b-3cde-4345-99a2-827f3912cce6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "A neural network with 1 layer and 1 node(s) has a test accuracy of 0.75\n",
            "A neural network with 1 layer and 2 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 3 node(s) has a test accuracy of 0.75\n",
            "A neural network with 1 layer and 4 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 5 node(s) has a test accuracy of 0.875\n",
            "A neural network with 1 layer and 6 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 7 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 8 node(s) has a test accuracy of 0.75\n",
            "A neural network with 1 layer and 9 node(s) has a test accuracy of 0.75\n",
            "A neural network with 1 layer and 10 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 11 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 12 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 13 node(s) has a test accuracy of 0.75\n",
            "A neural network with 1 layer and 14 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 15 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 16 node(s) has a test accuracy of 0.875\n",
            "A neural network with 1 layer and 17 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 18 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 19 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 20 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 21 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 22 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 23 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 24 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 25 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 26 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 27 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 28 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 29 node(s) has a test accuracy of 1.0\n",
            "A neural network with 1 layer and 30 node(s) has a test accuracy of 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Yes a single hidden layer can be used but the number of nodes changes the test accuracy of the neural network. Some might be over fitted giving a perfect test accuracy. Something like the 0.875 scores might be best and then the lowest number of layers would be ideal (5 nodes). However this might also be due to the small amount of data in the train/test set in general, more data would be ideal to determine what hyperparameters might be best."
      ],
      "metadata": {
        "id": "HioLm7cvwOg6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "f) Can we use 5 hidden layers? Is that a good idea? Justify your answer."
      ],
      "metadata": {
        "id": "JHxW4pE5sog-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "5 hidden layers can be used, its just likely to overfit the relationship. In general we could use hundred of layers if we wanted to, it would just be more computaitonally heavy and large for no real reason unless the relationship we are computing is difficult/has many inputs and requires the additional layers to properly classify inputs."
      ],
      "metadata": {
        "id": "bTeX4DGszkip"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "g) How did the neural net do in classifying the testing set? Comment on how good or bad it learned the function from the training set."
      ],
      "metadata": {
        "id": "RNWVcMcjsquD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The neural network does extremely well in some cases (when the test accuracy is 100%) and in other it does less (when it dips to 75% test accuracy in the 1 node layer tests and some of the grid options have a 50% accuracy). If assuming only this is the data we have access to then we can simply find a set of hyperparameters that yeilds 100% accuracy, but this is mostly due to the small size of data we have. In the aforementioned cases, we might sometime prefer a say 87.5% accuracy nueral network as it performs consistently well, but also allows some differences that it can potentially allow it to be more flexible in unseen data over potentially overfitting the relationship. Other accuracy measures such as precision, recall and F-score can be used in more complex cases to evaluate model accuracy. Ideally we should have more data, which would change the ideal hyperparameters, and would allow us to determine if the model is truly accurate or is overfitting to the training data. In general, a neural network can always provide a reasonable learning function from the training set, the size of the nerwork might be larger/computationally expensive as result, which might not always be ideal."
      ],
      "metadata": {
        "id": "yo_lL4z9w8jc"
      }
    }
  ]
}