code/test-env/test_env_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anomaly Detection Framework Example\n",
    "\n",
    "This notebook provides a walkthrough of using the anomaly detection framework in a test environment. This test environment was used as UDL's InfluxDB instance was still being setup with SkySpark data during the project. The test environment populates an instance of InfluxDB (created using Docker) with sensor data from `../../data/labelled-skyspark-data/`. The sensor data was manually downloaded from SkySpark and corresponds with five sensors used in Phase 1 model testing.\n",
    "\n",
    "The walk-through includes the following steps:\n",
    "\n",
    "1. Create local InfluxDB instance\n",
    "2. Populate InfluxDB with sensor data\n",
    "3. Test Anomaly Detection Model Training\n",
    "4. Test Anomaly Detection Predictions\n",
    "5. Setup Dashboard\n",
    "6. Setup Notifications\n",
    "7. Notifications Test\n",
    "8. Example of Threshold Modification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import sys\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# influxdb_client is used to populate InfluxDB with the csv data\n",
    "from influxdb_client import InfluxDBClient\n",
    "from influxdb_client.client.write_api import SYNCHRONOUS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the model package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# files are contained in a sibling folder\n",
    "sys.path.append(\"..\")\n",
    "\n",
    "import model.clean as cl\n",
    "import model.model_trainer as mt\n",
    "import model.model_predict as mp\n",
    "from model.influx_interact import influx_class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.random import set_seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Create Local InfluxDB Instance\n",
    "\n",
    "Copy `docker-compose.yml` located in this directory to a local directory. Then run the command `docker-compose up` from this local directory.\n",
    "\n",
    "Go to `http://localhost:8086/` and enter `MDS2021` as user name and `mypassword` to log in to the user interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Populate InfluxDB with Sensor Data\n",
    "\n",
    "This step will populate InfluxDB with csv files located in `../../data/labelled-skyspark-data/`. These files correspond with the Phase 1 model testing. The code presented in this section is also available in `populate_influx.py`.\n",
    "\n",
    "Note that this step is just for creating the data in this test environment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH_TO_CSVS = \"../../data/labelled-skyspark-data/\"\n",
    "CSVS_TO_LOAD = [\n",
    "    \"CEC_compiled_data_1b_updated.csv\",\n",
    "    \"CEC_compiled_data_2b_updated.csv\",\n",
    "    \"CEC_compiled_data_3b_updated.csv\",\n",
    "    \"CEC_compiled_data_4b_updated.csv\",\n",
    "    \"CEC_compiled_data_5b_updated.csv\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a look up table for sensors and their manually labeled data sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "JOIN_MANUAL_ANOMALIES = True\n",
    "PATH_TO_LABELLED_CSVS = \"../../data/labelled-skyspark-data/\"\n",
    "LABELLED_LOOKUP = {\n",
    "    \"Campus Energy Centre Campus HW Main Meter Power\" : \"CEC_compiled_data_1b_updated.csv\",\n",
    "    \"Campus Energy Centre Campus HW Main Meter Entering Water Temperature\" : \"CEC_compiled_data_2b_updated.csv\",\n",
    "    \"Campus Energy Centre Campus HW Main Meter Flow\" : \"CEC_compiled_data_3b_updated.csv\",\n",
    "    \"Campus Energy Centre Boiler B-1 Gas Pressure\" : \"CEC_compiled_data_4b_updated.csv\",\n",
    "    \"Campus Energy Centre Boiler B-1 Exhaust O2\" : \"CEC_compiled_data_5b_updated.csv\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for data viewing in notebook\n",
    "pd.set_option('display.expand_frame_repr', False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up InfluxDB connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# as setup in docker-compose.yml\n",
    "token = \"mytoken\"\n",
    "org = \"UBC\"\n",
    "bucket = \"MDS2021\"\n",
    "\n",
    "# setup InfluxDB client\n",
    "client = InfluxDBClient(url=\"http://localhost:8086\", token=token, timeout=999_000)\n",
    "write_api = client.write_api(write_options=SYNCHRONOUS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read each csv file and write the data to InfluxDB. This sets up the sensor data in InfluxDB in the READINGS measurement mimicing how SkySpark data exists in InfluxDB. Note that only the tags/field required for anomaly detection are populated.\n",
    "\n",
    "Important note: If the influx write times out, re-run and it should work on the second try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "writing: CEC_compiled_data_1b_updated.csv\n",
      "writing: CEC_compiled_data_2b_updated.csv\n",
      "writing: CEC_compiled_data_3b_updated.csv\n",
      "writing: CEC_compiled_data_4b_updated.csv\n",
      "writing: CEC_compiled_data_5b_updated.csv\n"
     ]
    }
   ],
   "source": [
    "for csv in CSVS_TO_LOAD:\n",
    "\n",
    "    # load and set up dataframes\n",
    "    df = pd.read_csv(PATH_TO_CSVS + csv, parse_dates=[\"Datetime\"])\n",
    "    df.rename(columns={\"Value\": \"val_num\"}, inplace=True)\n",
    "    df.rename(columns={\"ID\": \"uniqueID\"}, inplace=True)\n",
    "    df.rename(columns={\"Anomaly\": \"AH\"}, inplace=True)\n",
    "    df[\"siteRef\"] = \"Campus Energy Centre\"\n",
    "    df.set_index(\"Datetime\", drop=True, inplace=True)\n",
    "    df = df.drop([\"AH\"], axis=1)\n",
    "\n",
    "    print(\"writing: {}\".format(csv))\n",
    "    # write values\n",
    "    write_api.write(\n",
    "        bucket,\n",
    "        org,\n",
    "        record=df,\n",
    "        data_frame_measurement_name=\"READINGS\",\n",
    "        data_frame_tag_columns=[\"uniqueID\", \"navName\", \"siteRef\"],\n",
    "    )\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the `df` object to see what was written to influx."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                     val_num                                    uniqueID               siteRef\n",
       "Datetime                                                                                      \n",
       "2020-01-01 07:45:00      2.9  Campus Energy Centre Boiler B-1 Exhaust O2  Campus Energy Centre\n",
       "2020-01-01 08:00:00      2.9  Campus Energy Centre Boiler B-1 Exhaust O2  Campus Energy Centre\n",
       "2020-01-01 08:15:00      2.9  Campus Energy Centre Boiler B-1 Exhaust O2  Campus Energy Centre\n",
       "2020-01-01 08:30:00      2.9  Campus Energy Centre Boiler B-1 Exhaust O2  Campus Energy Centre\n",
       "2020-01-01 08:45:00      2.9  Campus Energy Centre Boiler B-1 Exhaust O2  Campus Energy Centre"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>val_num</th>\n      <th>uniqueID</th>\n      <th>siteRef</th>\n    </tr>\n    <tr>\n      <th>Datetime</th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2020-01-01 07:45:00</th>\n      <td>2.9</td>\n      <td>Campus Energy Centre Boiler B-1 Exhaust O2</td>\n      <td>Campus Energy Centre</td>\n    </tr>\n    <tr>\n      <th>2020-01-01 08:00:00</th>\n      <td>2.9</td>\n      <td>Campus Energy Centre Boiler B-1 Exhaust O2</td>\n      <td>Campus Energy Centre</td>\n    </tr>\n    <tr>\n      <th>2020-01-01 08:15:00</th>\n      <td>2.9</td>\n      <td>Campus Energy Centre Boiler B-1 Exhaust O2</td>\n      <td>Campus Energy Centre</td>\n    </tr>\n    <tr>\n      <th>2020-01-01 08:30:00</th>\n      <td>2.9</td>\n      <td>Campus Energy Centre Boiler B-1 Exhaust O2</td>\n      <td>Campus Energy Centre</td>\n    </tr>\n    <tr>\n      <th>2020-01-01 08:45:00</th>\n      <td>2.9</td>\n      <td>Campus Energy Centre Boiler B-1 Exhaust O2</td>\n      <td>Campus Energy Centre</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sensor data has now been written to the InfluxDB READINGS measurement. A screenshot of what this looks like in InfluxDB is shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](demo_screenshots/step2.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Test Anomaly Detection Model Training\n",
    "\n",
    "This step tests model training. This would be typically run on a selected interval (for example every month) to update the anomaly detection models. A script for model training that can be used with UDL's InfluxDB instance is available in `../code/sensor_training.py`. Code that is only applicable to this test environment or differs from what would exist in `../code/sensor_training.py` is noted. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This provides the option to subset the training data for faster testing. Model training can be completed using the entire sensors record by setting this to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "TESTING = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide sensor threshold ratios for anomaly detection. In the prediciton stage a 99.5 percentile will be calculated on the loss and saved. On prediction this percentile value will be loaded.  In each case the threshold_ratio will be multipled with the percentile to get the threshold used for flagging an anomaly. Selecting the best ratio is a manual process.\n",
    "\n",
    "** IMPORTANT NOTE: These manual threshold ratios are the same as those used in the Phase 1 Tests and presented in the report with the exception of the HW Main Meter Flow. It was found (late in the project timeline) that there can be variation in models when retraining due to the non-deterministic nature of the LSTM-ED (even with setting a random seed due to training on a GPU). This can require the use of a different error threshold. This will need to be resolved in future study to ensure there isn't a high level of effort associated with setting the threshold ration when models are retrained for this framework to function in an automated fashion. A more sophisticated threshold rule that looks at the distribution of error instead of setting an error threshold should also provide a more stable model when retranining is completed. Standardizing the error shoudl also be investigated to help provide a more stable rule.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "THRESHOLD_RATIOS = {\n",
    "    \"Campus Energy Centre Campus HW Main Meter Power\": 1.8,\n",
    "    \"Campus Energy Centre Boiler B-1 Exhaust O2\": 1,\n",
    "    \"Campus Energy Centre Boiler B-1 Gas Pressure\": 0.23,\n",
    "    \"Campus Energy Centre Campus HW Main Meter Entering Water Temperature\": 1.2,\n",
    "    \"Campus Energy Centre Campus HW Main Meter Flow\": 1.72,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the selected sequence time step sizes. They are currently all the same, but can individually be changed. A default of 15 is also used if no values are entered for a sensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "TIME_STEP_SIZES = {\n",
    "    \"Campus Energy Centre Campus HW Main Meter Power\":15,\n",
    "    \"Campus Energy Centre Boiler B-1 Exhaust O2\":15,\n",
    "    \"Campus Energy Centre Boiler B-1 Gas Pressure\":15,\n",
    "    \"Campus Energy Centre Campus HW Main Meter Entering Water Temperature\":15,\n",
    "    \"Campus Energy Centre Campus HW Main Meter Flow\":15,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "End time (as unix time stamp), to be used when reading data for model training such that data that will be predicted during Step 4 of this test environment is not used in model training In the `../code/sensor_training.py script` there is no need to set an end time as the model will train on all available data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "END_TIME = 1613109600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the paths to save the model and standard scaler from the cleaning pipeline and create the InfluxDB client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = \"./test_env_models/\"\n",
    "scaler_path = \"./test_env_standardizers/\"\n",
    "percentile_path = \"./test_env_loss_percentiles/\"\n",
    "\n",
    "# setup InfluxDB client\n",
    "token = \"mytoken\"\n",
    "org = \"UBC\"\n",
    "bucket = \"MDS2021\"\n",
    "url = \"http://localhost:8086\"\n",
    "\n",
    "influx_conn = influx_class(\n",
    "    org=org,\n",
    "    url=url,\n",
    "    bucket=bucket,\n",
    "    token=token,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read data for model training from the InfluxDB READINGS measurement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_read_df = influx_conn.make_query(\n",
    "    location=\"Campus Energy Centre\",\n",
    "    measurement=\"READINGS\",\n",
    "    end=END_TIME,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data based on uniqueID into individual sensor dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_df_split_on_sensors = cl.split_sensors(influx_read_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `influx_df_split_on_sensors` object is a dictionary with the name of the sensor as the key and then the value is another dict of data objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "dict_keys(['Campus Energy Centre Boiler B-1 Exhaust O2', 'Campus Energy Centre Boiler B-1 Gas Pressure', 'Campus Energy Centre Campus HW Main Meter Entering Water Temperature', 'Campus Energy Centre Campus HW Main Meter Flow', 'Campus Energy Centre Campus HW Main Meter Power'])"
      ]
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "source": [
    "influx_df_split_on_sensors.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Populate the `manual_anomaly` column with False. This step is not required in `../code/sensor_training.py script` and is just to setup the `manual_anomal` tag in the `TRAINING_ANOMALY` measurement in InfluxDB. The `manual_anomaly` tag allows a user to set tags as `True` such that they are removed from the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in influx_df_split_on_sensors.keys():\n",
    "    df = influx_df_split_on_sensors[key].copy(deep=True)\n",
    "    df[\"manual_anomaly\"] = False\n",
    "    influx_df_split_on_sensors[key] = df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell provides model training by iterating over each sensor in `influx_df_split_on_sensors` and:\n",
    "\n",
    "1. Removes anomalous data based on manual_anomaly labels available in the TRAINING_ANOMALY measurement\n",
    "2. Standardizes the values for training and saves the standardizer\n",
    "3. Subsets the data for faster training if specified in the `TESTING` variable\n",
    "4. Sequences the values into windows for the LSTM-ED anomaly detection model\n",
    "5. Fits the LSTM-ED and saves the model \n",
    "6. Writes model training anomaly predictions to the TRAINING_ANOMALY Measurement model_anomaly field in InfluxDB\n",
    "\n",
    "**Note:** 3. only applies to this test environment and would not exist in `sensor_training.py`.\n",
    "\n",
    "**IMPORTANT NOTE: InfluxDB does not overwrite tags. For example, if a datetime entry and sensor value was originally input to the `TRAINING_ANOMALY` measurement in InfluxDB with the tag `model_anomaly` as False and `manual_anomaly` as False and a new entry was written to InfluxDB with the same dateime entry and sensor value but with tags `model_anomaly` as True and `manual_anomaly` as False, this will create a new duplicated date entry in InfluxDB but with the different tags. Accordingly, anytime tags are updated in InfluxDB, the data from InfluxDB should be read, the tags changed, then the data deleted from InfluxDB before the data with the changed tags are written in InfluxDB.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "961\n",
      "Campus Energy Centre Boiler B-1 Exhaust O2\n",
      "loss percentile  0.6341222702479441\n",
      "threshold_ratio  1\n",
      "threshold 0.6341222702479441\n",
      "               uniqueID  val_num  manual_anomaly\n",
      "model_anomaly                                   \n",
      "False             29879    29879           29879\n",
      "True                107      107             107\n",
      "                uniqueID  val_num  model_anomaly\n",
      "manual_anomaly                                  \n",
      "False              29986    29986          29986\n",
      "Training for : Campus Energy Centre Boiler B-1 Gas Pressure\n",
      "_measurement=\"TRAINING_ANOMALY\" AND uniqueID=\"Campus Energy Centre Boiler B-1 Gas Pressure\"\n",
      "Epoch 1/100\n",
      "29/29 [==============================] - 1s 44ms/step - loss: 0.2114 - val_loss: 0.2496\n",
      "Epoch 2/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.1464 - val_loss: 0.1980\n",
      "Epoch 3/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0976 - val_loss: 0.1427\n",
      "Epoch 4/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0784 - val_loss: 0.1155\n",
      "Epoch 5/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0696 - val_loss: 0.0905\n",
      "Epoch 6/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0626 - val_loss: 0.0967\n",
      "Epoch 7/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0535 - val_loss: 0.1259\n",
      "Epoch 8/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0475 - val_loss: 0.0492\n",
      "Epoch 9/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0435 - val_loss: 0.0418\n",
      "Epoch 10/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0418 - val_loss: 0.0361\n",
      "Epoch 11/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0408 - val_loss: 0.0463\n",
      "Epoch 12/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0387 - val_loss: 0.0404\n",
      "Epoch 13/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0386 - val_loss: 0.0865\n",
      "Epoch 14/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0397 - val_loss: 0.0537\n",
      "Epoch 15/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0359 - val_loss: 0.0584\n",
      "Epoch 16/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0363 - val_loss: 0.0357\n",
      "Epoch 17/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0341 - val_loss: 0.0172\n",
      "Epoch 18/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0370 - val_loss: 0.0541\n",
      "Epoch 19/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0345 - val_loss: 0.0247\n",
      "Epoch 20/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0340 - val_loss: 0.0365\n",
      "Epoch 21/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0406 - val_loss: 0.0504\n",
      "Epoch 22/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0339 - val_loss: 0.0406\n",
      "Epoch 23/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0332 - val_loss: 0.0305\n",
      "Epoch 24/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0323 - val_loss: 0.0353\n",
      "Epoch 25/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0318 - val_loss: 0.0184\n",
      "Epoch 26/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0326 - val_loss: 0.0243\n",
      "Epoch 27/100\n",
      "29/29 [==============================] - 1s 23ms/step - loss: 0.0333 - val_loss: 0.0465\n",
      "Campus Energy Centre Boiler B-1 Gas Pressure\n",
      "loss percentile  5.356589291853497\n",
      "threshold_ratio  0.23\n",
      "threshold 1.2320155371263044\n",
      "               uniqueID  val_num  manual_anomaly\n",
      "model_anomaly                                   \n",
      "False             29508    29508           29508\n",
      "True                478      478             478\n",
      "                uniqueID  val_num  model_anomaly\n",
      "manual_anomaly                                  \n",
      "False              29986    29986          29986\n",
      "Training for : Campus Energy Centre Campus HW Main Meter Entering Water Temperature\n",
      "_measurement=\"TRAINING_ANOMALY\" AND uniqueID=\"Campus Energy Centre Campus HW Main Meter Entering Water Temperature\"\n",
      "Epoch 1/100\n",
      "29/29 [==============================] - 1s 46ms/step - loss: 0.1202 - val_loss: 0.0683\n",
      "Epoch 2/100\n",
      "29/29 [==============================] - 1s 25ms/step - loss: 0.0673 - val_loss: 0.0626\n",
      "Epoch 3/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0576 - val_loss: 0.0404\n",
      "Epoch 4/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0484 - val_loss: 0.0476\n",
      "Epoch 5/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0427 - val_loss: 0.0469\n",
      "Epoch 6/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0387 - val_loss: 0.0336\n",
      "Epoch 7/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0342 - val_loss: 0.0234\n",
      "Epoch 8/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0280 - val_loss: 0.0251\n",
      "Epoch 9/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0304 - val_loss: 0.0288\n",
      "Epoch 10/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0261 - val_loss: 0.0125\n",
      "Epoch 11/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0252 - val_loss: 0.0157\n",
      "Epoch 12/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0218 - val_loss: 0.0131\n",
      "Epoch 13/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0210 - val_loss: 0.0070\n",
      "Epoch 14/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0213 - val_loss: 0.0240\n",
      "Epoch 15/100\n",
      "29/29 [==============================] - 1s 25ms/step - loss: 0.0215 - val_loss: 0.0098\n",
      "Epoch 16/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0230 - val_loss: 0.0065\n",
      "Epoch 17/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0189 - val_loss: 0.0062\n",
      "Epoch 18/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0184 - val_loss: 0.0126\n",
      "Epoch 19/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0179 - val_loss: 0.0031\n",
      "Epoch 20/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0173 - val_loss: 0.0109\n",
      "Epoch 21/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0174 - val_loss: 0.0101\n",
      "Epoch 22/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0175 - val_loss: 0.0042\n",
      "Epoch 23/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0176 - val_loss: 0.0101\n",
      "Epoch 24/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0175 - val_loss: 0.0138\n",
      "Epoch 25/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0188 - val_loss: 0.0123\n",
      "Epoch 26/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0166 - val_loss: 0.0061\n",
      "Epoch 27/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0164 - val_loss: 0.0020\n",
      "Epoch 28/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0162 - val_loss: 0.0028\n",
      "Epoch 29/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0158 - val_loss: 0.0053\n",
      "Epoch 30/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0162 - val_loss: 0.0215\n",
      "Epoch 31/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0170 - val_loss: 0.0136\n",
      "Epoch 32/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0172 - val_loss: 0.0066\n",
      "Epoch 33/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0163 - val_loss: 0.0050\n",
      "Epoch 34/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0163 - val_loss: 0.0016\n",
      "Epoch 35/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0160 - val_loss: 0.0029\n",
      "Epoch 36/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0162 - val_loss: 0.0072\n",
      "Epoch 37/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0156 - val_loss: 0.0025\n",
      "Epoch 38/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0162 - val_loss: 0.0083\n",
      "Epoch 39/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0166 - val_loss: 0.0128\n",
      "Epoch 40/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0165 - val_loss: 0.0064\n",
      "Epoch 41/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0159 - val_loss: 0.0071\n",
      "Epoch 42/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0164 - val_loss: 0.0058\n",
      "Epoch 43/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0159 - val_loss: 0.0101\n",
      "Epoch 44/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0167 - val_loss: 0.0161\n",
      "Campus Energy Centre Campus HW Main Meter Entering Water Temperature\n",
      "loss percentile  0.29426034690627445\n",
      "threshold_ratio  1.2\n",
      "threshold 0.3531124162875293\n",
      "               uniqueID  val_num  manual_anomaly\n",
      "model_anomaly                                   \n",
      "False             29918    29918           29918\n",
      "True                 68       68              68\n",
      "                uniqueID  val_num  model_anomaly\n",
      "manual_anomaly                                  \n",
      "False              29986    29986          29986\n",
      "Training for : Campus Energy Centre Campus HW Main Meter Flow\n",
      "_measurement=\"TRAINING_ANOMALY\" AND uniqueID=\"Campus Energy Centre Campus HW Main Meter Flow\"\n",
      "Epoch 1/100\n",
      "29/29 [==============================] - 2s 52ms/step - loss: 0.4233 - val_loss: 0.2319\n",
      "Epoch 2/100\n",
      "29/29 [==============================] - 1s 25ms/step - loss: 0.2206 - val_loss: 0.1368\n",
      "Epoch 3/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1433 - val_loss: 0.1031\n",
      "Epoch 4/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1297 - val_loss: 0.0698\n",
      "Epoch 5/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1092 - val_loss: 0.0664\n",
      "Epoch 6/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1253 - val_loss: 0.1015\n",
      "Epoch 7/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1123 - val_loss: 0.1070\n",
      "Epoch 8/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1080 - val_loss: 0.0560\n",
      "Epoch 9/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0961 - val_loss: 0.0643\n",
      "Epoch 10/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0937 - val_loss: 0.0692\n",
      "Epoch 11/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0981 - val_loss: 0.0605\n",
      "Epoch 12/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0900 - val_loss: 0.0716\n",
      "Epoch 13/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0934 - val_loss: 0.0516\n",
      "Epoch 14/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0850 - val_loss: 0.0500\n",
      "Epoch 15/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0851 - val_loss: 0.0648\n",
      "Epoch 16/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0862 - val_loss: 0.1385\n",
      "Epoch 17/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0931 - val_loss: 0.0570\n",
      "Epoch 18/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0875 - val_loss: 0.0452\n",
      "Epoch 19/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0846 - val_loss: 0.0723\n",
      "Epoch 20/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0826 - val_loss: 0.1918\n",
      "Epoch 21/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.1035 - val_loss: 0.0721\n",
      "Epoch 22/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0909 - val_loss: 0.0426\n",
      "Epoch 23/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0817 - val_loss: 0.0427\n",
      "Epoch 24/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0777 - val_loss: 0.0608\n",
      "Epoch 25/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0836 - val_loss: 0.0401\n",
      "Epoch 26/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0812 - val_loss: 0.0401\n",
      "Epoch 27/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0793 - val_loss: 0.0446\n",
      "Epoch 28/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0784 - val_loss: 0.0489\n",
      "Epoch 29/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0745 - val_loss: 0.0357\n",
      "Epoch 30/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0744 - val_loss: 0.0557\n",
      "Epoch 31/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0710 - val_loss: 0.0329\n",
      "Epoch 32/100\n",
      "29/29 [==============================] - 1s 25ms/step - loss: 0.0720 - val_loss: 0.0315\n",
      "Epoch 33/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0718 - val_loss: 0.0544\n",
      "Epoch 34/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0703 - val_loss: 0.0826\n",
      "Epoch 35/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0758 - val_loss: 0.0317\n",
      "Epoch 36/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0669 - val_loss: 0.0524\n",
      "Epoch 37/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0695 - val_loss: 0.0246\n",
      "Epoch 38/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0660 - val_loss: 0.0284\n",
      "Epoch 39/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0655 - val_loss: 0.0233\n",
      "Epoch 40/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0644 - val_loss: 0.0207\n",
      "Epoch 41/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0670 - val_loss: 0.0214\n",
      "Epoch 42/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0603 - val_loss: 0.0271\n",
      "Epoch 43/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0609 - val_loss: 0.0377\n",
      "Epoch 44/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0724 - val_loss: 0.0350\n",
      "Epoch 45/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0611 - val_loss: 0.0172\n",
      "Epoch 46/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0624 - val_loss: 0.0332\n",
      "Epoch 47/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0594 - val_loss: 0.0164\n",
      "Epoch 48/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0586 - val_loss: 0.0203\n",
      "Epoch 49/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0581 - val_loss: 0.0305\n",
      "Epoch 50/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0609 - val_loss: 0.0173\n",
      "Epoch 51/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0587 - val_loss: 0.0239\n",
      "Epoch 52/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0570 - val_loss: 0.0160\n",
      "Epoch 53/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0577 - val_loss: 0.0183\n",
      "Epoch 54/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0592 - val_loss: 0.0403\n",
      "Epoch 55/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0549 - val_loss: 0.0190\n",
      "Epoch 56/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0547 - val_loss: 0.0157\n",
      "Epoch 57/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0549 - val_loss: 0.0315\n",
      "Epoch 58/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0546 - val_loss: 0.0271\n",
      "Epoch 59/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0544 - val_loss: 0.0369\n",
      "Epoch 60/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0545 - val_loss: 0.0132\n",
      "Epoch 61/100\n",
      "29/29 [==============================] - 1s 25ms/step - loss: 0.0543 - val_loss: 0.0375\n",
      "Epoch 62/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0590 - val_loss: 0.0186\n",
      "Epoch 63/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0538 - val_loss: 0.0203\n",
      "Epoch 64/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0571 - val_loss: 0.0576\n",
      "Epoch 65/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0544 - val_loss: 0.0189\n",
      "Epoch 66/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0527 - val_loss: 0.0397\n",
      "Epoch 67/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0523 - val_loss: 0.0427\n",
      "Epoch 68/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0524 - val_loss: 0.0181\n",
      "Epoch 69/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0507 - val_loss: 0.0288\n",
      "Epoch 70/100\n",
      "29/29 [==============================] - 1s 24ms/step - loss: 0.0518 - val_loss: 0.0167\n",
      "Campus Energy Centre Campus HW Main Meter Flow\n",
      "loss percentile  0.2736800010477066\n",
      "threshold_ratio  1.72\n",
      "threshold 0.47072960180205536\n",
      "               uniqueID  val_num  manual_anomaly\n",
      "model_anomaly                                   \n",
      "False             29928    29928           29928\n",
      "True                 58       58              58\n",
      "                uniqueID  val_num  model_anomaly\n",
      "manual_anomaly                                  \n",
      "False              29986    29986          29986\n",
      "Training for : Campus Energy Centre Campus HW Main Meter Power\n",
      "_measurement=\"TRAINING_ANOMALY\" AND uniqueID=\"Campus Energy Centre Campus HW Main Meter Power\"\n",
      "Epoch 1/100\n",
      "29/29 [==============================] - 1s 47ms/step - loss: 0.3264 - val_loss: 0.3923\n",
      "Epoch 2/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.1583 - val_loss: 0.1733\n",
      "Epoch 3/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.1252 - val_loss: 0.1255\n",
      "Epoch 4/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.1161 - val_loss: 0.1244\n",
      "Epoch 5/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.1022 - val_loss: 0.1172\n",
      "Epoch 6/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0991 - val_loss: 0.1111\n",
      "Epoch 7/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0946 - val_loss: 0.1040\n",
      "Epoch 8/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.0949 - val_loss: 0.1897\n",
      "Epoch 9/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0928 - val_loss: 0.1250\n",
      "Epoch 10/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0865 - val_loss: 0.0917\n",
      "Epoch 11/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0819 - val_loss: 0.0770\n",
      "Epoch 12/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0790 - val_loss: 0.0743\n",
      "Epoch 13/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0791 - val_loss: 0.1668\n",
      "Epoch 14/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0792 - val_loss: 0.1567\n",
      "Epoch 15/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0765 - val_loss: 0.1793\n",
      "Epoch 16/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0701 - val_loss: 0.0742\n",
      "Epoch 17/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0640 - val_loss: 0.0307\n",
      "Epoch 18/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.0635 - val_loss: 0.0666\n",
      "Epoch 19/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0620 - val_loss: 0.1028\n",
      "Epoch 20/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.0604 - val_loss: 0.0715\n",
      "Epoch 21/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0606 - val_loss: 0.0716\n",
      "Epoch 22/100\n",
      "29/29 [==============================] - 1s 28ms/step - loss: 0.0572 - val_loss: 0.1350\n",
      "Epoch 23/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0621 - val_loss: 0.0814\n",
      "Epoch 24/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0545 - val_loss: 0.0692\n",
      "Epoch 25/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0536 - val_loss: 0.0767\n",
      "Epoch 26/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0537 - val_loss: 0.1272\n",
      "Epoch 27/100\n",
      "29/29 [==============================] - 1s 27ms/step - loss: 0.0532 - val_loss: 0.0671\n",
      "Campus Energy Centre Campus HW Main Meter Power\n",
      "loss percentile  0.6330957491760557\n",
      "threshold_ratio  1.8\n",
      "threshold 1.1395723485169003\n",
      "               uniqueID  val_num  manual_anomaly\n",
      "model_anomaly                                   \n",
      "False             29950    29950           29950\n",
      "True                 36       36              36\n",
      "                uniqueID  val_num  model_anomaly\n",
      "manual_anomaly                                  \n",
      "False              29986    29986          29986\n"
     ]
    }
   ],
   "source": [
    "set_seed(2)\n",
    "for key, df in influx_df_split_on_sensors.items():\n",
    "    print(\"Training for : {}\".format(key))\n",
    "\n",
    "    # Delete data that is about to be written, to prevent duplicaete values in InfluxDB measurements\n",
    "    delete_predicate = '_measurement=\"TRAINING_ANOMALY\" AND uniqueID=\"{}\"'.format(key)\n",
    "    print(delete_predicate)\n",
    "    delete_api = client.delete_api()\n",
    "    min_time = df[\"DateTime\"].values[0]\n",
    "    max_time = df[\"DateTime\"].values[-1]\n",
    "    delete_api.delete(\n",
    "        str(min_time) + \"Z\",\n",
    "        str(max_time) + \"Z\",\n",
    "        delete_predicate,\n",
    "        bucket=bucket,\n",
    "        org=org,\n",
    "    )\n",
    "\n",
    "    # avoid stale imports in notebooks (notebook specific code)\n",
    "    import importlib\n",
    "    importlib.reload(mt)\n",
    "    importlib.reload(mp)\n",
    "    importlib.reload(cl)\n",
    "\n",
    "    # creates standardized column for each sensor in main bucket\n",
    "    df[\"Stand_Val\"] = cl.std_val_train(\n",
    "        df[[\"Value\"]],\n",
    "        influx_df_split_on_sensors[key][\"ID\"].any(),\n",
    "        scaler_path,\n",
    "    )\n",
    "\n",
    "    # train on only data points not flagged manually\n",
    "    df = df[df.manual_anomaly != True]\n",
    "\n",
    "    if TESTING:\n",
    "        df = df.tail(30000)\n",
    "\n",
    "\n",
    "    # creates sequences for sliding windows for training\n",
    "    threshold_ratio = THRESHOLD_RATIOS[key]\n",
    "    time_steps = TIME_STEP_SIZES[key]\n",
    "    window_size = time_steps\n",
    "    x_train, y_train = mt.create_sequences(df[\"Stand_Val\"], df[\"Stand_Val\"], time_steps, window_size)\n",
    "\n",
    "    x_eval, y_eval = mt.create_sequences(df[\"Stand_Val\"], df[\"Stand_Val\"], time_steps, 1)\n",
    "\n",
    "    # trains and saves model\n",
    "    normal_dict = cl.model_parser(df, x_train, y_train, x_eval)\n",
    "    mt.fit_models(normal_dict, model_path)\n",
    "\n",
    "    # creates sequences for sliding windows for predicting on the train set\n",
    "    timestamps = df[\"DateTime\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "    val_nums = df[\"Value\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "    manual_anomaly = df[\"manual_anomaly\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "    loss_percentile = cl.load_loss_percentile(key, file_path=percentile_path)\n",
    "    threshold = loss_percentile * threshold_ratio\n",
    "\n",
    "    print(key)\n",
    "    print(\"loss percentile \", loss_percentile)\n",
    "    print(\"threshold_ratio \", threshold_ratio)\n",
    "    print(\"threshold\", threshold)\n",
    "\n",
    "    # predicting and prediction formatting\n",
    "    pred_df = mp.make_prediction(\n",
    "        key,\n",
    "        x_eval,\n",
    "        timestamps,\n",
    "        threshold,\n",
    "        val_nums,\n",
    "        model_path,\n",
    "        anomaly_type=\"model_anomaly\",\n",
    "        manual_anomaly = manual_anomaly\n",
    "    )\n",
    "    pred_df = pred_df[[\"uniqueID\", \"val_num\", \"model_anomaly\", \"manual_anomaly\"]]\n",
    "\n",
    "    print(pred_df.groupby(\"model_anomaly\").count())\n",
    "    print(pred_df.groupby(\"manual_anomaly\").count())\n",
    "\n",
    "\n",
    "    influx_conn.write_data(pred_df, \"TRAINING_ANOMALY\", tags=[\"uniqueID\", \"model_anomaly\", \"manual_anomaly\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the files in the saved model directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Campus Energy Centre Boiler B-1 Exhaust O2\nCampus Energy Centre Boiler B-1 Gas Pressure\nCampus Energy Centre Campus HW Main Meter Entering Water Temperature\nCampus Energy Centre Campus HW Main Meter Flow\nCampus Energy Centre Campus HW Main Meter Power\nREADME.md\n"
     ]
    }
   ],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "model_files = [f for f in listdir(model_path) if isfile(join(model_path, f))]\n",
    "for model_file in model_files:\n",
    "    print(model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following screenshots show InfluxDB with the TRAINING_ANOMALY measurements with the model_anomaly field written from the above process. Note that as the anomaly labels are tags, it is best to view the data as a scatter plot with the symbol column as uniqueID and the fill column as `model_anomaly` or `manual_anomaly`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](demo_screenshots/step3.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Test Anomaly Detection Predictions\n",
    "\n",
    "This step tests anomaly predictions and includes reading recent data from InfluxDB (including the window of data required to make predictions), loading previously saved anomaly detection models, running these models on the data to provide predictions, and writing the results back to InfluxDB. This would be typically by completed on a high frequency interval (for example every minute or 5 minutes). A script for anomaly predictions that can be used with UDL's InfluxDB instance is available in `../code/sensor_predict.py`. Code that is only applicable to this test environment or differs from what would exist in `../code/sensor_predict.py` is noted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First setup start and end times for the prediction data set in this testing environment. In `sensor_predict.py` END_TIME would be `now()` and START_TIME would be `now() - 1d`. Note that the timestamps below are UNIX format.\n",
    "\n",
    "The START_TIME here is the END_TIME from the training read. This uses all the data up until `END_TIME` and then uses past `END_TIME` for predicting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# END TIME FOR TRAINING SET BECOMES PREDICTING'S START TIME\n",
    "START_TIME = 1613109600\n",
    "END_TIME = 1613196000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read data from InfluxDB:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_read_df_for_pred = influx_conn.make_query(\n",
    "    location=\"Campus Energy Centre\",\n",
    "    measurement=\"READINGS\",\n",
    "    start=START_TIME,\n",
    "    end=END_TIME,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data based on uniqueID into individual sensor dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_for_pred = cl.split_sensors(influx_read_df_for_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell provides predictions by iterating over each sensor in `dfs_for_pred` and:\n",
    "\n",
    "1. Standardizes the values for training by loading the standardizer\n",
    "2. Sequences the values into windows for the LSTM-ED and other reshaping for the prediction step\n",
    "3. Creates predictions for the data and returns the prediction object\n",
    "4. Shapes the prediction object and write predictions to the PREDICT_ANOMALY Measurement realtime_anomaly field in InfluxDB\n",
    "\n",
    "**Note: measurements do not need to be deleted from InfluxDB before data is written in this step as the prediction step only provides data to the `PREDICT_ANOMALY` measurement in InfluxDB that does not already exist in the measurement.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key, df in dfs_for_pred.items():\n",
    "    dfs_for_pred[key][\"Stand_Val\"] = cl.std_val_predict(\n",
    "        dfs_for_pred[key][[\"Value\"]],\n",
    "        dfs_for_pred[key][\"ID\"].any(),\n",
    "        scaler_path,\n",
    "    )\n",
    "\n",
    "    # avoid stale imports in notebooks\n",
    "    import importlib\n",
    "    importlib.reload(mp)\n",
    "\n",
    "    # creates arrays for sliding windows\n",
    "    time_steps = TIME_STEP_SIZES[key]\n",
    "    window_size = 1\n",
    "    x_train, y_train = mt.create_sequences(df[\"Stand_Val\"], df[\"Stand_Val\"], time_steps, window_size)\n",
    "\n",
    "    # set up lists for passing to predict\n",
    "    timestamps = df[\"DateTime\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "    val_nums = df[\"Value\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "\n",
    "    loss_percentile = cl.load_loss_percentile(key, file_path=percentile_path)\n",
    "    threshold = THRESHOLD_RATIOS[key] * loss_percentile\n",
    "\n",
    "\n",
    "    # predicting and prediction formatting\n",
    "    pred_df = mp.make_prediction(\n",
    "        key,\n",
    "        x_train,\n",
    "        timestamps,\n",
    "        threshold,\n",
    "        val_nums,\n",
    "        model_path,\n",
    "        anomaly_type=\"realtime_anomaly\"\n",
    "    )\n",
    "    pred_df = pred_df[[\"uniqueID\", \"val_num\", \"realtime_anomaly\"]]\n",
    "\n",
    "    influx_conn.write_data(pred_df, \"PREDICT_ANOMALY\", tags=[\"uniqueID\", \"realtime_anomaly\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predictions are now written to InfluxDB and a screenshot of the PREDICT_ANOMALY measurement in InfluxDB is shown below. Note that as the anomaly labels are tags, it is best to view the data as a scatter plot with the symbol column as uniqueID and the fill column as `realtime_anomaly`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](demo_screenshots/step4.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test environment will now have three measurements:\n",
    "\n",
    "- READINGS: the raw data  \n",
    "- TRAINING_ANOMALY: data with the `manual_anomaly` tag (if a user has input this) and `model_anomaly` tag generated from model training step\n",
    "- PREDICT_ANOMALY: data with the `realtime_anomaly` tag generated from the prediction step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Setup Dashboard\n",
    "\n",
    "A template for the dashboard has been provided in this `test-env` directory as `cec_boiler_sensors_(test).json`\n",
    "\n",
    "**To upload the dashboard template:**\n",
    "\n",
    "1. Navigate to the `Dashboards` tab on the left panel of the influxdb user interface  \n",
    "2. Click `Create Dashboard` in the top right  \n",
    "3. Click `Import Dashboard` from the drop down  \n",
    "4. Click then upload `cec_boiler_sensors_(test).json`  \n",
    "5. Click the new dashboard to view, you will have to change the start date to view data (try 2020-12-20 to now)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dashboard allows the user to change the time period viewed, whether the dashboard is continuously updated, and select the `anomaly_type` variable dropdown which will change the view between: `manual` (user entered `manual_anomaly` tag from the `TRAINING_ANOMALY` measurement), `model` (model training predictions from the `model_anomaly` tag from the `TRAINING_ANOMALY` measurement), or realtime (realtime predictions from the `realtime_anomaly` tag from the `PREDICT_ANOMALY` measurement`).\n",
    "\n",
    "Screenshot of the user controls and the full dashboard for the 5 sensors are shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](demo_screenshots/step5a.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](demo_screenshots/step5b.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several challenges with the dashboard interface in InfluxDB including:\n",
    "\n",
    "- Coloring is not consistent between True/False labels on graphs and it does not appear possible to change this\n",
    "- It does not appear possible to change the point sizes\n",
    "- anomalies were input as tags as plotting boolean field data did not appear possible (there are workarounds to this)\n",
    "\n",
    "As such, it is recommended to explore Grafana if additional styling/capability is required. It may also be necessary to consider modifing the schema such that anomalies are field values intsead of tag values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Setup Notifications\n",
    "\n",
    "This is done manually within InfluxDB. There may be a way to upload a template but this was not explored. The notification rule works by filtering for any data in the `PREDICT_ANOMALY` measurement that has the `realtime_anomaly` tag = True.\n",
    "\n",
    "The notification functionality was tested at a very high level in this study. Basically tyring to answer the question: can notifications be sent using InfluxDB on predicted anomalous data. The answer is `Yes` but additional investigations on notification settings should be completed.\n",
    "\n",
    "The process involves creating three objects:\n",
    "\n",
    "1. Checks  \n",
    "2. Notification Endpoints  \n",
    "3. Notification Rules  \n",
    "\n",
    "### 1) To Create a Check\n",
    "\n",
    "1. Navigate to the `Alerts` tab on the left panel of the influxdb user interface.\n",
    "2. Click `Create` in the top right  \n",
    "3. Click `Threshold Check` from the drop down  \n",
    "4. Define the query to look like (note that prior to creating this, the data explorer must be set on a timeframe that contains data): \n",
    "\n",
    "![query](./demo_screenshots/step6_1a.png)\n",
    "\n",
    "5. Configure Check as follows: \n",
    "\n",
    "![check](./demo_screenshots/step6_1b.png)\n",
    "\n",
    "6. Click the green check box\n",
    "\n",
    "### 2) To Create an Endpoint\n",
    "1. Create a new slack app and copy the incoming webhook https://api.slack.com/messaging/webhooks#create_a_webhook  \n",
    "2. Click `Notification Endpoints` on the middle banner  \n",
    "3. Click `Create` in the top right  \n",
    "4. Choose `Slack` from the drop down, name the endpoint, and paste your incoming webhook from your slack app and click Create    \n",
    "\n",
    "### 3) To Create a Notification Rule  \n",
    "1. Click `Notification Rules` from the middle banner  \n",
    "2. Click `Create` in the top right  \n",
    "3. Configure the Notification Rule to look like:\n",
    "\n",
    "![rule](./demo_screenshots/step6_2a.png)\n",
    "\n",
    "4. Click `Create Notification Rule`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - Dashboard/Notification Test\n",
    "\n",
    "Upload data that has been flagged as anomalous to InfluxDB to test the notification system.\n",
    "\n",
    "The test data is set up to have 3 time stamps, now, 5 mins ago, and 10 mins ago. The notification system will only trigger on fresh data.\n",
    "\n",
    "**NOTE:** It was found during testing that notifications were sometimes inconsistent. Additional testing on the notification system would be required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                     val_num                                         uniqueID realtime_anomaly\n",
       "DateTime                                                                                      \n",
       "1624232230062270100    140.0  Campus Energy Centre Campus HW Main Meter Power             True\n",
       "1624231930062270208    -40.0  Campus Energy Centre Campus HW Main Meter Power             True\n",
       "1624231630062270208     40.0  Campus Energy Centre Campus HW Main Meter Power            False"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>val_num</th>\n      <th>uniqueID</th>\n      <th>realtime_anomaly</th>\n    </tr>\n    <tr>\n      <th>DateTime</th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1624232230062270100</th>\n      <td>140.0</td>\n      <td>Campus Energy Centre Campus HW Main Meter Power</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>1624231930062270208</th>\n      <td>-40.0</td>\n      <td>Campus Energy Centre Campus HW Main Meter Power</td>\n      <td>True</td>\n    </tr>\n    <tr>\n      <th>1624231630062270208</th>\n      <td>40.0</td>\n      <td>Campus Energy Centre Campus HW Main Meter Power</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "source": [
    "DateTime = [int(time.time_ns()), int(time.time_ns() - 3e11), int(time.time_ns() - 6e11),]\n",
    "val_num = [140.0, -40.0, 40.0]\n",
    "realtime_anomaly = [\"True\", \"True\", \"False\"]\n",
    "uniqueID = [\"Campus Energy Centre Campus HW Main Meter Power\"] * 3\n",
    "\n",
    "data = {\"DateTime\": DateTime, \"val_num\":val_num, \"uniqueID\":uniqueID, \"realtime_anomaly\": realtime_anomaly}\n",
    "test_realtime = pd.DataFrame(data)\n",
    "test_realtime.set_index(\"DateTime\", drop=True, inplace=True)\n",
    "test_realtime.index.rename(\"DateTime\", inplace=True)\n",
    "test_realtime.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_conn.write_data(test_realtime, \"PREDICT_ANOMALY\", tags=[\"uniqueID\", \"realtime_anomaly\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A flagged point will appear in the check's history:\n",
    "\n",
    "![notification](./demo_screenshots/step7a.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a notification will be pushed to slack:\n",
    "\n",
    "![notification](./demo_screenshots/step7b.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8 - Example of Threshold Modification\n",
    "\n",
    "This step looks at rerunning the anomaly predictions with different thresholds. This doesnt retrain the model, just runs a prediction based on the new threshold set. This requires the training to have already been run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose the time period for the new analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TIME = 1613109600\n",
    "END_TIME = 1613196000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose a sensor and a new threshold ratio to test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['Campus Energy Centre Campus HW Main Meter Flow']\n"
     ]
    }
   ],
   "source": [
    "update_data = {\n",
    "    \"Campus Energy Centre Campus HW Main Meter Flow\" : 1.5\n",
    "}\n",
    "print(list(update_data.keys()))"
   ]
  },
  {
   "source": [
    "Create the connection to read from InfluxDB:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_read_df_for_pred = influx_conn.make_query(\n",
    "    location=\"Campus Energy Centre\",\n",
    "    measurement=\"READINGS\",\n",
    "    start=START_TIME,\n",
    "    end=END_TIME,\n",
    "    id = list(update_data.keys())\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_for_test = cl.split_sensors(influx_read_df_for_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Name the measurement you want your threshold experiment to be sent to (alternatively, this could write directly to the `TRAINING_ANOMALY` measurement, just note that the measurement data will need to be deleted before rewriting as described in Step 4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "measurement_name = \"TEST_THRESHOLD_METER_FLOW\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Campus Energy Centre Campus HW Main Meter Flow\n"
     ]
    }
   ],
   "source": [
    "for key, df in dfs_for_test.items():\n",
    "    dfs_for_test[key][\"Stand_Val\"] = cl.std_val_predict(\n",
    "        dfs_for_test[key][[\"Value\"]],\n",
    "        dfs_for_test[key][\"ID\"].any(),\n",
    "        scaler_path,\n",
    "    )\n",
    "    print(key)\n",
    "\n",
    "    # keeps external packages updated in\n",
    "    import importlib\n",
    "    importlib.reload(mp)\n",
    "\n",
    "    # sets up sequencing\n",
    "    time_steps = TIME_STEP_SIZES[key]\n",
    "    window_size = 1\n",
    "    x_train, y_train = mt.create_sequences(df[\"Stand_Val\"], df[\"Stand_Val\"], time_steps, window_size)\n",
    "\n",
    "    # set up lists for passing to predict\n",
    "    timestamps = df[\"DateTime\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "    val_nums = df[\"Value\"].tail(len(df) - x_train.shape[1]+1).values\n",
    "\n",
    "    # gets training loss percentile for threshold setting\n",
    "    loss_percentile = cl.load_loss_percentile(key, file_path=percentile_path)\n",
    "    threshold = THRESHOLD_RATIOS[key] * loss_percentile\n",
    "\n",
    "\n",
    "    # predicting and prediction formatting\n",
    "    pred_df = mp.make_prediction(\n",
    "        key,\n",
    "        x_train,\n",
    "        timestamps,\n",
    "        threshold,\n",
    "        val_nums,\n",
    "        model_path,\n",
    "        anomaly_type=\"realtime_anomaly\"\n",
    "    )\n",
    "    pred_df = pred_df[[\"uniqueID\", \"val_num\", \"realtime_anomaly\"]]\n",
    "\n",
    "    influx_conn.write_data(pred_df, measurement_name, tags=[\"uniqueID\", \"realtime_anomaly\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "influx_conn.client.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.3 64-bit ('base': conda)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "metadata": {
   "interpreter": {
    "hash": "7d2dde933846643c70a0971eba9e216bf0ce928f2ec74020d36b05b86c71e3e9"
   }
  },
  "interpreter": {
   "hash": "f8276c093d7ded3cd3b967974e4dc7a67e95d353506c8a69e8f4f7f803efaf80"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}