Recurrent_Neural_Network_Play_Generator.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Recurrent Neural Network Play Generator.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMt0UtUOmZNGL2q20F9eoQi",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nasiksami/Deep-Learning/blob/main/Recurrent_Neural_Network_Play_Generator.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lcHLIGEQE20u"
      },
      "source": [
        "#####From Romeo and Juliet play we will feed our model with many data. Then the model will predict the next character based on the previous character. So we will input some starting point for the play and eventually the model will make up the whole lines by combining all the predicted characters\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WUo6w4E1FaMF",
        "outputId": "d1881218-db78-499b-f6b6-05b8865a673a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "%tensorflow_version 2.x  # this line is not required unless you are in a notebook\n",
        "from keras.preprocessing import sequence\n",
        "import keras\n",
        "import tensorflow as tf\n",
        "import os\n",
        "import numpy as np"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "`%tensorflow_version` only switches the major version: 1.x or 2.x.\n",
            "You set: `2.x  # this line is not required unless you are in a notebook`. This will be interpreted as: `2.x`.\n",
            "\n",
            "\n",
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C_0SBOVaFvDt",
        "outputId": "83b83ed8-77db-4116-84f0-fe29c8ddaf1f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#data import\n",
        "path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n",
            "1122304/1115394 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q05e935fF5WD"
      },
      "source": [
        "#we can import out own dataset with.txt format\n",
        "#from google.colab import files\n",
        "#path_to_file = list(files.upload().keys())[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ndhNrxVXGIZt",
        "outputId": "8e870fe0-4b0d-4201-da8d-a575bd8ef1c6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Read, then decode for py2 compat.\n",
        "text = open(path_to_file, 'rb').read().decode(encoding='utf-8')\n",
        "# length of text is the number of characters in it\n",
        "print ('Length of text: {} characters'.format(len(text)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Length of text: 1115394 characters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l4HZDsUvGxhR"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ynY3I5OXG8Oi"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7OgpoqJKG8R9"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DisDwXM4G8Xb",
        "outputId": "36ec78ef-26bc-41fa-cd17-679d727d5030",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#now we will proceed with the romeo juliet file \n",
        "# Read, then decode for py2 compat.\n",
        "text = open(path_to_file, 'rb').read().decode(encoding='utf-8')\n",
        "# length of text is the number of characters in it\n",
        "print ('Length of text: {} characters'.format(len(text)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Length of text: 1115394 characters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qRX2AZZFG8d7",
        "outputId": "6c2c4b01-fb30-4bfd-f8ac-31f566611f52",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Take a look at the first 250 characters in text\n",
        "print(text[:250])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First Citizen:\n",
            "Before we proceed any further, hear me speak.\n",
            "\n",
            "All:\n",
            "Speak, speak.\n",
            "\n",
            "First Citizen:\n",
            "You are all resolved rather to die than to famish?\n",
            "\n",
            "All:\n",
            "Resolved. resolved.\n",
            "\n",
            "First Citizen:\n",
            "First, you know Caius Marcius is chief enemy to the people.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EfoPhg_mG8bx"
      },
      "source": [
        "#Encoding\n",
        "#convert it into integer. All the characters.\n",
        "\n",
        "vocab = sorted(set(text))  #to sort all the unique characters\n",
        "# Creating a mapping from unique characters to indices\n",
        "char2idx = {u:i for i, u in enumerate(vocab)}  #mapping\n",
        "idx2char = np.array(vocab) #making list or array to use the indexes\n",
        "\n",
        "def text_to_int(text):\n",
        "  return np.array([char2idx[c] for c in text]) #convert every single characters from the text into their integer representation \n",
        "\n",
        "text_as_int = text_to_int(text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-F9A_i3OG8Vg",
        "outputId": "a3c52a9a-0db2-404e-9dbb-350443907c24",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# lets look at how part of our text is encoded\n",
        "print(\"Text:\", text[:13])\n",
        "print(\"Encoded:\", text_to_int(text[:13]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Text: First Citizen\n",
            "Encoded: [18 47 56 57 58  1 15 47 58 47 64 43 52]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "paGtZ6ubIbhP",
        "outputId": "d72bc619-6611-4f7d-f57a-89d08f5916a0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def int_to_text(ints):\n",
        "  try:\n",
        "    ints = ints.numpy() #if its not already a numpy array, it will make it in numpy array\n",
        "  except:\n",
        "    pass\n",
        "  return ''.join(idx2char[ints])\n",
        "\n",
        "print(int_to_text(text_as_int[:133]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First Citizen:\n",
            "Before we proceed any further, hear me speak.\n",
            "\n",
            "All:\n",
            "Speak, speak.\n",
            "\n",
            "First Citizen:\n",
            "You are all resolved rather to die t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G19Jy64XItW1"
      },
      "source": [
        "#training\n",
        "#whatever input we have, we will train in such a way that the output is one character ahead\n",
        "#for example We will Input Cana the O/P will be anad\n",
        "seq_length = 100  # length of sequence for a training example\n",
        "examples_per_epoch = len(text)//(seq_length+1)\n",
        "\n",
        "# Create training examples / targets\n",
        "char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int) #converts string to characters"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fz0D50Y5JSdd"
      },
      "source": [
        "sequences = char_dataset.batch(seq_length+1, drop_remainder=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yuAC-BpKJcJp"
      },
      "source": [
        "#split input target\n",
        "def split_input_target(chunk):  # for the example: hello\n",
        "    input_text = chunk[:-1]  # hell\n",
        "    target_text = chunk[1:]  # ello\n",
        "    return input_text, target_text  # hell, ello\n",
        "\n",
        "dataset = sequences.map(split_input_target)  # we use map to apply the above function to every entry"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fq4XqOpgJx_d",
        "outputId": "e027f333-b493-4427-feb2-74669cbd99cc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "for x, y in dataset.take(2):\n",
        "  print(\"\\n\\nEXAMPLE\\n\")\n",
        "  print(\"INPUT\")\n",
        "  print(int_to_text(x))\n",
        "  print(\"\\nOUTPUT\")\n",
        "  print(int_to_text(y))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "EXAMPLE\n",
            "\n",
            "INPUT\n",
            "First Citizen:\n",
            "Before we proceed any further, hear me speak.\n",
            "\n",
            "All:\n",
            "Speak, speak.\n",
            "\n",
            "First Citizen:\n",
            "You\n",
            "\n",
            "OUTPUT\n",
            "irst Citizen:\n",
            "Before we proceed any further, hear me speak.\n",
            "\n",
            "All:\n",
            "Speak, speak.\n",
            "\n",
            "First Citizen:\n",
            "You \n",
            "\n",
            "\n",
            "EXAMPLE\n",
            "\n",
            "INPUT\n",
            "are all resolved rather to die than to famish?\n",
            "\n",
            "All:\n",
            "Resolved. resolved.\n",
            "\n",
            "First Citizen:\n",
            "First, you \n",
            "\n",
            "OUTPUT\n",
            "re all resolved rather to die than to famish?\n",
            "\n",
            "All:\n",
            "Resolved. resolved.\n",
            "\n",
            "First Citizen:\n",
            "First, you k\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEbcQEeWJ0Zh"
      },
      "source": [
        "#Batch her means 64 different sequences of training examples\n",
        "BATCH_SIZE = 64\n",
        "VOCAB_SIZE = len(vocab)  # vocab is number of unique characters\n",
        "EMBEDDING_DIM = 256\n",
        "RNN_UNITS = 1024\n",
        "\n",
        "# Buffer size to shuffle the dataset\n",
        "# (TF data is designed to work with possibly infinite sequences,\n",
        "# so it doesn't attempt to shuffle the entire sequence in memory. Instead,\n",
        "# it maintains a buffer in which it shuffles elements).\n",
        "BUFFER_SIZE = 10000\n",
        "\n",
        "data = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ALJk7HvcKR1k"
      },
      "source": [
        "###Building the Model\n",
        "Now it is time to build the model. We will use an embedding layer a LSTM and one dense layer that contains a node for each unique character in our training data. The dense layer will give us a probability distribution over all nodes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CaZZGKhkKLv2",
        "outputId": "dea6186e-8fce-4a2e-c0ec-891d2c474a4a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
        "  model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim,\n",
        "                              batch_input_shape=[batch_size, None]),\n",
        "    tf.keras.layers.LSTM(rnn_units,\n",
        "                        return_sequences=True,\n",
        "                        stateful=True,\n",
        "                        recurrent_initializer='glorot_uniform'),\n",
        "    tf.keras.layers.Dense(vocab_size)\n",
        "  ])\n",
        "  return model\n",
        "\n",
        "model = build_model(VOCAB_SIZE,EMBEDDING_DIM, RNN_UNITS, BATCH_SIZE)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (64, None, 256)           16640     \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (64, None, 1024)          5246976   \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (64, None, 65)            66625     \n",
            "=================================================================\n",
            "Total params: 5,330,241\n",
            "Trainable params: 5,330,241\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xObhhFEVKbuf"
      },
      "source": [
        "#creating a loss function"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PtgdyK8aKcBn",
        "outputId": "780d2306-a63a-4407-d9de-1fe7320d723e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "for input_example_batch, target_example_batch in data.take(1):\n",
        "  example_batch_predictions = model(input_example_batch)  # ask our model for a prediction on our first batch of training data (64 entries)\n",
        "  print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")  # print out the output shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(64, 100, 65) # (batch_size, sequence_length, vocab_size)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xlXiy9dRKcIB",
        "outputId": "10e3a000-5e9a-4a71-ef19-7f0fb866531f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# we can see that the predicition is an array of 64 arrays, one for each entry in the batch\n",
        "print(len(example_batch_predictions))\n",
        "print(example_batch_predictions)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "64\n",
            "tf.Tensor(\n",
            "[[[ 6.7274657e-04 -5.0319202e-04 -4.6850252e-04 ... -1.3898171e-03\n",
            "    1.6326888e-04  1.9410336e-03]\n",
            "  [ 6.7354497e-03  2.2096853e-03 -4.2898953e-03 ... -1.4658133e-03\n",
            "    2.4623428e-03  1.5205378e-04]\n",
            "  [ 4.2199073e-03  1.4884276e-03 -2.6576980e-03 ...  6.0320632e-03\n",
            "   -2.5269757e-03  4.2872038e-03]\n",
            "  ...\n",
            "  [-1.4479773e-04 -1.7485372e-03  4.5530051e-03 ...  8.4786564e-03\n",
            "   -1.1058331e-03 -1.2404581e-03]\n",
            "  [-1.0719297e-03 -4.7317124e-04  3.3642091e-03 ...  4.4608871e-03\n",
            "    4.3269736e-04  1.2044776e-03]\n",
            "  [-5.6531129e-04  2.7402239e-03  5.4865382e-03 ... -2.0118044e-03\n",
            "   -2.9888859e-03  1.5888268e-03]]\n",
            "\n",
            " [[ 6.7274657e-04 -5.0319202e-04 -4.6850252e-04 ... -1.3898171e-03\n",
            "    1.6326888e-04  1.9410336e-03]\n",
            "  [-4.0302812e-03 -1.2220307e-03 -2.2942881e-04 ... -5.2367169e-03\n",
            "   -1.9886238e-03  3.7209084e-03]\n",
            "  [ 4.6400935e-03  2.2028410e-03 -4.1117007e-03 ... -3.6126545e-03\n",
            "    6.3564366e-04  1.7407113e-03]\n",
            "  ...\n",
            "  [-5.9013148e-03  1.7260138e-03  3.3845860e-03 ...  9.8720584e-03\n",
            "   -3.3423025e-04  3.1131930e-03]\n",
            "  [-2.6627169e-03  9.9419546e-04 -5.1122652e-03 ...  7.8887893e-03\n",
            "    8.4253302e-04  4.4485868e-04]\n",
            "  [-1.0383311e-03  1.7397967e-03  3.0403722e-03 ...  1.4276191e-03\n",
            "    9.8027755e-04 -1.5073584e-04]]\n",
            "\n",
            " [[ 6.4463411e-03  1.9398467e-03 -4.4970121e-03 ...  4.8894435e-04\n",
            "    2.0223020e-03 -1.2951341e-03]\n",
            "  [ 7.4671241e-03  7.3353825e-03 -4.3669478e-03 ... -3.4282189e-03\n",
            "   -5.1471270e-03  1.5148974e-04]\n",
            "  [ 1.1692035e-02  7.7528846e-03 -8.1040626e-03 ... -1.9631255e-03\n",
            "   -1.8598074e-03 -2.1803558e-03]\n",
            "  ...\n",
            "  [ 3.2924970e-03  3.6256784e-03 -2.9592535e-03 ...  3.9308043e-03\n",
            "   -1.4547848e-03 -5.1599834e-04]\n",
            "  [ 7.0950263e-03  5.7291635e-03 -6.7681745e-03 ...  2.6805417e-03\n",
            "    4.1592855e-04 -1.6622656e-03]\n",
            "  [ 5.6431126e-03  8.6303428e-03 -1.6458600e-04 ... -5.4837978e-03\n",
            "    1.1907516e-03 -1.2238019e-03]]\n",
            "\n",
            " ...\n",
            "\n",
            " [[-6.8437651e-04 -1.4457197e-03  5.1118468e-04 ...  7.7209570e-03\n",
            "   -4.5454628e-03  4.5525078e-03]\n",
            "  [ 3.7283075e-04 -2.4249111e-03  2.6179894e-03 ...  8.3086658e-03\n",
            "   -5.5344878e-03  2.7096081e-03]\n",
            "  [-6.8837823e-04  3.1111829e-03 -2.7281379e-03 ...  9.8819593e-03\n",
            "   -1.1657011e-03 -1.2634348e-03]\n",
            "  ...\n",
            "  [-6.8851653e-03 -7.0136422e-03 -5.6490991e-03 ...  1.0574711e-02\n",
            "    1.5445473e-03 -8.7481597e-04]\n",
            "  [-6.6131069e-03 -1.8761954e-03 -4.9785953e-03 ...  5.7332157e-03\n",
            "   -2.4698465e-03 -1.3906823e-03]\n",
            "  [-1.1807761e-02  8.0897659e-04  1.2045365e-03 ...  9.7773364e-03\n",
            "   -7.7256868e-03 -4.6572248e-03]]\n",
            "\n",
            " [[ 6.7274657e-04 -5.0319202e-04 -4.6850252e-04 ... -1.3898171e-03\n",
            "    1.6326888e-04  1.9410336e-03]\n",
            "  [ 2.8833367e-03 -9.4952114e-04  7.1741100e-03 ... -5.4110647e-03\n",
            "    5.1677949e-04  5.1342067e-04]\n",
            "  [-9.7979931e-04  2.5779114e-03  6.1880001e-03 ... -6.4053857e-03\n",
            "    9.2998543e-04  1.0702977e-02]\n",
            "  ...\n",
            "  [ 1.1995213e-02 -2.8461781e-03  8.1597026e-03 ... -2.7446757e-04\n",
            "    2.2828351e-03 -3.0412122e-03]\n",
            "  [ 9.6492181e-03 -2.9004912e-03  6.8118740e-03 ... -4.0147552e-04\n",
            "    1.0523443e-03  3.4695657e-04]\n",
            "  [ 1.0538653e-02 -3.9183991e-03  7.3607415e-03 ...  7.4029132e-04\n",
            "    8.5980596e-04 -1.2299914e-03]]\n",
            "\n",
            " [[-2.5435037e-03  2.9307932e-03  8.6627976e-04 ... -1.3613271e-03\n",
            "    1.0186870e-03  9.5734717e-03]\n",
            "  [-2.0067920e-03  2.0526648e-03  7.6279975e-06 ... -2.0026811e-03\n",
            "    1.1556688e-03  8.3746687e-03]\n",
            "  [ 5.3962753e-03  4.4085658e-03  3.4672935e-03 ... -1.8030353e-03\n",
            "    3.4323311e-03  4.6535595e-03]\n",
            "  ...\n",
            "  [-9.1307866e-04 -1.0420012e-03 -2.6764236e-03 ...  5.9874025e-03\n",
            "    3.4316191e-03 -8.2154293e-06]\n",
            "  [-1.9186221e-03 -2.3772577e-03  6.1500631e-03 ...  9.2547918e-03\n",
            "    2.9628500e-03  1.9452885e-03]\n",
            "  [-2.4194121e-03 -1.5396848e-03  3.7968678e-03 ...  5.5077430e-03\n",
            "    2.7556578e-03  4.8031607e-03]]], shape=(64, 100, 65), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SJK2z1ItKcdK",
        "outputId": "27f8f4b4-933a-43b5-9885-baef403fb2b5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# lets examine one prediction\n",
        "pred = example_batch_predictions[0]\n",
        "print(len(pred))\n",
        "print(pred)\n",
        "# notice this is a 2d array of length 100, where each interior array is the prediction for the next character at each time step"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100\n",
            "tf.Tensor(\n",
            "[[ 0.00067275 -0.00050319 -0.0004685  ... -0.00138982  0.00016327\n",
            "   0.00194103]\n",
            " [ 0.00673545  0.00220969 -0.0042899  ... -0.00146581  0.00246234\n",
            "   0.00015205]\n",
            " [ 0.00421991  0.00148843 -0.0026577  ...  0.00603206 -0.00252698\n",
            "   0.0042872 ]\n",
            " ...\n",
            " [-0.0001448  -0.00174854  0.00455301 ...  0.00847866 -0.00110583\n",
            "  -0.00124046]\n",
            " [-0.00107193 -0.00047317  0.00336421 ...  0.00446089  0.0004327\n",
            "   0.00120448]\n",
            " [-0.00056531  0.00274022  0.00548654 ... -0.0020118  -0.00298889\n",
            "   0.00158883]], shape=(100, 65), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bltwp4StKcgW",
        "outputId": "91dc1bcd-ed45-4856-a278-dc5d15ed4974",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# and finally well look at a prediction at the first timestep\n",
        "time_pred = pred[0]\n",
        "print(len(time_pred))\n",
        "print(time_pred)\n",
        "# and of course its 65 values representing the probabillity of each character occuring next"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "65\n",
            "tf.Tensor(\n",
            "[ 6.7274657e-04 -5.0319202e-04 -4.6850252e-04 -9.0439426e-05\n",
            "  2.2186176e-03 -4.2153057e-03  2.5387893e-03 -1.5957992e-03\n",
            " -4.2079929e-03 -5.8606951e-03 -3.5182529e-03  3.4699186e-03\n",
            "  1.8370716e-03  1.6031422e-03  8.6772554e-03 -1.7934158e-03\n",
            "  2.3158290e-04  2.9231776e-03  3.7459540e-03  1.4502495e-03\n",
            "  2.3439331e-03  1.2771345e-03  4.0824446e-03  4.1545639e-03\n",
            "  1.5291943e-03  2.2731654e-03 -1.7526307e-03 -6.4960530e-04\n",
            " -1.5813748e-03 -2.5857263e-03  2.0201434e-04 -2.9485051e-03\n",
            "  4.3277862e-03 -1.0354127e-03 -4.2676581e-03  5.6235085e-04\n",
            "  2.7679717e-03  4.8067174e-03 -4.0163812e-03  6.0905153e-03\n",
            " -1.7487206e-03  1.3212223e-03  4.7131398e-06 -8.3247357e-04\n",
            " -1.4839387e-03  2.6353826e-03  2.3818181e-03  6.0473019e-03\n",
            " -2.9647106e-03  3.0515757e-03  5.4815160e-03  4.6843477e-03\n",
            " -2.0907405e-03  9.0660609e-04 -3.4957391e-03  4.4471002e-04\n",
            "  4.2588376e-03  2.8613787e-03 -6.4568496e-03  6.3332240e-04\n",
            "  1.2470410e-03  4.2957452e-04 -1.3898171e-03  1.6326888e-04\n",
            "  1.9410336e-03], shape=(65,), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1anQ531YKcaX",
        "outputId": "0efdf884-3f16-432f-d750-bfc2d247e9f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# If we want to determine the predicted character we need to sample the output distribution (pick a value based on probabillity)\n",
        "sampled_indices = tf.random.categorical(pred, num_samples=1)\n",
        "\n",
        "# now we can reshape that array and convert all the integers to numbers to see the actual characters\n",
        "sampled_indices = np.reshape(sampled_indices, (1, -1))[0]\n",
        "predicted_chars = int_to_text(sampled_indices)\n",
        "\n",
        "predicted_chars  # and this is what the model predicted for training sequence 1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'dvL\\nseM&RMqXRj:$oPVWA J;JmolRO.kZ-wQC?IyQGZZALzLenpnExmOe:Xg.TcPNpfzvxUTP.jmUVgrwjr33HFYLpZOlHGEtOAR'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8981czIBKcE7"
      },
      "source": [
        "def loss(labels, logits):\n",
        "  return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnTQNcCkKb0T"
      },
      "source": [
        "#completing the model\n",
        "model.compile(optimizer='adam', loss=loss)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n0_dGSRkKby5"
      },
      "source": [
        "#creating checkpoints\n",
        "# Directory where the checkpoints will be saved\n",
        "checkpoint_dir = './training_checkpoints'\n",
        "# Name of the checkpoint files\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt_{epoch}\")\n",
        "\n",
        "checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_prefix,\n",
        "    save_weights_only=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OQC469nwKbhR",
        "outputId": "53f13447-6390-49c7-852f-3188028c16e5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#Training\n",
        "history = model.fit(data, epochs=50, callbacks=[checkpoint_callback])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "172/172 [==============================] - 12s 67ms/step - loss: 2.5974\n",
            "Epoch 2/50\n",
            "172/172 [==============================] - 12s 68ms/step - loss: 1.8904\n",
            "Epoch 3/50\n",
            "172/172 [==============================] - 12s 69ms/step - loss: 1.6414\n",
            "Epoch 4/50\n",
            "172/172 [==============================] - 12s 69ms/step - loss: 1.5052\n",
            "Epoch 5/50\n",
            "172/172 [==============================] - 12s 70ms/step - loss: 1.4233\n",
            "Epoch 6/50\n",
            "172/172 [==============================] - 12s 71ms/step - loss: 1.3658\n",
            "Epoch 7/50\n",
            "172/172 [==============================] - 12s 72ms/step - loss: 1.3199\n",
            "Epoch 8/50\n",
            "172/172 [==============================] - 12s 72ms/step - loss: 1.2808\n",
            "Epoch 9/50\n",
            "172/172 [==============================] - 12s 72ms/step - loss: 1.2437\n",
            "Epoch 10/50\n",
            "172/172 [==============================] - 13s 73ms/step - loss: 1.2079\n",
            "Epoch 11/50\n",
            "172/172 [==============================] - 13s 73ms/step - loss: 1.1713\n",
            "Epoch 12/50\n",
            "172/172 [==============================] - 13s 74ms/step - loss: 1.1357\n",
            "Epoch 13/50\n",
            "172/172 [==============================] - 13s 74ms/step - loss: 1.0970\n",
            "Epoch 14/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 1.0570\n",
            "Epoch 15/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 1.0167\n",
            "Epoch 16/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.9735\n",
            "Epoch 17/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.9322\n",
            "Epoch 18/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.8910\n",
            "Epoch 19/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.8505\n",
            "Epoch 20/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.8120\n",
            "Epoch 21/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.7764\n",
            "Epoch 22/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.7410\n",
            "Epoch 23/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.7104\n",
            "Epoch 24/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.6808\n",
            "Epoch 25/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.6546\n",
            "Epoch 26/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.6306\n",
            "Epoch 27/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.6093\n",
            "Epoch 28/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.5904\n",
            "Epoch 29/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.5727\n",
            "Epoch 30/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.5566\n",
            "Epoch 31/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.5426\n",
            "Epoch 32/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.5290\n",
            "Epoch 33/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.5167\n",
            "Epoch 34/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.5065\n",
            "Epoch 35/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.4993\n",
            "Epoch 36/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.4889\n",
            "Epoch 37/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4818\n",
            "Epoch 38/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4754\n",
            "Epoch 39/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4677\n",
            "Epoch 40/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4623\n",
            "Epoch 41/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4562\n",
            "Epoch 42/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4510\n",
            "Epoch 43/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4459\n",
            "Epoch 44/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4436\n",
            "Epoch 45/50\n",
            "172/172 [==============================] - 13s 75ms/step - loss: 0.4393\n",
            "Epoch 46/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4359\n",
            "Epoch 47/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4340\n",
            "Epoch 48/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4316\n",
            "Epoch 49/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4255\n",
            "Epoch 50/50\n",
            "172/172 [==============================] - 13s 76ms/step - loss: 0.4247\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RES0HKucK5k4"
      },
      "source": [
        "#loading the model\n",
        "model = build_model(VOCAB_SIZE, EMBEDDING_DIM, RNN_UNITS, batch_size=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zsp4FwQlK5-x"
      },
      "source": [
        "model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n",
        "model.build(tf.TensorShape([1, None]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qu41Dg6SK6Eo"
      },
      "source": [
        "#We can load any checkpoint we want by specifying the exact file to load"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ovWNET3bK6Ca"
      },
      "source": [
        "checkpoint_num = 10\n",
        "model.load_weights(tf.train.load_checkpoint(\"./training_checkpoints/ckpt_\" + str(checkpoint_num)))\n",
        "model.build(tf.TensorShape([1, None]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IsNiVG83LFGP"
      },
      "source": [
        "Generating text"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KanKBQ7PK57f"
      },
      "source": [
        "def generate_text(model, start_string):\n",
        "  # Evaluation step (generating text using the learned model)\n",
        "\n",
        "  # Number of characters to generate\n",
        "  num_generate = 800\n",
        "\n",
        "  # Converting our start string to numbers (vectorizing)\n",
        "  input_eval = [char2idx[s] for s in start_string]\n",
        "  input_eval = tf.expand_dims(input_eval, 0)\n",
        "\n",
        "  # Empty string to store our results\n",
        "  text_generated = []\n",
        "\n",
        "  # Low temperatures results in more predictable text.\n",
        "  # Higher temperatures results in more surprising text.\n",
        "  # Experiment to find the best setting.\n",
        "  temperature = 1.0\n",
        "\n",
        "  # Here batch size == 1\n",
        "  model.reset_states()\n",
        "  for i in range(num_generate):\n",
        "      predictions = model(input_eval)\n",
        "      # remove the batch dimension\n",
        "    \n",
        "      predictions = tf.squeeze(predictions, 0)\n",
        "\n",
        "      # using a categorical distribution to predict the character returned by the model\n",
        "      predictions = predictions / temperature\n",
        "      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
        "\n",
        "      # We pass the predicted character as the next input to the model\n",
        "      # along with the previous hidden state\n",
        "      input_eval = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "      text_generated.append(idx2char[predicted_id])\n",
        "\n",
        "  return (start_string + ''.join(text_generated))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A969vq3bK5h_",
        "outputId": "27e3e6d3-798b-475c-e200-e82fcb22f4b2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "inp = input(\"Type a starting string: \")\n",
        "print(generate_text(model, inp))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Type a starting string: juliet\n",
            "julietely pardon lead against the tist: so, for thee thy bed,\n",
            "And bring in cloudy blood of Montague.\n",
            "Consider your friends at enough.\n",
            "\n",
            "ANGELO:\n",
            "Meantime, nay; the yice and old Derby,\n",
            "Stand for me! O, the state should kiss on him.\n",
            "\n",
            "ISABELLA:\n",
            "Well, passion, house.\n",
            "\n",
            "FRIAR LAURENA:\n",
            "And live you lent him go this fashion to death,\n",
            "I will not be allow'd to make King Richard lew him.\n",
            "\n",
            "DUKE OF YORK:\n",
            "Give me my boots. If I pires your worship did beat down;\n",
            "It shall go warrant him with colours;\n",
            "And where it feather brave beneft an emb.\n",
            "I have been canched nine my will were so;\n",
            "And thus I'll give my stranger eyes, or else\n",
            "Before the house of Lancaster and his.\n",
            "\n",
            "FLORIZEL:\n",
            "Pardon me, Kate, Hermione,\n",
            "I mind with intere be intent of love tell her my servant Catesby\n",
            "Where nothing can proceed that toucheth us both\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Eyz2GSNsVEgf"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}