PPLM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PPLM.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SAswinGiridhar/Tried_jupyter_notebooks/blob/master/PPLM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFmN0zIBXAuQ",
        "colab_type": "text"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W5UCitCoVLZC",
        "colab_type": "code",
        "outputId": "7535cdcf-cf91-4f66-b677-feee2a101648",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "source": [
        "!git clone https://github.com/uber-research/PPLM.git"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'PPLM'...\n",
            "remote: Enumerating objects: 77, done.\u001b[K\n",
            "remote: Counting objects: 100% (77/77), done.\u001b[K\n",
            "remote: Compressing objects: 100% (52/52), done.\u001b[K\n",
            "remote: Total 77 (delta 38), reused 59 (delta 22), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (77/77), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SMxM3cm1X2Pj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "os.chdir('PPLM')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dQ9z0QkTWt1V",
        "colab_type": "code",
        "outputId": "d1f41e6c-4608-4bdb-e756-ae8af0b88977",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 728
        }
      },
      "source": [
        "!pip install -r requirements.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1)) (1.3.1)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 2)) (3.2.5)\n",
            "Collecting colorama\n",
            "  Downloading https://files.pythonhosted.org/packages/4f/a6/728666f39bfff1719fc94c481890b2106837da9318031f71a8424b662e12/colorama-0.4.1-py2.py3-none-any.whl\n",
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/70/1a/364556102943cacde1ee00fdcae3b1615b39e52649eddbf54953e5b144c9/transformers-2.2.1-py3-none-any.whl (364kB)\n",
            "\u001b[K     |████████████████████████████████| 368kB 4.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 5)) (0.3.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch->-r requirements.txt (line 1)) (1.17.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk->-r requirements.txt (line 2)) (1.12.0)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 4)) (1.10.27)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 4)) (2.21.0)\n",
            "Collecting sentencepiece\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/3d/efb655a670b98f62ec32d66954e1109f403db4d937c50d779a75b9763a29/sentencepiece-0.1.83-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)\n",
            "\u001b[K     |████████████████████████████████| 1.0MB 43.3MB/s \n",
            "\u001b[?25hCollecting regex\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/8e/cbf2295643d7265e7883326fb4654e643bfc93b3a8a8274d8010a39d8804/regex-2019.11.1-cp36-cp36m-manylinux1_x86_64.whl (643kB)\n",
            "\u001b[K     |████████████████████████████████| 645kB 40.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 4)) (4.28.1)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1f/8e/ed5364a06a9ba720fddd9820155cc57300d28f5f43a6fd7b7e817177e642/sacremoses-0.0.35.tar.gz (859kB)\n",
            "\u001b[K     |████████████████████████████████| 860kB 13.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers->-r requirements.txt (line 4)) (0.2.1)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers->-r requirements.txt (line 4)) (0.9.4)\n",
            "Requirement already satisfied: botocore<1.14.0,>=1.13.27 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers->-r requirements.txt (line 4)) (1.13.27)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 4)) (2019.9.11)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 4)) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 4)) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 4)) (2.8)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 4)) (7.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 4)) (0.14.0)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.27->boto3->transformers->-r requirements.txt (line 4)) (0.15.2)\n",
            "Requirement already satisfied: python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.27->boto3->transformers->-r requirements.txt (line 4)) (2.6.1)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.35-cp36-none-any.whl size=883999 sha256=b749467d6f310a5cffbbed35df18de96c7483632457df6f44cf8a4f246cdf4a1\n",
            "  Stored in directory: /root/.cache/pip/wheels/63/2a/db/63e2909042c634ef551d0d9ac825b2b0b32dede4a6d87ddc94\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: colorama, sentencepiece, regex, sacremoses, transformers\n",
            "Successfully installed colorama-0.4.1 regex-2019.11.1 sacremoses-0.0.35 sentencepiece-0.1.83 transformers-2.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yM43ataHq2rJ",
        "colab_type": "code",
        "outputId": "ba1b34b2-7542-47d2-bf1a-e21c5a592a82",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 100
        }
      },
      "source": [
        "from transformers.modeling_gpt2 import GPT2LMHeadModel\n",
        "# This downloads GPT-2 Medium, it takes a little while\n",
        "_ = GPT2LMHeadModel.from_pretrained(\"gpt2-medium\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 293/293 [00:00<00:00, 113884.82B/s]\n",
            "100%|██████████| 1520013706/1520013706 [00:31<00:00, 47760550.51B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DfNwx_F_YZXE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from run_pplm import run_pplm_example"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5Yl_u6BYnS5",
        "colab_type": "text"
      },
      "source": [
        "# Let's generate some text!\n",
        "When you specify a number of samples, PPLM will first generate a sample without any modification for reference (called \"unperturbed\"), and then the number of samples specified (called \"perturbed\"). If you want to generate different samples given the same parameters, change the `seed` value to a number other than 0 (the default value)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zWbu3hPoYrVR",
        "colab_type": "code",
        "outputId": "e2249851-0f4b-4725-c78e-efc6ebc1d1ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "run_pplm_example(\n",
        "    cond_text=\"The potato\",\n",
        "    num_samples=3,\n",
        "    bag_of_words='military',\n",
        "    length=50,\n",
        "    stepsize=0.03,\n",
        "    sample=True,\n",
        "    num_iterations=3,\n",
        "    window_length=5,\n",
        "    gamma=1.5,\n",
        "    gm_scale=0.95,\n",
        "    kl_scale=0.01,\n",
        "    verbosity='regular'\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 1042301/1042301 [00:00<00:00, 5873467.06B/s]\n",
            "100%|██████████| 456318/456318 [00:00<00:00, 3660605.13B/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "= Prefix of sentence =\n",
            "<|endoftext|>The potato\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 1896/1896 [00:00<00:00, 680856.20B/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Using PPLM-BoW\n",
            "<|endoftext|>The potato is\n",
            "<|endoftext|>The potato is probably\n",
            "<|endoftext|>The potato is probably the\n",
            "<|endoftext|>The potato is probably the world\n",
            "<|endoftext|>The potato is probably the world's\n",
            "<|endoftext|>The potato is probably the world's most\n",
            "<|endoftext|>The potato is probably the world's most widely\n",
            "<|endoftext|>The potato is probably the world's most widely eaten\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant.\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades,\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease.\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline,\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline, which\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline, which started\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline, which started in\n",
            "<|endoftext|>The potato-\n",
            "<|endoftext|>The potato-like\n",
            "<|endoftext|>The potato-like,\n",
            "<|endoftext|>The potato-like, potato\n",
            "<|endoftext|>The potato-like, potato-\n",
            "<|endoftext|>The potato-like, potato-sized\n",
            "<|endoftext|>The potato-like, potato-sized thing\n",
            "<|endoftext|>The potato-like, potato-sized thing is\n",
            "<|endoftext|>The potato-like, potato-sized thing is the\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military,\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle.\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle.\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle.\n",
            "\n",
            "\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle.\n",
            "\n",
            "It\n",
            "<|endoftext|>The potato bomb\n",
            "<|endoftext|>The potato bomb and\n",
            "<|endoftext|>The potato bomb and the\n",
            "<|endoftext|>The potato bomb and the anti\n",
            "<|endoftext|>The potato bomb and the anti-\n",
            "<|endoftext|>The potato bomb and the anti-Semitic\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"un\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\"\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism,\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will not\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will not tolerate\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will not tolerate hate\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will not tolerate hate,\"\n",
            "<|endoftext|>The potato chip\n",
            "<|endoftext|>The potato chip cra\n",
            "<|endoftext|>The potato chip craze\n",
            "<|endoftext|>The potato chip craze is\n",
            "<|endoftext|>The potato chip craze is in\n",
            "<|endoftext|>The potato chip craze is in full\n",
            "<|endoftext|>The potato chip craze is in full swing\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years,\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream.\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips cra\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze,\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands who\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands who haven\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands who haven't\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands who haven't.\n",
            "================================================================================\n",
            "= Unperturbed generated text =\n",
            "<|endoftext|>The potato is probably the world's most widely eaten plant. But what if it's also the most dangerous?\n",
            "\n",
            "In the last two decades, there's been a dramatic decrease in potato crop damage from crop rot and disease. The decline, which started in\n",
            "\n",
            "= Perturbed generated text 1 =\n",
            "<|endoftext|>The potato-like, potato-sized thing is the most popular of all the things we do in the military, especially if you're a soldier in the field.\n",
            "\n",
            "It's an assault weapon.\n",
            "\n",
            "It's an armored vehicle.\n",
            "\n",
            "It\n",
            "\n",
            "= Perturbed generated text 2 =\n",
            "<|endoftext|>The potato bomb and the anti-Semitic attack that killed four Jewish students at a Jewish school in France are \"unacceptable\" but have no connection with terrorism, Prime Minister Justin Trudeau told a news conference on Wednesday.\n",
            "\n",
            "\"We will not tolerate hate,\"\n",
            "\n",
            "= Perturbed generated text 3 =\n",
            "<|endoftext|>The potato chip craze is in full swing.\n",
            "\n",
            "The popular snacks have been around for years, but now they're hitting the mainstream. And while some brands have tried to capitalize on the potato chips craze, there are many brands who haven't.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "stmBEfq6Zler",
        "colab_type": "code",
        "outputId": "2e85bf24-7f3b-4cd3-c9af-5f0fb835006b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "run_pplm_example(\n",
        "    cond_text=\"My dog died\",\n",
        "    num_samples=3,\n",
        "    discrim='sentiment',\n",
        "    class_label='very_positive',\n",
        "    length=50,\n",
        "    stepsize=0.05,\n",
        "    sample=True,\n",
        "    num_iterations=10,\n",
        "    gamma=1,\n",
        "    gm_scale=0.9,\n",
        "    kl_scale=0.02,\n",
        "    verbosity='regular'\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "= Prefix of sentence =\n",
            "<|endoftext|>My dog died\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 21041/21041 [00:00<00:00, 791991.01B/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Using PPLM-Discrim\n",
            "<|endoftext|>My dog died in\n",
            "<|endoftext|>My dog died in February\n",
            "<|endoftext|>My dog died in February,\n",
            "<|endoftext|>My dog died in February, after\n",
            "<|endoftext|>My dog died in February, after suffering\n",
            "<|endoftext|>My dog died in February, after suffering from\n",
            "<|endoftext|>My dog died in February, after suffering from severe\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis.\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break.\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet.\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew the\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew the vet\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew the vet would\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew the vet would be\n",
            "<|endoftext|>My dog died.\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweet\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart.\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would.\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would.\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older,\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older, he\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older, he's\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older, he's become\n",
            "<|endoftext|>My dog died from\n",
            "<|endoftext|>My dog died from a\n",
            "<|endoftext|>My dog died from a rare\n",
            "<|endoftext|>My dog died from a rare form\n",
            "<|endoftext|>My dog died from a rare form of\n",
            "<|endoftext|>My dog died from a rare form of cancer\n",
            "<|endoftext|>My dog died from a rare form of cancer,\n",
            "<|endoftext|>My dog died from a rare form of cancer, which\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line,\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown.\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment.\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment. This\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment. This new\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment. This new treatment\n",
            "<|endoftext|>My dog died after\n",
            "<|endoftext|>My dog died after getting\n",
            "<|endoftext|>My dog died after getting stuck\n",
            "<|endoftext|>My dog died after getting stuck in\n",
            "<|endoftext|>My dog died after getting stuck in a\n",
            "<|endoftext|>My dog died after getting stuck in a tree\n",
            "<|endoftext|>My dog died after getting stuck in a tree.\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down.\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused power loss\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused power loss power\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused power loss power down\n",
            "================================================================================\n",
            "= Unperturbed generated text =\n",
            "<|endoftext|>My dog died in February, after suffering from severe arthritis. He had been suffering with a terrible cold that was causing his skin to break. I couldn't afford a replacement dog and couldn't afford to have him taken to the vet. I knew the vet would be\n",
            "\n",
            "= Perturbed generated text 1 =\n",
            "<|endoftext|>My dog died.\n",
            "\n",
            "He was a beautiful, loving, wonderful, intelligent, sweetheart. He did everything I hoped he would. I knew he would. It's been over a year since I got him, he's just gotten older, he's become\n",
            "\n",
            "= Perturbed generated text 2 =\n",
            "<|endoftext|>My dog died from a rare form of cancer, which is now being treated more effectively with a new cancer cell line, the new treatment has shown. This was the first time that a cancer was found to be successfully cured using a new cancer treatment. This new treatment\n",
            "\n",
            "= Perturbed generated text 3 =\n",
            "<|endoftext|>My dog died after getting stuck in a tree. I have had the same bug with a man in power and that power power had power on power go into power go power down power down power down power power down power power down. This has caused power loss power down\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}