CA3_khalafi_810199414.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WbhkscB78r0U"
      },
      "source": [
        "## CA 3, LLMs Spring 2024\n",
        "\n",
        "- **Name: Bardia Khalafi**\n",
        "- **Student ID: 810199414**\n",
        "\n",
        "---\n",
        "### This is due on **May 11th, 2024**, submitted via [elearn](https://elearn.ut.ac.ir/).\n",
        "#### Your submission should be named using the following format: `CA3_LASTNAME_STUDENTID.ipynb`.\n",
        "\n",
        "---\n",
        "\n",
        "##### *How to do this problem set:*\n",
        "\n",
        "- Some questions require writing Python code and computing results, and the rest of them have written answers. For coding problems, you will have to fill out all code blocks that say `WRITE YOUR CODE HERE`.\n",
        "\n",
        "- For text-based answers, you should replace the text that says \"Write your answer here...\" with your actual answer.\n",
        "\n",
        "- There is no penalty for using AI assistance on this homework as long as you fully disclose it in the final cell of this notebook (this includes storing any prompts that you feed to large language models). That said, anyone caught using AI assistance without proper disclosure will receive a zero on the assignment (we have several automatic tools to detect such cases). We're literally allowing you to use it with no limitations, so there is no reason to lie!\n",
        "\n",
        "---\n",
        "\n",
        "##### *Academic honesty*\n",
        "\n",
        "- We will audit the Colab notebooks from a set number of students, chosen at random. The audits will check that the code you wrote actually generates the answers in your notebook. If you turn in correct answers on your notebook without code that actually generates those answers, we will consider this a serious case of cheating.\n",
        "\n",
        "- We will also run automatic checks of Colab notebooks for plagiarism. Copying code from others is also considered a serious case of cheating.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yxUrkg0u8r0W"
      },
      "source": [
        "# Chain-of-Thought (CoT) (20 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZOGsTSi_x0rP"
      },
      "source": [
        "If you have any further questions or concerns, contact the TA via email: mehdimohajeri@ut.ac.ir"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r26yOcE98r0X"
      },
      "source": [
        "LLMs have demonstrated good reasoning abilities. Furthermore, their capabilities can be further improved by incorporating reasoning techniques. One of the most notable developments in this area is the [Chain-of-Thought (CoT)](https://arxiv.org/abs/2201.11903), which was introduced by Google. This approach has shown promising results in improving the reasoning capabilities of language models across a variety of tasks. Can you explain what CoT is and how it works? (2.5 Points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "el05JZDn8r0X"
      },
      "source": [
        "its a concept to give the model chain of ideas to how it should generate text ( or solve a problem in a high level view )\n",
        "\n",
        "and this chain of ideas usually is given to model through example ( shots ) or steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hC13Hsv68r0X"
      },
      "source": [
        "In this section, you should use the CoT technique. firstly you need to load the [Phi-2 model](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/). This model has been introduced by Microsoft as a small LLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZZ1v6FHF8r0Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 580,
          "referenced_widgets": [
            "d9197efa50c0409badd62b208955da99",
            "487b57afcff846408329bc223af4af54",
            "0c2cc1ac6e2e447fb289bc674e8991d0",
            "83f741b12891401b86b154768ed0764c",
            "7d07b1a185fd4193ad9a587d58372725",
            "6bdac027f1aa46198a2026cc1fbf5e66",
            "463b315fe84b4e418a33d389c9027d3f",
            "12f8d488491e4ba3bc09ad2ea2123839",
            "cadbfc9dbb7945359fe88e9107143113",
            "bbc0142a866946199dc4d5c65bc2c76a",
            "c131a5284fb244649d139277fbaaee14",
            "206d22fc95f14cd183845940599fd309",
            "5d02ac7499b04de38483987896babf9d",
            "9b47bd13a77840f180c86e02a8567661",
            "fdf5e1cecbed4fe2bb9f81e18cde5a20",
            "71898a02d09f4b6788a17bf68e3803b3",
            "915dc0fb02644206aa5f6b2d794ffd63",
            "f02d864bf71144fc8f6c7af23cb03d04",
            "73718607c5e04b419dfbabb89c53650e",
            "fd3c8edacd094dff993a3985e543d2d9",
            "e40ba557258b40fd94a9569a4fc2fe64",
            "6cc9fffaae7f43e7a24dd02128de8adb",
            "2f4c3b4360fe4b40aec75639ef04b0ba",
            "bd4e39a7cb3840278e46a57f58ad549a",
            "319f26fc6d974ecab4c5181cdd518db8",
            "551a1787efc64e2a965d1ec2b7b6a33e",
            "2ed2e8eeda0d4a2984aa946dae2a51f9",
            "99f9ad06f6204cac9cc56cbfd0f60e0a",
            "ecfc5471abb84f20b07a24a8bc7fd5d8",
            "e6902997ac014b2ca23c9bca6930c1df",
            "d0d8ba2714754022aa5c64c3a74679ea",
            "ccc2c822359a4daf8170c9f0eb4f43bb",
            "5c74fdf30b764487aa0c1fdee4f4a7d8",
            "a7bee5212626426eac81918ee2822c76",
            "3391229bd6a04425bac90c64bac9ea5c",
            "b3391543994740e0920d5a05e55158c8",
            "bb8da3b5a17f41b39218b7a6fd4ba4ee",
            "1e23a34c29364d80ac315167492e15c5",
            "e26e8fb829114bfa9f639d8dc048764a",
            "99a7edbb40884ea19ebc806409c5368f",
            "f0471388d80e49deb2c00dd4b46982eb",
            "ec71c86bc8094ac193ebd22e8e8b52a0",
            "51eddce4bcd04bf2ac2255b64d0d068a",
            "6f08974e738a4977969eb8c5e5077378",
            "3ba5c4ff1d8047c4b77cc6d976f7ba2e",
            "924aca0af5fd421fbd64abae3963e858",
            "9dee6aafacf045f4add5a170765a29e8",
            "9f9440cfefd94619ae41a473741c8c3a",
            "33de55bda78142a4a5f9a1e132797836",
            "95fadf5577694edc9c655683f291d06b",
            "ca29b7d5749c4740b1879ed83e0024e7",
            "99119d0591eb49aeb261fcbac15d8489",
            "be50ebf187594348a3355cd96414a4f7",
            "bdc70d5af3a44a6dbfdf41ef63f86b5e",
            "b4dd617c8fb047c89378dcd0be5bfeef",
            "ef18c41010c644969f3cc2ff4cc78a04",
            "6c761c0dead6477faa7809abbc5dc8c7",
            "544a261fb49d4eb8bed84c3e15a3e17d",
            "cf66918d62d34a2ca21767c27d83c2c6",
            "b02f04e6801e464783abc9ad7b4a2ac8",
            "6561ce4bcf9c47168bbb02e195479907",
            "61ea2a70ac024677bb9b28d1140c88c1",
            "586d91b90d4a4eb1bb02ec65989b49ed",
            "a8aa72b64dd84ff39abc919943b68c12",
            "355b58cae4d84604a324e52e492970a5",
            "f3470052e4304f3cb1b741fd7a2a9c20",
            "981a04cde7194b54b067b8f3ee646bb0",
            "e7874c315554454694df390c280ea5d2",
            "4a8b11f843a74aa2bd22cc8abdf0a643",
            "0ddbc70b75e14c05a75eac4b692acdd8",
            "73076eedd8fe4a71aeaf978b7a1645cc",
            "c6104917058446df9c66e5586b3b0bd1",
            "b53d4cecef07433e8c1fe9f0af048189",
            "31ed325291e34b0181a4a1e6711a05ce",
            "3e384669c3a04b9c81111d92217ea4c4",
            "b0e6732f89b142a1ba1766db44aeab37",
            "453b70ae65ee472fbbefaa064a333c65",
            "70922c1e59cd417a91c3ca0d96ed4949",
            "118c8bbd5f384eb9a62665211d686ef6",
            "a503f3c8a0a84b09a1b1ace6ed317610",
            "c0f6f7797b7b4af7a8057be57b4cb41a",
            "eb0abe109f074fb1a30ae19e9ec7c849",
            "4b3121750a5b429c98c57f39fd7200b2",
            "2b11ce7bd0f3440fa0b5cb325590a637",
            "2c7dd0c7ad6e422cb085239a282b4cdf",
            "c4fdab7ee2414971bc315380d867a214",
            "92ef96d1c1f044e0b0066d66d4cb62a4",
            "ca22b928b9e746a6b650786f9af7b54c",
            "5531392cb8074487beb6506a80667ce1",
            "3c05a956926740a492dea9a651d900e9",
            "76888876c4ef4db490f3af865a27759d",
            "4d21e662de8f49cb8e33122ad70fad1d",
            "eba96b522a5041a8b28bb4100cdf143c",
            "6d4ddbdf309746c782cc9563051b8159",
            "1de81640ead54f8eb8c6fe3a96590bb3",
            "f2931e0a2c0d4267bfbb32369090a494",
            "883a4e8f600b489bb033a903a04c0432",
            "58ee935a5114426ebd9869b57ab1ee77",
            "8cf2fdfe310846e1ac02247c3f838004",
            "e3219593faf6410b9b02705206583005",
            "2fd9622bea434537b525730b70987c6f",
            "7043923b92a0433398571bc5c5117acf",
            "9b545098dbdf4821a0289e43f24b4195",
            "8bf15b4d9b82411181448ccf6b868a9c",
            "3597c865672642edabbb1bad0f6e8e75",
            "eaddca42b52f4df7b286401fd0306e12",
            "4d4033c56e244e669680f6b0ee87376e",
            "2a6ffe20f4644c778f02771e23041aa7",
            "421e1c82100748898a337fced5746935",
            "8df25b079a934ecfb0bed7f99d13036f",
            "977b61cffd364545a70a37b3456b97fe",
            "98dfe25e5a4d43deabbbe3edd4abe458",
            "c30a6f3c5fae430bb8d75dbf7a17218d",
            "eb861d5ce9554365939d9bf7c6e184d3",
            "8606cda6a4b1423fb5a65db1a4ca8ba9",
            "2ba408e5c20946dabb6a86ed8ad042d9",
            "6daa92960107487581486af97777ba0a",
            "35a69c629f6046f9ab8569266e669cae",
            "d6c7fad2417741e5bba9573e3ecb14b5",
            "52b58700ad9c4c9ebd1cfe526478d92a",
            "abba72ae7655477b8ebabe2bf7ed1ccd",
            "6681dafe4f7e4f3182792a5cffe8173f",
            "33e1490306f04a55865154776988dd16",
            "3ac141263cd84ea890e6a523769a3aae",
            "31e04955761a437187a60e51740b266b",
            "ddca84c3ef8e46668c142b9cb3ee0f9a",
            "0648f13f79ac46e28dab74bb15a81274",
            "74d33146d0b045aeb5c7f3ebe23f9c91",
            "74b342af82fc46e2815f452644eb78a2",
            "3ef09fae339f463789b21c62f8c4e932",
            "da0d56d661e249f884d0c15d6ae929c3",
            "7828e6d687fc43e49524db69901fa39e",
            "e3e509dc4465408f8caaa01e2e1dc3b4",
            "75b98e3ab1234fe7b920c8d0d89b5bed",
            "71badbcf6f574960a1410d4f75a62eb0",
            "fa100600f25b46f3b8cdd735a9c26bfb",
            "0a8904fb77ac4151be7c52407afc6fa9",
            "fc6b47254636478cb51347e206766efe",
            "bc371ea22e8549429ab605ae60e7126d",
            "8c2ba792ebf14cc49d0c9045967b8fc9",
            "ebfbbe5f7c71453ab530dbab11dd317a",
            "52784b87399b4dee92e48862c7442746",
            "1cb4ac14d7da4178b8ec207946facd6c"
          ]
        },
        "outputId": "ff2bc563-deb9-4f46-9e56-b47536af74b0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/735 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d9197efa50c0409badd62b208955da99"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors.index.json:   0%|          | 0.00/35.7k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "206d22fc95f14cd183845940599fd309"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2f4c3b4360fe4b40aec75639ef04b0ba"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model-00001-of-00002.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a7bee5212626426eac81918ee2822c76"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model-00002-of-00002.safetensors:   0%|          | 0.00/564M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3ba5c4ff1d8047c4b77cc6d976f7ba2e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ef18c41010c644969f3cc2ff4cc78a04"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "981a04cde7194b54b067b8f3ee646bb0"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/7.34k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "70922c1e59cd417a91c3ca0d96ed4949"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/798k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5531392cb8074487beb6506a80667ce1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e3219593faf6410b9b02705206583005"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/2.11M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "977b61cffd364545a70a37b3456b97fe"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "added_tokens.json:   0%|          | 0.00/1.08k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6681dafe4f7e4f3182792a5cffe8173f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/99.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e3e509dc4465408f8caaa01e2e1dc3b4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "torch.set_default_device(\"cuda\")\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\", torch_dtype=\"auto\", trust_remote_code=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\", trust_remote_code=True)\n",
        "\n",
        "def generate_output(model, input, max_length=300):\n",
        "  input = f\"Question: {input}\\nOutput:\"\n",
        "  input = tokenizer(input, return_tensors=\"pt\", return_attention_mask=False)\n",
        "  outputs = model.generate(**input, max_length=max_length)\n",
        "  text = tokenizer.batch_decode(outputs)[0]\n",
        "  return text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0ZVZVgi8r0Y"
      },
      "source": [
        "Use Phi-2 to answer the questions below with and without CoT. Compare results and explain their difference. (4 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_rJQJ3VY8r0Z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd9cb892-9f21-4de7-e3fb-1a9d4667e7fa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Without CoT\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n",
            "Output: Weng earned $9 for babysitting.\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: To find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "\n",
            "2 liters x 20% = 0.4 liters\n",
            "\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "\n",
            "0.4 liters x 1000 = 400 ml\n",
            "\n",
            "Therefore, Jack will get 400 ml of salt when all the water evaporates.\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers a total of 36 hours per year (2 hours x 12 months = 24 hours per year; 24 hours x 2 months = 48 hours; 48 hours - 24 hours = 24 hours).\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so there are 32/2 = 16 tables with 2 chairs each.\n",
            "\n",
            "5 tables have 3 chairs each, so there are 5 x 3 = 15 tables with 3 chairs each.\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each.\n",
            "\n",
            "The total number of chairs in the hall is 16 x 2 + 15 x 3 + 11 x 4 = 32 + 45 + 44 = 121.\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, it takes Bert 1050 words to use up a pencil, so he uses up a pencil every 1050/2 = 525 words.\n",
            "Since he fills out the crossword puzzle every day, he fills out a crossword puzzle every 525/7 = 75 words.\n",
            "Therefore, on average, there are 75 words in each crossword puzzle.\n",
            "<|endoftext|>\n"
          ]
        }
      ],
      "source": [
        "def generate_output(model, input, max_length=300):\n",
        "  input = f\"Question: {input}\\nOutput:\"\n",
        "  input = tokenizer(input, return_tensors=\"pt\", return_attention_mask=False)\n",
        "  outputs = model.generate(**input, max_length=max_length)\n",
        "  text = tokenizer.batch_decode(outputs)[0]\n",
        "  return text\n",
        "\n",
        "import logging\n",
        "\n",
        "# Set logging level to ERROR to suppress warning messages\n",
        "logging.getLogger(\"transformers\").setLevel(logging.ERROR)\n",
        "\n",
        "questions_without_CoT = [\"Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\",\n",
        "\"Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\",\n",
        "\"John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\",\n",
        "\"There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\",\n",
        "\"Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\"\n",
        "]\n",
        "\n",
        "answers_without_CoT = []\n",
        "print('Without CoT')\n",
        "for question in questions_without_CoT:\n",
        "    print(100*'-')\n",
        "    generated_text = generate_output(model, question)\n",
        "    print(100*'-')\n",
        "    print(generated_text)\n",
        "    answers_without_CoT.append(generated_text)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "questions_with_CoT = [\"\"\"Weng earns $21 an hour for babysitting. Yesterday, she just did 20 minutes of babysitting. How much did she earn?\n",
        "Output: 20 minutes is 20/60 hours, so she earned 20/60 * 21 = 7\n",
        "The answer is 7\n",
        "Now answer this question:\n",
        "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\"\"\",\n",
        "\n",
        "\"\"\"Jack is stranded on a desert island. He wants some salt to season his fish. He collects 6 liters of seawater in an old bucket. If the water is 50% salt, how many ml of salt will Jack get when all the water evaporates?\n",
        "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt.\n",
        "6 liters x 50% = 3 liters\n",
        "To convert liters to milliliters, we need to multiply by 1000.\n",
        "3 liters x 1000 = 3000 ml\n",
        "Therefore, Jack will get 3000 ml of salt when all the water evaporates.\n",
        "The answer is 3000\n",
        "Now answer this question:\n",
        "Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\"\"\",\n",
        "\n",
        "\n",
        "\"\"\"John volunteers at a shelter twice a month for 2 hours at a time. How many hours does he volunteer per year?\n",
        "Output: John volunteers 2 hours * 2 times = 4 hours per month\n",
        "so because each year has 12 months, he volunteers a total 12 * 4 = 48 hours\n",
        "The answer is 48\n",
        "Now answer this question:\n",
        "John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\"\"\",\n",
        "\n",
        "\n",
        "\"\"\"There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
        "Output: There are 40 tables in the hall.\n",
        "\n",
        "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
        "\n",
        "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
        "\n",
        "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
        "\n",
        "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
        "The answer is 124\n",
        "\n",
        "\n",
        "Now answer this question:\n",
        "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\"\"\",\n",
        "\n",
        "\n",
        "\"\"\"Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
        "Output: On average, Bert fills out 7 number of puzzles each week.\n",
        " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
        "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
        "The answer is 100\n",
        "Now answer this question:\n",
        "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\"\"\"\n",
        "]\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "answers_with_CoT = []\n",
        "print('With CoT')\n",
        "for question in questions_with_CoT:\n",
        "    print(100*'-')\n",
        "    generated_text = generate_output(model, question, max_length = 500)\n",
        "    print(100*'-')\n",
        "    print(generated_text)\n",
        "    answers_with_CoT.append(generated_text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LujSRc2e18gU",
        "outputId": "a74ada18-9a2d-41aa-a228-e4f30ad7020d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "With CoT\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Weng earns $21 an hour for babysitting. Yesterday, she just did 20 minutes of babysitting. How much did she earn?\n",
            "Output: 20 minutes is 20/60 hours, so she earned 20/60 * 21 = 7\n",
            "The answer is 7\n",
            "Now answer this question: \n",
            "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n",
            "Output: 50 minutes is 50/60 hours, so she earned 50/60 * 12 = 10\n",
            "The answer is 10\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Jack is stranded on a desert island. He wants some salt to season his fish. He collects 6 liters of seawater in an old bucket. If the water is 50% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "6 liters x 50% = 3 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "3 liters x 1000 = 3000 ml\n",
            "Therefore, Jack will get 3000 ml of salt when all the water evaporates.\n",
            "The answer is 3000\n",
            "Now answer this question: \n",
            "Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "2 liters x 20% = 0.4 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "0.4 liters x 1000 = 400 ml\n",
            "Therefore, Jack will get 400 ml of salt when all the water evaporates.\n",
            "The answer is 400\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: John volunteers at a shelter twice a month for 2 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 2 hours * 2 times = 4 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 4 = 48 hours\n",
            "The answer is 48\n",
            "Now answer this question: \n",
            "John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 3 hours * 2 times = 6 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 6 = 72 hours\n",
            "The answer is 72\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 2 chairs each (16 * 2 = 32 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each. ( 11 * 4 = 44 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 32 + 15 + 44 = 91.\n",
            "The answer is 91.\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 7 * 2 = 14 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 14 = 75 words.\n",
            "The answer is 75\n",
            "<|endoftext|>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5sg-W9Wz8r0Z"
      },
      "source": [
        "we can see that the normal prompting ( without CoT ) the model answered Q1, 3, 4 wrong.\n",
        "but in CoT we improved the model's answer by providing chain of ideas through an example for each prompt and the model answers all the questions correctly"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pknDd4gH8r0Z"
      },
      "source": [
        "## Other Methods for Reasoning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2e89r-dJ8r0Z"
      },
      "source": [
        "There are many other approaches to utilize the reasoning abilities of LLMs. Describe the [Tree-of-Thought (ToT)](https://arxiv.org/abs/2305.10601) and [Self-Consistency](https://arxiv.org/abs/2203.11171) within these approaches. (3.5 Points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EMtIiwoy8r0a"
      },
      "source": [
        "ToT: similar to CoT, the model is given the chain of ideas, but in this method instead of a linear approach to the answer, its given the different branches and alternate approaches of solving the problem, and in the end these branchs should converge for a single solution task\n",
        "\n",
        "self-Consistency: it gives the CoT version of prompt to the model and uses sampling for decoding ( not using greedy decoding ) to get a diverse set of reasoning paths, and marginalize out the reasoning part and aggregate by choosing the most consistent answer ( final answer )."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4j3GS92m8r0a"
      },
      "source": [
        "Now, implement Self-Consistency to answer the questions of the previous section. (6 Points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "johUo1HM8r0a"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "def generate_output(model, input, max_length=300, temperature=0.7):\n",
        "    input_text = f\"Question: {input}\\nOutput:\"\n",
        "    input_tokens = tokenizer(input_text, return_tensors=\"pt\", return_attention_mask=False)\n",
        "\n",
        "    outputs = model.generate(\n",
        "        **input_tokens,\n",
        "        max_length=max_length,\n",
        "        do_sample=True,\n",
        "        temperature=temperature\n",
        "    )\n",
        "\n",
        "    text = tokenizer.batch_decode(outputs)[0]\n",
        "    return text\n",
        "\n",
        "def find_most_consistent_answer(model, input, path_nums, max_length=300, temperature=0.7):\n",
        "    consistent_answers = {}\n",
        "\n",
        "    for path_index in range(path_nums):\n",
        "        # Generate output using sampling\n",
        "        output_text = generate_output(model, input, max_length=max_length, temperature=temperature)\n",
        "\n",
        "        # Extract the last sentence\n",
        "        last_sentence = output_text.split('\\n')[-2]\n",
        "\n",
        "        # Update consistent_answers dictionary\n",
        "        if last_sentence.lower().startswith(\"the answer is\"):\n",
        "            answer = last_sentence[len(\"The answer is \"):].rstrip('.')\n",
        "            if answer not in consistent_answers:\n",
        "                consistent_answers[answer] = []\n",
        "            consistent_answers[answer].append(output_text)\n",
        "\n",
        "    return consistent_answers\n",
        "\n",
        "def print_answers_count_and_output(consistent_answers, path_nums):\n",
        "    for answer, outputs in consistent_answers.items():\n",
        "        percentage_of_answer = ( len(outputs) / path_nums ) * 100\n",
        "        output = outputs[0] # for simplicity we return the first one\n",
        "        print(f'Answer = {answer}, has {percentage_of_answer}% of the total answers\\nOutput: {output}')\n",
        "        print(100*'*')\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "PATHS_NUM = 5\n",
        "for question in questions_with_CoT:\n",
        "\n",
        "    consistent_answers = find_most_consistent_answer(model, question, PATHS_NUM, max_length = 500)\n",
        "    print_answers_count_and_output(consistent_answers, PATHS_NUM)\n",
        "    print(100*'-')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1Q-7trnuGas4",
        "outputId": "0d0245fe-3df4-42fa-e431-b875ca079d82"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Answer = 10, has 100.0% of the total answers\n",
            "Output: Question: Weng earns $21 an hour for babysitting. Yesterday, she just did 20 minutes of babysitting. How much did she earn?\n",
            "Output: 20 minutes is 20/60 hours, so she earned 20/60 * 21 = 7\n",
            "The answer is 7\n",
            "Now answer this question: \n",
            "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n",
            "Output: 50 minutes is 50/60 hours, so she earned 50/60 * 12 = 10\n",
            "The answer is 10\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 400, has 80.0% of the total answers\n",
            "Output: Question: Jack is stranded on a desert island. He wants some salt to season his fish. He collects 6 liters of seawater in an old bucket. If the water is 50% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "6 liters x 50% = 3 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "3 liters x 1000 = 3000 ml\n",
            "Therefore, Jack will get 3000 ml of salt when all the water evaporates.\n",
            "The answer is 3000\n",
            "Now answer this question: \n",
            "Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: \n",
            "To find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "2 liters x 20% = 0.4 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "0.4 liters x 1000 = 400 ml\n",
            "Therefore, Jack will get 400 ml of salt when all the water evaporates.\n",
            "The answer is 400\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 72, has 100.0% of the total answers\n",
            "Output: Question: John volunteers at a shelter twice a month for 2 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 2 hours * 2 times = 4 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 4 = 48 hours\n",
            "The answer is 48\n",
            "Now answer this question: \n",
            "John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 3 hours * 2 times = 6 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 6 = 72 hours\n",
            "The answer is 72\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 91, has 40.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 2 chairs each (16 * 2 = 32 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each. (11 * 4 = 44 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 32 + 15 + 44 = 91.\n",
            "The answer is 91.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 83, has 20.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "The first step is to calculate the total number of chairs in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 2 chairs each (16 * 2 = 32 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 9 tables with 4 chairs each. ( 9 * 4 = 36 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 32 + 15 + 36 = 83.\n",
            "The answer is 83.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 131.25, has 40.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output:\n",
            "On average, Bert fills out 4 number of puzzles each week.\n",
            "He uses up a pencil to fill out the puzzles every two weeks, so he fills 4 * 2 = 8 puzzles with one pencil.\n",
            "So if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 8 = 131.25 words.\n",
            "The answer is 131.25\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 87.5, has 20.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 6 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 6 * 2 = 12 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 12 = 87.5 words.\n",
            "The answer is 87.5\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "PATHS_NUM = 20\n",
        "for question in questions_with_CoT:\n",
        "\n",
        "    consistent_answers = find_most_consistent_answer(model, question, PATHS_NUM, max_length = 500)\n",
        "    print_answers_count_and_output(consistent_answers, PATHS_NUM)\n",
        "    print(100*'-')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sI5HgvasQF00",
        "outputId": "e6a00cb8-0a7d-4d04-ddd3-7fc0d1a35328"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Answer = 10, has 95.0% of the total answers\n",
            "Output: Question: Weng earns $21 an hour for babysitting. Yesterday, she just did 20 minutes of babysitting. How much did she earn?\n",
            "Output: 20 minutes is 20/60 hours, so she earned 20/60 * 21 = 7\n",
            "The answer is 7\n",
            "Now answer this question: \n",
            "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?\n",
            "Output: 50 minutes is 50/60 hours, so she earned 50/60 * 12 = 10\n",
            "The answer is 10\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 400, has 75.0% of the total answers\n",
            "Output: Question: Jack is stranded on a desert island. He wants some salt to season his fish. He collects 6 liters of seawater in an old bucket. If the water is 50% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "6 liters x 50% = 3 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "3 liters x 1000 = 3000 ml\n",
            "Therefore, Jack will get 3000 ml of salt when all the water evaporates.\n",
            "The answer is 3000\n",
            "Now answer this question: \n",
            "Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: 2 liters x 20% = 0.4 liters\n",
            "0.4 liters x 1000 = 400 ml\n",
            "Therefore, Jack will get 400 ml of salt when all the water evaporates.\n",
            "The answer is 400\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 1000, has 5.0% of the total answers\n",
            "Output: Question: Jack is stranded on a desert island. He wants some salt to season his fish. He collects 6 liters of seawater in an old bucket. If the water is 50% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: to find the amount of salt in the seawater, we need to multiply the volume of water by the percentage of salt. \n",
            "6 liters x 50% = 3 liters\n",
            "To convert liters to milliliters, we need to multiply by 1000.\n",
            "3 liters x 1000 = 3000 ml\n",
            "Therefore, Jack will get 3000 ml of salt when all the water evaporates.\n",
            "The answer is 3000\n",
            "Now answer this question: \n",
            "Jack is stranded on a desert island. He wants some salt to season his fish. He collects 2 liters of seawater in an old bucket. If the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?\n",
            "Output: you have made a mistake in the percentage of salt in the seawater. It is not 20%, but 50%. \n",
            "2 liters x 50% = 1 liter\n",
            "1 liter x 1000 = 1000 ml\n",
            "Therefore, Jack will get 1000 ml of salt when all the water evaporates.\n",
            "The answer is 1000\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 72, has 95.0% of the total answers\n",
            "Output: Question: John volunteers at a shelter twice a month for 2 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 2 hours * 2 times = 4 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 4 = 48 hours\n",
            "The answer is 48\n",
            "Now answer this question: \n",
            "John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 3 hours * 2 times = 6 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 6 = 72 hours\n",
            "The answer is 72\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 18, has 5.0% of the total answers\n",
            "Output: Question: John volunteers at a shelter twice a month for 2 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 2 hours * 2 times = 4 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 4 = 48 hours\n",
            "The answer is 48\n",
            "Now answer this question: \n",
            "John volunteers at a shelter twice a month for 3 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 3 hours * 2 times = 6 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 6 = 72 hours\n",
            "The answer is 72\n",
            "Next question: \n",
            "John volunteers at a shelter once a month for 1.5 hours at a time. How many hours does he volunteer per year?\n",
            "Output: John volunteers 1.5 hours * 1 times = 1.5 hours per month\n",
            "so because each year has 12 months, he volunteers a total 12 * 1.5 = 18 hours\n",
            "The answer is 18\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 59, has 5.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 4 chairs each (16 * 2 = 32 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 15 = 3 tables with 4 chairs each. ( 3 * 4 = 12 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 32 + 15 + 12 = 59.\n",
            "The answer is 59.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 107, has 5.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 3 chairs each (16 * 3 = 48 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each. ( 11 * 4 = 44 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 48 + 15 + 44 = 107.\n",
            "The answer is 107.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 91, has 65.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 2 chairs each (16 * 2 = 32 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each. (11 * 4 = 44 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 32 + 15 + 44 = 91.\n",
            "The answer is 91.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 123, has 10.0% of the total answers\n",
            "Output: Question: There are 40 tables in a hall. Half the tables have 3 chairs each, 8 have 5 chairs each and the rest have 2 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 40 tables in the hall.\n",
            "\n",
            "Half of the tables have 3 chairs each, so half of the 40 are 40/2 = 20 tables with 2 chairs each (20 * 3 = 60 chairs).\n",
            "\n",
            "8 tables have 5 chairs each, (so there are more 8 x 5 = 40 chairs)\n",
            "\n",
            "The rest of the tables have 2 chairs each, so there are 40 - 20 - 8 = 12 tables with 2 chairs each. ( 12 * 2 = 24 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 60 + 40 + 24 = 124.\n",
            "The answer is 124\n",
            "\n",
            "\n",
            "Now answer this question:\n",
            "There are 32 tables in a hall. Half the tables have 2 chairs each, 5 have 3 chairs each and the rest have 4 chairs each. How many chairs in total are in the hall?\n",
            "Output: There are 32 tables in the hall.\n",
            "\n",
            "Half of the tables have 2 chairs each, so half of the 32 are 32/2 = 16 tables with 4 chairs each (16 * 4 = 64 chairs).\n",
            "\n",
            "5 tables have 3 chairs each, (so there are more 5 x 3 = 15 chairs)\n",
            "\n",
            "The rest of the tables have 4 chairs each, so there are 32 - 16 - 5 = 11 tables with 4 chairs each. ( 11 * 4 = 44 chairs)\n",
            "\n",
            "The total number of chairs in the hall is 64 + 15 + 44 = 123.\n",
            "The answer is 123.\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Answer = 75, has 40.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week. \n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 7 * 2 = 14 puzzles with one pencil. \n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 14 = 75 words. \n",
            "The answer is 75\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 58.33, has 5.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 9 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 9 * 2 = 18 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 18 = 58.33 words.\n",
            "The answer is 58.33\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 175, has 15.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: Bert fills out 3 number of puzzles each week. He uses up a pencil to fill out the puzzles every two weeks, so he fills 3 * 2 = 6 puzzles with one pencil.\n",
            "So if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 6 = 175 words.\n",
            "The answer is 175\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 525, has 5.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 1 number of puzzles each week.\n",
            "He uses up a pencil to fill out the puzzles every two weeks, so he fills 1 * 2 = 2 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 2 = 525 words.\n",
            "The answer is 525\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 38.571428571428575, has 5.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 14 number of puzzles each week. He uses up a pencil to fill out the puzzles every two weeks, so he fills 14 * 2 = 28 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 28 = 38.571428571428575 words.\n",
            "The answer is 38.571428571428575\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 105, has 5.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 5 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 5 * 2 = 10 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 10 = 105 words.\n",
            "The answer is 105\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "Answer = 300, has 5.0% of the total answers\n",
            "Output: Question: Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every four weeks. On average, it takes him 2800 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every four weeks, so he fills 7 * 4 = 28 puzzles with one pencil.\n",
            "so if it takes him 2800 words to use up a pencil, each puzzle should have about 2800 / 28 = 100 words.\n",
            "The answer is 100\n",
            "Now answer this question:\n",
            "Bert fills out the daily crossword puzzle in the newspaper every day. He uses up a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use up a pencil. How many words are in each crossword puzzle on average?\n",
            "Output: On average, Bert fills out 7 number of puzzles each week.\n",
            " He uses up a pencil to fill out the puzzles every two weeks, so he fills 7 / 2 = 3.5 puzzles with one pencil.\n",
            "so if it takes him 1050 words to use up a pencil, each puzzle should have about 1050 / 3.5 = 300 words.\n",
            "The answer is 300\n",
            "<|endoftext|>\n",
            "****************************************************************************************************\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "if the sum of percentages is not 100, its because the format of the generated prompt ( last sentece ) wasnt what we wanted\n",
        "\n",
        "we asume that the model should generate ( The answer is x ) based on the provided example"
      ],
      "metadata": {
        "id": "JpxgTstQOpS3"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S72rMyRB8r0a"
      },
      "source": [
        "Consider LLMs' features and propose a new approach based on them to enhance LLMs' reasoning abilities. Why do you believe this approach could enhance LLMs' reasoning abilities? (4 Points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FE_ofNWt8r0a"
      },
      "source": [
        "we can provide a template like this so we drive the model to fill some tokens like \" %TEXT% \" and define it for the model so it knows where to fill\n",
        "\n",
        "it can help us control the output of the model and drive it to what we desire."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZEJ24NQ8r0a"
      },
      "source": [
        "# PEFT (30 + 5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gkO9dQKLx0rT"
      },
      "source": [
        "If you have any further questions or concerns, contact the TA via email: pedram.rostami@ut.ac.ir"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PqVk1GA88r0a"
      },
      "source": [
        "## Why We Are Using PEFT (5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2qqYmR1r8r0a"
      },
      "source": [
        "In this question, we're delving into PEFT. First, let's start by exploring why PEFT is crucial when training LLMs. For instance, let's consider the scenario where we want to train the [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) model. To get started, take a look at the Huggingface blog post on [model memory anatomy](https://huggingface.co/docs/transformers/en/model_memory_anatomy) to estimate how much memory we'll require. Just assume we're sticking to pure fp16 with Adam optimizer and a batch size of 1. (4 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NolwIS0f8r0b"
      },
      "source": [
        "there are many components during training that use GPU memory. The components on GPU memory are the following:\n",
        "\n",
        "model_parameters = 2.7Bill\n",
        "\n",
        "model weights ( 2Byte * 2.7Bill = 5.4GByte) 2 because of fp16\n",
        "\n",
        "optimizer states ( twice of the model weights: 10.8GByte)\n",
        "\n",
        "gradients: ( 4Byte * 2.7Bill = 10.8GByte) gradients are fp32\n",
        "\n",
        "for a backward pass: 21.6GByte\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A9qk7vrV8r0b"
      },
      "source": [
        "Compare your estimation with the memory estimation provided by the [Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage). (1 point)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNgYjy_G8r0b"
      },
      "source": [
        "they are around the same amount, it varies a little because of the model parameters arent exactly 2.7"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOUMsW8d8r0b"
      },
      "source": [
        "## Preparing Dataset (5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IBcrnR0J8r0b"
      },
      "source": [
        "We're going to train the phi-2 model for a question generation task based on passages. For this purpose, we're using the Super-NaturalInstruction dataset, which comprises instruction tuning datasets for over 1600 tasks across different languages. While the dataset is available on the [Huggingface Hub](https://huggingface.co/datasets/Muennighoff/natural-instructions), downloading all its components consumes considerable time. Consequently, we're opting to download only the English Question Generation segment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RT3tUiuV8r0b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d24e24d-8ad9-4e67-be98-641642766b77"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-08 08:58:21--  https://huggingface.co/datasets/Muennighoff/natural-instructions/resolve/main/train/task001_quoref_question_generation_train.jsonl\n",
            "Resolving huggingface.co (huggingface.co)... 13.33.30.49, 13.33.30.114, 13.33.30.23, ...\n",
            "Connecting to huggingface.co (huggingface.co)|13.33.30.49|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://cdn-lfs.huggingface.co/repos/a1/fe/a1fedd93d2c00f67a096c36747356c03b6f01649bae4b4be932e6531a496022a/89ad3018bdb2cec45afea661fbe2fc8df9593243f58531d381c19b5fb13ce581?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27task001_quoref_question_generation_train.jsonl%3B+filename%3D%22task001_quoref_question_generation_train.jsonl%22%3B&Expires=1715417901&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxNTQxNzkwMX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9hMS9mZS9hMWZlZGQ5M2QyYzAwZjY3YTA5NmMzNjc0NzM1NmMwM2I2ZjAxNjQ5YmFlNGI0YmU5MzJlNjUzMWE0OTYwMjJhLzg5YWQzMDE4YmRiMmNlYzQ1YWZlYTY2MWZiZTJmYzhkZjk1OTMyNDNmNTg1MzFkMzgxYzE5YjVmYjEzY2U1ODE%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=u%7EAsSirNLpbYGkg4bKNMivjbELDfEyn%7EcoiQpLZ4rYU-VOnf%7Ed20dKcTlodTdzUXLaT8lgi0Pej7cbUMeDDPOVH8cAB%7EpRsjOWbMIOHUjwhKb52f5zpbQwnkq0pcT4gbI%7EUcOiM18nudFzw0Aw%7ELbiLqMhAscPFNbEE0a12ThBwZWxJpTsUgwdvxSG9bYECehi3rbnnQ6RxOhQeQ-hwbgK62fbWA%7E%7E901RFgvxErS3XgAsqQTHYdTq7WlhLpsDbvJY81VX%7EiaiH-8OLvm%7EjwEj0G0W5jctLEI8giDYazDgnr5WM9mBw9LEB-Yw1lq7yhxhFkuaVt2lJ419dREnr-Tg__&Key-Pair-Id=KVTP0A1DKRTAX [following]\n",
            "--2024-05-08 08:58:21--  https://cdn-lfs.huggingface.co/repos/a1/fe/a1fedd93d2c00f67a096c36747356c03b6f01649bae4b4be932e6531a496022a/89ad3018bdb2cec45afea661fbe2fc8df9593243f58531d381c19b5fb13ce581?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27task001_quoref_question_generation_train.jsonl%3B+filename%3D%22task001_quoref_question_generation_train.jsonl%22%3B&Expires=1715417901&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxNTQxNzkwMX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9hMS9mZS9hMWZlZGQ5M2QyYzAwZjY3YTA5NmMzNjc0NzM1NmMwM2I2ZjAxNjQ5YmFlNGI0YmU5MzJlNjUzMWE0OTYwMjJhLzg5YWQzMDE4YmRiMmNlYzQ1YWZlYTY2MWZiZTJmYzhkZjk1OTMyNDNmNTg1MzFkMzgxYzE5YjVmYjEzY2U1ODE%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qIn1dfQ__&Signature=u%7EAsSirNLpbYGkg4bKNMivjbELDfEyn%7EcoiQpLZ4rYU-VOnf%7Ed20dKcTlodTdzUXLaT8lgi0Pej7cbUMeDDPOVH8cAB%7EpRsjOWbMIOHUjwhKb52f5zpbQwnkq0pcT4gbI%7EUcOiM18nudFzw0Aw%7ELbiLqMhAscPFNbEE0a12ThBwZWxJpTsUgwdvxSG9bYECehi3rbnnQ6RxOhQeQ-hwbgK62fbWA%7E%7E901RFgvxErS3XgAsqQTHYdTq7WlhLpsDbvJY81VX%7EiaiH-8OLvm%7EjwEj0G0W5jctLEI8giDYazDgnr5WM9mBw9LEB-Yw1lq7yhxhFkuaVt2lJ419dREnr-Tg__&Key-Pair-Id=KVTP0A1DKRTAX\n",
            "Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 18.155.68.94, 18.155.68.73, 18.155.68.128, ...\n",
            "Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|18.155.68.94|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 65450952 (62M) [text/plain]\n",
            "Saving to: ‘task001_quoref_question_generation_train.jsonl’\n",
            "\n",
            "task001_quoref_ques 100%[===================>]  62.42M   266MB/s    in 0.2s    \n",
            "\n",
            "2024-05-08 08:58:22 (266 MB/s) - ‘task001_quoref_question_generation_train.jsonl’ saved [65450952/65450952]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://huggingface.co/datasets/Muennighoff/natural-instructions/resolve/main/train/task001_quoref_question_generation_train.jsonl"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEUzDJvM8r0b"
      },
      "source": [
        "Read the dataset file and convert it into a `dataset` object. Then, split the dataset, selecting 95% for the training set and 5% for the test set. (5 points)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q datasets"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dEkCfqMtYBAW",
        "outputId": "f3375639-86d3-475a-9ca2-4cea0517bcc8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.0/542.0 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m15.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m401.2/401.2 kB\u001b[0m \u001b[31m32.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from torch.utils.data import Dataset, DataLoader, random_split\n",
        "from datasets import load_dataset\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "'''class JSONLineDataset(Dataset):\n",
        "    def __init__(self, file_path):\n",
        "        self.data = []\n",
        "        with open(file_path, 'r') as file:\n",
        "            for line in file:\n",
        "                self.data.append(json.loads(line))\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        # Assuming each line has a 'text' and a 'label' field\n",
        "        instruction = self.data[idx]['definition']\n",
        "        input = self.data[idx]['inputs']\n",
        "        response = self.data[idx]['targets']\n",
        "        return {'Instruction': instruction, 'Input': input, 'Response': response}\n",
        "'''\n",
        "# Example usage\n",
        "jsonl_file = 'task001_quoref_question_generation_train.jsonl'\n",
        "#custom_dataset = JSONLineDataset(jsonl_file)\n",
        "custom_dataset = load_dataset('json', data_files=jsonl_file)\n",
        "custom_dataset = custom_dataset['train'].train_test_split(test_size=0.05)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "J_TYGfgAuYLE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_puOe458r0b"
      },
      "source": [
        "## Pretrained Model (5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPto8Fqy8r0c"
      },
      "source": [
        "Choose random samples from the test set, apply the [Alpaca template](https://github.com/tatsu-lab/stanford_alpaca?tab=readme-ov-file#data-release) to them, and obtain the model outputs (If you are using the [sample code](https://huggingface.co/microsoft/phi-2#sample-code) provided by Microsoft for using the model, please comment out the `torch.set_default_device(\"cuda\")` line to conserve memory. Instead, you can move the model to the GPU using the `.to` function after loading it.). (5 points)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -i https://pypi.org/simple/ bitsandbytes"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9tdWcFQzlMPW",
        "outputId": "6501b929-bd35-46fa-b4dc-5cf24f3c7345"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple/\n",
            "Collecting bitsandbytes\n",
            "  Downloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl (119.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.8/119.8 MB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from bitsandbytes) (2.2.1+cu121)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bitsandbytes) (1.25.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (3.14.0)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (4.11.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (3.1.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (2023.6.0)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "Collecting nvidia-curand-cu12==10.3.2.106 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "Collecting nvidia-nccl-cu12==2.19.3 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
            "Collecting nvidia-nvtx-cu12==12.1.105 (from torch->bitsandbytes)\n",
            "  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch->bitsandbytes) (2.2.0)\n",
            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch->bitsandbytes)\n",
            "  Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->bitsandbytes) (2.1.5)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->bitsandbytes) (1.3.0)\n",
            "Installing collected packages: nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, nvidia-cusparse-cu12, nvidia-cudnn-cu12, nvidia-cusolver-cu12, bitsandbytes\n",
            "Successfully installed bitsandbytes-0.43.1 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.1.105\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#!pip install transformers==4.30 #downgrade for quantization\n",
        "! pip install -U accelerate\n",
        "! pip install -U transformers"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UOhWlE1AmrN8",
        "outputId": "77047b54-a074-4168-e5e6-ad58493fe132"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting accelerate\n",
            "  Downloading accelerate-0.30.0-py3-none-any.whl (302 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.4/302.4 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.25.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (24.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.1)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.2.1+cu121)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.23.0)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.4.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.14.0)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (4.11.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.1.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2023.6.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate) (12.4.127)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->accelerate) (2.31.0)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub->accelerate) (4.66.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.5)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub->accelerate) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub->accelerate) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub->accelerate) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub->accelerate) (2024.2.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)\n",
            "Installing collected packages: accelerate\n",
            "Successfully installed accelerate-0.30.0\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.40.1)\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.40.2-py3-none-any.whl (9.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.0/9.0 MB\u001b[0m \u001b[31m70.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.14.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2023.12.25)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.11.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.2.2)\n",
            "Installing collected packages: transformers\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.40.1\n",
            "    Uninstalling transformers-4.40.1:\n",
            "      Successfully uninstalled transformers-4.40.1\n",
            "Successfully installed transformers-4.40.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yH5Waa3E8r0c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287,
          "referenced_widgets": [
            "b075cd11497d46ffa0b09c4fbe04dfb5",
            "5fdbc9d55ddb47c79c2dca69049fbbec",
            "7e0109f5225747eda16efdf2daee1654",
            "f83ad35247814c6cb926943a3c425c7f",
            "6125807b32964b969cceadb16b6fd056",
            "d415ef1b1292483480d325cc6e425f5c",
            "1a82a2d7ea744508ad1fe7e7a8cb5f67",
            "60903ed736a94152a0dceb34c726c2fb",
            "26efd76a115f4f54ab2f3bab056cc0a0",
            "708c63985d0f4df881d49e7d700dccc7",
            "b03e66157d5b4f2a90a249d0f74e1200"
          ]
        },
        "outputId": "4e749eac-de10-4bbf-8eb8-ab7a3aa675b4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n",
            "`low_cpu_mem_usage` was None, now set to True since model is quantized.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b075cd11497d46ffa0b09c4fbe04dfb5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
        "\n",
        "#torch.set_default_device(\"cuda\")\n",
        "\n",
        "quant_config = BitsAndBytesConfig(\n",
        "        load_in_4bit=True,\n",
        "        bnb_4bit_quant_type=\"nf4\",\n",
        "        bnb_4bit_compute_dtype=getattr(torch, \"bfloat16\"),\n",
        "        bnb_4bit_use_double_quant=True,\n",
        "    )\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    \"microsoft/phi-2\",\n",
        "    torch_dtype=\"auto\",\n",
        "    trust_remote_code=True,\n",
        "    quantization_config=quant_config\n",
        "    )\n",
        "#model.to('cuda') #model is already in gpu cause of quantization\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\", trust_remote_code=True)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "test_dataset.keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wkmLsjIg1JL0",
        "outputId": "9f9914ea-69fd-4164-cb2c-db68c7a8750c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['task_name', 'id', 'definition', 'inputs', 'targets'])"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "\n",
        "# Set logging level to ERROR to suppress warning messages\n",
        "logging.getLogger(\"transformers\").setLevel(logging.ERROR)"
      ],
      "metadata": {
        "id": "0wtouxVW5354"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "# Function to apply the Alpaca template to input data\n",
        "def apply_template(input_data):\n",
        "    instruction = input_data['definition']\n",
        "    input_text = input_data['inputs']\n",
        "    response = input_data['targets']\n",
        "    return instruction, input_text, response\n",
        "\n",
        "\n",
        "def get_model_output(input_text):\n",
        "    # Tokenize the input text\n",
        "    inputs = tokenizer(input_text, return_tensors=\"pt\", return_attention_mask=False).to('cuda')\n",
        "\n",
        "    # Generate outputs using the model\n",
        "    outputs = model.generate(**inputs, max_length=1000)\n",
        "\n",
        "    # Decode the generated outputs\n",
        "    output_text = tokenizer.batch_decode(outputs)[0]\n",
        "\n",
        "    return output_text\n",
        "\n",
        "\n",
        "num_samples = 5  # Number of samples to choose\n",
        "random_indices = random.sample(range(len(custom_dataset['test'])), num_samples)\n",
        "random_samples = [custom_dataset['test'][i] for i in random_indices]\n",
        "# Apply Alpaca template and obtain model outputs for each sample\n",
        "\n",
        "for sample in random_samples:\n",
        "    instruction, input_text, response = apply_template(sample)\n",
        "    prompt = f'Instruction:\\n{instruction}\\nInput:\\n{input_text}\\nResponse:'\n",
        "    model_output = get_model_output(prompt)\n",
        "    print(model_output)\n",
        "    print(\"-\" * 100)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K38I2KB50px1",
        "outputId": "31629f96-4553-4a0c-a171-d60b1e9cbf85"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Instruction:\n",
            "In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.\n",
            "Input:\n",
            "Passage: The duo were disappointed with their performance, particularly Garfunkel, who felt that he sang poorly. Simon said that he did not immediately realize the magnitude of the event: \"I didn't get what had happened – how big it was – until I went home, turned on the television and saw it on all the news... and later that night on the front pages of all the newspapers. Then I got it.\"In May 1982, Simon & Garfunkel went on a world tour with stops in Japan, Germany, Denmark, Sweden, Switzerland, the Netherlands, Ireland, France, Great Britain, New Zealand, the US and Canada. The European leg of their tour began on May 28, 1982, at the Stadion am Bieberer Berg in Offenbach am Main. This was their first performance in Germany, and had an attendance of around 40,000 spectators.When they were not on the road, the duo went into the studio to work on what was to be a reunion Simon & Garfunkel album, tentatively entitled Think Too Much, with Garfunkel adding harmony vocals to a bunch of new songs for which Simon had already laid down some backing tracks. They set a release date of spring 1983 to coincide with their planned North American tour, but after increasingly acrimonious delays and disagreements, Simon told Warner Brothers he could no longer work with Garfunkel and that the project as an S&G album was cancelled. Thus Garfunkel dropped out of the project, which then became Simon's November 1983 solo album Hearts and Bones.Several years would pass before Simon & Garfunkel worked together again. Their next joint public appearance was in 1990, when they performed for their induction into the Rock and Roll Hall of Fame. When Simon gave another free concert in Central Park on August 15, 1991, he rejected Garfunkel's offer to participate. However, they agreed to perform together in 1993 for 21 sold out concerts in New York, with half of the show being Paul Simon solo with a band and the other half Simon and Garfunkel. Later the same year, they did some charity concerts, including the Bridge School Benefit concerts and a benefit for United Way of Canada Children's Charities at SkyDome in Toronto. Their next performance as a duo was in December 2003, at New York's Madison Square Garden during the Old Friends Tour. This concert was recorded, and released in December 2004 as the album Old Friends: Live on Stage.Simon & Garfunkel's Concert in Central Park raised around $51,000 for Central Park. Benefit concerts by other musicians followed, and helped to raise awareness of the park's state. With donations from the general public and with the help of wealthy benefactors, the park was restored during the 1980s and gained recognition as a major tourist attraction. As of 2011, donations still make up the majority of its budget. Today concerts and other benefits are regularly held on the Great Lawn.\n",
            "Response:\n",
            "1. What was the name of the duo that performed in Central Park?\n",
            "2. What was the name of the duo's reunion album?\n",
            "3. What was the name of the duo's first performance in Germany?\n",
            "4. What was the name of the duo's first performance in Germany?\n",
            "5. What was the name of the duo's first performance in Germany?\n",
            "6. What was the name of the duo's first performance in Germany?\n",
            "7. What was the name of the duo's first performance in Germany?\n",
            "8. What was the name of the duo's first performance in Germany?\n",
            "9. What was the name of the duo's first performance in Germany?\n",
            "10. What was the name of the duo's first performance in Germany?\n",
            "11. What was the name of the duo's first performance in Germany?\n",
            "12. What was the name of the duo's first performance in Germany?\n",
            "13. What was the name of the duo's first performance in Germany?\n",
            "14. What was the name of the duo's first performance in Germany?\n",
            "15. What was the name of the duo's first performance in Germany?\n",
            "16. What was the name of the\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Instruction:\n",
            "In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.\n",
            "Input:\n",
            "Passage: Dave Lizewski, bored after having retired from fighting crime as Kick-Ass, begins training with Hit-Girl Mindy Macready to become a real hero. Following the death of his father, Chris D'Amico accidentally kills his own mother by short-circuiting her tanning bed; Now in control of his father's criminal empire, Chris decides to become a supervillain named The Motherfucker, and assembles a gang of supervillains called the Toxic Mega Cunts with his aide Javier and has gained a cult following on Twitter, swearing vengeance on Kick-Ass.\n",
            "Mindy's guardian, Marcus, discovers she is still fighting crime and makes her promise to give it up. Dave resumes his life as Kick-Ass, joining the superhero team Justice Forever (which Dave had inspired), led by Colonel Stars and Stripes. Kick-Ass begins a sexual relationship with Night Bitch, one of the members after breaking up with Katie Deauxma. He and Marty, who is also on the team as Battle Guy, alienate their friend Todd from participating in their heroics. Mindy, attempting to lead a normal life, tries out for the dance team at school, and promptly asks a boy to take her on a date after declining to join Justice Forever. The date ends up as a cruel prank planned by bullies in her school, but Mindy gets her revenge the next day, resulting in her suspension from school.\n",
            "Response:\n",
            "1. What is the name of the superhero team that Dave joins?\n",
            "2. Who is Mindy's guardian?\n",
            "3. What is the name of the supervillain that Chris becomes?\n",
            "4. Who is the leader of the superhero team?\n",
            "5. What is the name of the dance team that Mindy tries out for?\n",
            "6. What happens to Mindy after she gets her revenge on the bullies?\n",
            "7. Who is the leader of the Toxic Mega Cunts?\n",
            "8. What is the name of the boy that Mindy asks to take her on a date?\n",
            "9. What is the name of the girl that Dave has a sexual relationship with?\n",
            "10. What is the name of the girl that Mindy is trying to lead a normal life with?\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Instruction:\n",
            "In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.\n",
            "Input:\n",
            "Passage: The number of victims is estimated at about 22,000, with a lower limit of confirmed dead of 21,768. According to Soviet documents declassified in 1990, 21,857 Polish internees and prisoners were executed after 3 April 1940: 14,552 prisoners of war (most or all of them from the three camps) and 7,305 prisoners in western parts of the Byelorussian and Ukrainian SSRs. Of them 4,421 were from Kozelsk, 3,820 from Starobelsk, 6,311 from Ostashkov, and 7,305 from Byelorussian and Ukrainian prisons. The head of the NKVD POW department, Maj. General P. K. Soprunenko, organized \"selections\" of Polish officers to be massacred at Katyn and elsewhere.Those who died at Katyn included soldiers (an admiral, two generals, 24 colonels, 79 lieutenant colonels, 258 majors, 654 captains, 17 naval captains, 85 privates, 3,420 non-commissioned officers, and seven chaplains), 200 pilots, government representatives and royalty (a prince, 43 officials), and civilians (three landowners, 131 refugees, 20 university professors, 300 physicians; several hundred lawyers, engineers, and teachers; and more than 100 writers and journalists). In all, the NKVD executed almost half the Polish officer corps. Altogether, during the massacre, the NKVD executed 14 Polish generals: Leon Billewicz (ret.), Bronisław Bohatyrewicz (ret.), Xawery Czernicki (admiral), Stanisław Haller (ret.), Aleksander Kowalewski (ret.), Henryk Minkiewicz (ret.), Kazimierz Orlik-Łukoski, Konstanty Plisowski (ret.), Rudolf Prich (killed in Lviv), Franciszek Sikorski (ret.), Leonard Skierski (ret.), Piotr Skuratowicz, Mieczysław Smorawiński, and Alojzy Wir-Konas (promoted posthumously). Not all of the executed were ethnic Poles, because the Second Polish Republic was a multiethnic state, and its officer corps included Belarusians, Ukrainians, and Jews. It is estimated about 8% of the Katyn massacre victims were Polish Jews. 395 prisoners were spared from the slaughter, among them Stanisław Swianiewicz and Józef Czapski. They were taken to the Yukhnov camp or  Pavlishtchev Bor and then to Gryazovets.\n",
            "Response: The number of victims is estimated at about 22,000, with a lower limit of confirmed dead of 21,768. According to Soviet documents declassified in 1990, 21,857 Polish internees and prisoners were executed after 3 April 1940: 14,552 prisoners of war (most or all of them from the three camps) and 7,305 prisoners in western parts of the Byelorussian and Ukrainian SSRs. Of them 4,421 were from Kozelsk, 3,820 from Starobelsk, 6,311 from Ostashkov, and 7,305 from Byelorussian and Ukrainian prisons. The head of the NKVD POW department, Maj. General P. K. Soprunenko, organized \"selections\" of Polish officers to be massacred at Katyn and elsewhere.Those who died at Katyn included soldiers (an admiral, two generals, 24 colonels, 79 lieutenant colonels, 258 majors, 654 captains, 17 naval captains, 85 privates, 3,420 non-commissioned officers, and seven chaplains). Altogether, the NKVD executed almost half the Polish officer corps. Altogether, during the massacre, the NKVD executed 14 Polish generals: Leon Billewicz (ret.), Bronisław Bohatyrewicz (ret.), Xawery Czernicki (admiral), Stanisław Haller (ret.), Aleksander Kowalewski (\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Instruction:\n",
            "In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.\n",
            "Input:\n",
            "Passage: The story opens with Lazlo missing, and Clam and Raj relating the tale up to this point.\n",
            "The first segment reveals how Raj and Clam meet. They meet a common enemy, Edward, who is the camp bully. Most of the other campers follow Edward's lead and after a scuffle, Lazlo makes his appearance. What follows builds Edward's growing resentfulness towards Lazlo, and Lumpus' dissatisfaction with the three new scouts' behavior. After choosing to name their cabin after the jelly bean, Lazlo builds a totem pole to decorate their new cabin, when Lazlo hears an animal in distress. Given Lazlo's nature, he goes to help it, while Clam and Raj choose not to accompany him.\n",
            "Lazlo finds a bear with a pinecone stuck in his nose, and pulls it out, earning the bear's gratefulness. The bear, now named Fluffy, follows Lazlo home and he hides it in his cabin. When Edward tells Lumpus that Lazlo has left camp, they both attempt to confront Lazlo, but are instead met by Fluffy. Protecting Lazlo, Fluffy attacks Edward and Lumpus. While everyone hides in Lumpus' cabin, Lazlo follows Fluffy out of the camp; when Lazlo's torn Bean Scout cap is later found in a gory, flesh-like mess the next day, the others assume that Lazlo was eaten by the bear.\n",
            "\n",
            "When Edward can find neither the bear nor Lazlo, he concocts a story about how he scared Fluffy off by his \"skills\" after witnessing the bear devour Lazlo, and demands the camp's respect. The next series of scenes deal with both Edward spinning a web of lies, and Lumpus trying to come to grips with Lazlo's disappearance, but only due to his fear of Commander Hoo-Ha, not over any concern for the missing scouts.\n",
            "Response:\n",
            "1. Who are the main characters in the story?\n",
            "2. What is the name of the bear that Lazlo helps?\n",
            "3. What does Edward do to try and scare Fluffy off?\n",
            "4. Why does Lumpus fear Commander Hoo-Ha?\n",
            "5. What is the name of the cabin that Lazlo and his friends choose to name after a type of candy?\n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Instruction:\n",
            "In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.\n",
            "Input:\n",
            "Passage: Rosa Moline is the dissatisfied, restless wife of Lewis, a small-town Wisconsin doctor. She is easily bored, uninterested in her husband's career or in anything to do with her current circumstances. She has long desired a glamorous life, in a world where she can have expensive things and meet truly interesting people. For over a year, she has been having an affair with Neil Latimer, a Chicago businessman who owns the local hunting lodge. Tired of waiting for him to ask her to marry and move to Chicago, Rosa extorts money from Lewis' patients - who often do not have cash but pay him in produce or in other non-financial ways - to finance her trip to the city.\n",
            "Lewis does not yet know about the affair, but he is used to his wife's unease with her life; he discovers the extortion and throws the cash at her, telling her that if she goes to Chicago, she need not come back. Rosa immediately leaves and fully expects Latimer to welcome her. However, he avoids her at first, then when he does meet her, he tells her he is love with another woman and intends to marry. Devastated, Rosa returns to Wisconsin, where Lewis forgives her. She soon becomes pregnant and, briefly, seems to be trying to settle down.\n",
            "During a party for Moose, the man who tends to the hunting lodge, Latimer shows up. He lets Rosa know that he has changed his mind and wants to marry her. Moose overhears the couple planning for her divorce and their marriage; the next day, as everyone is heading out on a hunting trip, Moose bets that her lover will not want the baby and advises Rosa that she had better tell Latimer about it, or he will. To prevent that eventuality, she shoots and kills Moose during the hunt. She is acquitted of this act by claiming she thought he was a deer.\n",
            "Response:\n",
            "1. What is the name of the woman who has an affair with Neil Latimer?\n",
            "2. What does Rosa do to finance her trip to Chicago?\n",
            "3. What does Lewis do when he discovers the extortion?\n",
            "4. What does Moose do when he overhears the couple planning for her divorce?\n",
            "5. What does Rosa do to prevent Moose from wanting the baby?\n",
            "Output: \n",
            "<|endoftext|>\n",
            "----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kmPm3TTn8r0c"
      },
      "source": [
        "## Fine-tuning with LoRA (15 + 5 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7jqVDxtV8r0c"
      },
      "source": [
        "In this phase, we're fine-tuning the phi-2 model on a question generation dataset. To begin, we need to format our dataset into the instruction tuning format. For this task, we can employ `DataCollatorForCompletionOnlyLM`. Look at the [example](https://huggingface.co/docs/trl/en/sft_trainer#train-on-completions-only) in the HuggingFace documentation and instantiate the data collator using the Alpaca template. (3 points)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q trl"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1vevslXc6MEY",
        "outputId": "150c2495-a8ee-4b51-e865-59775f6c8393"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m245.2/245.2 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.0/102.0 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def formatting_prompts_func(example):\n",
        "    output_texts = []\n",
        "    for i in range(len(example['definition'])):\n",
        "        text = f\"### Instruction:{example['definition'][i]}### Input:{example['inputs'][i]}### Response:{example['targets'][i]}\"\n",
        "        output_texts.append(text)\n",
        "    return output_texts"
      ],
      "metadata": {
        "id": "gd8Sc5lW7Ta5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "from datasets import load_dataset\n",
        "from trl import SFTTrainer, DataCollatorForCompletionOnlyLM\n",
        "\n",
        "instruction_template = \"### Instruction:\"\n",
        "response_template = \"### Response:\"\n",
        "collator = DataCollatorForCompletionOnlyLM(instruction_template=instruction_template, response_template=response_template, tokenizer=tokenizer)\n",
        "tokenizer.pad_token = tokenizer.eos_token"
      ],
      "metadata": {
        "id": "Vs6_YK4e-Oxh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "llB9U2wG8r0c"
      },
      "source": [
        "Refer to the HuggingFace [documentation](https://huggingface.co/docs/trl/en/sft_trainer#training-adapters) and instantiate the Lora config. (3 points)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q peft"
      ],
      "metadata": {
        "id": "KkMF8tMZHkU8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "297a2fae-32cd-4460-f016-d84cf659eebf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/199.1 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.4/199.1 kB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.1/199.1 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A2lzaYII8r0c"
      },
      "outputs": [],
      "source": [
        "from peft import LoraConfig, PeftConfig, get_peft_model\n",
        "\n",
        "peft_config = LoraConfig(\n",
        "    r=4,\n",
        "    lora_alpha=8,\n",
        "    lora_dropout=0.05,\n",
        "    bias=\"none\",\n",
        "    task_type=\"CAUSAL_LM\",\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = get_peft_model(model, peft_config)\n",
        "model.print_trainable_parameters()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NMs9eyzuuKSs",
        "outputId": "a3ba3521-d79e-47e5-a2c8-7b014191bd5c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "trainable params: 4,587,520 || all params: 2,784,271,360 || trainable%: 0.16476554928898884\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Enu-R_du8r0c"
      },
      "source": [
        "Configure other training arguments. [Here](https://huggingface.co/docs/transformers/v4.40.1/en/main_classes/trainer#transformers.TrainingArguments) is a list of available options. Consider using a small batch size to prevent CUDA out of memory errors. You can augment batch size artificially through gradient accumulation. Enabling gradient checkpointing can further save memory. You may train the model for tens of steps. (3 points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3fvfNjd8r0d"
      },
      "outputs": [],
      "source": [
        "from transformers import TrainingArguments\n",
        "lr = 1e-3\n",
        "batch_size = 1\n",
        "num_epochs = 5\n",
        "\n",
        "\n",
        "# define training arguments\n",
        "training_args = TrainingArguments(\n",
        "    output_dir='model_output/',\n",
        "    learning_rate=lr,\n",
        "    per_device_train_batch_size=batch_size,\n",
        "    per_device_eval_batch_size=batch_size,\n",
        "    num_train_epochs=num_epochs,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2u0TDhHy8r0d"
      },
      "source": [
        "Take a look at the HuggingFace [documentation](https://huggingface.co/docs/trl/en/sft_trainer) on supervised fine-tuning trainers. Instantiate the trainer and train the model ( Note that you should initialize the phi-2 model with `bfloat16` or `float16` dtype to avoid encountering Cuda out of memory errors.). (3 points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wejfVaOt8r0d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85,
          "referenced_widgets": [
            "ab5d7785310b4478acde258ade28cdb5",
            "36d8746835364efeb8b8728b4ef79192",
            "92302e928c5542f69eb217f6704285b5",
            "ee06008b781043a7a646a9522a9697f2",
            "43d0ddb27ddf40f5b99c20c98d2ef08b",
            "be1f01f0cff549d99412fd87edad2af1",
            "bdfea40c721e459798533b9205dc51ae",
            "af20e0a306634e9199a7702cfaff18f4",
            "318d6257c7204e0f90a68a464b28be3f",
            "14236a65e2ef4dffb3d4ab7390d52b6d",
            "9e998baf3970435cbe1a92229192e3c1"
          ]
        },
        "outputId": "6d2cee4c-785a-43c4-9933-5b20b4ab002a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:246: UserWarning: You didn't pass a `max_seq_length` argument to the SFTTrainer, this will default to 1024\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/20726 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ab5d7785310b4478acde258ade28cdb5"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Add \"from peft import PeftConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training\" to sft_trainer.py\n",
        "trainer = SFTTrainer(\n",
        "    model,\n",
        "    args=training_args,\n",
        "    train_dataset=custom_dataset['train'],\n",
        "    #peft_config=peft_config,\n",
        "    tokenizer=tokenizer,\n",
        "    packing=False,\n",
        "    formatting_func=formatting_prompts_func,\n",
        "    data_collator=collator,\n",
        "\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eISQ1xHG8r0e"
      },
      "source": [
        "Get the final model from the trainer and merge the Lora weights with it. Then, test the model with the inputs you gave to the pretrained model and compare the results. (3 points)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i1BvzLkj8r0e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "3c19a271-66bd-4f5d-87a2-7c9baa0343ed"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='1508' max='103630' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [  1508/103630 1:20:15 < 90:41:55, 0.31 it/s, Epoch 0.07/5]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>500</td>\n",
              "      <td>1.754200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1000</td>\n",
              "      <td>1.648900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1500</td>\n",
              "      <td>1.539600</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/utils.py:168: UserWarning: Could not find response key `### Response:` in the following instance: ### Instruction:In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.### Input:Passage: In 1951, with secret means of communications established, Wolters sent his first letter to Speer in five years.  He suggested that Speer move ahead with his memoirs.  In January 1953, Speer began work on his draft memoirs, and over the next year lengthy missives, sometimes written on tobacco wrappings or candy wrappers but most often on toilet paper, made their way to Wolters' office in Coesfeld.  Marion Riesser, who had continued as Wolters' secretary as he began private architectural practice, transcribed these notes into as many as forty closely typed pages per missive, and the draft totalled 1,100 pages.  Wolters objected that Speer called Hitler a criminal in the draft, and Speer presciently observed that he would likely lose a good many friends were the memoirs ever to be published.  Wolters had come to believe that reports of Nazi genocide were exaggerated by a factor of at least ten, that Hitler had not been given credit for the things he did right and that Germany had been harshly treated by the Allies.In the mid-1950s, Wolters quarrelled with Kempf who effectively dropped out of the network for a number of years, adding to the burden on Wolters and Riesser.  While Speer's pleas for his former associate and his former secretary to work together eventually brought about a healing of the breach, this was to some degree superficial as Kempf was aware that Wolters, even then, disagreed with Speer's opinions.  Wolters questioned Speer's readiness to accept responsibility for the Nazi regime's excesses and did not believe Speer had anything to apologise for, though the strength of his feelings on this point was kept from Speer—but not from Kempf and Riesser.Wolters was tireless in his efforts on behalf of Speer and his family to such an extent that his son, Fritz, later expressed feelings of neglect.  For Speer's fiftieth birthday in March 1955, Wolters gathered letters from many of Speer's friends and wartime associates, and saw to it that they made their way inside the walls of Spandau in time for Speer's birthday.  Wolters gave Speer's son Albert a summer job in his Düsseldorf office and a place to stay—in fact, Wolters hosted all six of the Speer children at one time or another.  By prior arrangement, he and Speer tried to get in touch with each other by telepathy one New Year's Eve—but both men fell asleep before midnight struck.Wolters constantly sought Speer's early release, which required the consent of the four occupying powers.  He engaged Düsseldorf attorney, and later state minister, Werner Schütz to lobby high German officials to get them to advocate Speer's release.  Schütz, who refused to ask for his expenses, was unsuccessful even though Lübke, West German President for the last seven years of Speer's incarceration, had worked under Speer.  Wolters had more success fending off denazification proceedings against Speer, collecting many affidavits in Speer's favor, including one from Tessenow whom Speer had shielded during the war.  Those proceedings dragged on for years, and were eventually ended by order of Willy Brandt, a strong supporter of Speer's.As early as 1956, Wolters feared the effect that disclosure of the GBI's eviction of Jewish tenants might have on Speer.  Wolters wrote to Kempf concerning the denazification proceedings, \"I am only anxious about the matter of the clearance of Jew-flats in Berlin.  That could be a bullseye.  And this is the point to which the defense should direct itself...\"  In 1964, Speer mentioned to Wolters in a letter that he would need the Chronik as a reference in revising his memoirs upon his release.  Wolter's response was to have Riesser retype the entire Chronik, leaving out any mention of the GBI's involvement in the persecution of the Jews, without telling Speer what he was doing.   This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_seq_length`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/utils.py:168: UserWarning: Could not find response key `### Response:` in the following instance: ### Instruction:In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.### Input:Passage: On October 4, 2011, Dylan's label, Egyptian Records, released an album of previously unheard Hank Williams songs, The Lost Notebooks of Hank Williams. Dylan had helped to curate this project, in which songs unfinished when Williams died in 1953 were completed and recorded by a variety of artists, including Dylan himself, his son Jakob Dylan, Levon Helm, Norah Jones, Jack White, and others.On May 29, 2012, U.S. President Barack Obama awarded Dylan a Presidential Medal of Freedom in the White House. At the ceremony, Obama praised Dylan's voice for its \"unique gravelly power that redefined not just what music sounded like but the message it carried and how it made people feel\".On September 11, 2012, Dylan released his 35th studio album, Tempest. The album features a tribute to John Lennon, \"Roll On John\", and the title track is a 14-minute song about the sinking of the Titanic. Reviewing Tempest for Rolling Stone, Will Hermes gave the album five out of five stars, writing: \"Lyrically, Dylan is at the top of his game, joking around, dropping wordplay and allegories that evade pat readings and quoting other folks' words like a freestyle rapper on fire.\" Hermes called Tempest \"one of [Dylan's] weirdest albums ever\", and opined, \"It may also be the single darkest record in Dylan's catalog.\" The critical aggregator website Metacritic awarded the album a score of 83 out of 100, indicating \"universal acclaim\".On August 27, 2013, Columbia Records released Volume 10 of Dylan's Bootleg Series, Another Self Portrait (1969–1971). The album contained 35 previously unreleased tracks, including alternative takes and demos from Dylan's 1969–1971 recording sessions during the making of the Self Portrait and New Morning albums. The box set also included a live recording of Dylan's performance with the Band at the Isle of Wight Festival in 1969. Another Self Portrait received favorable reviews, earning a score of 81 on the critical aggregator, Metacritic, indicating \"universal acclaim\". AllMusic critic Thom Jurek wrote, \"For fans, this is more than a curiosity, it's an indispensable addition to the catalog.\"On November 4, 2013, Columbia Records released Bob Dylan: Complete Album Collection: Vol. One, a boxed set containing all 35 of Dylan's studio albums, six albums of live recordings, and a collection, entitled Sidetracks, of singles, songs from films and non-album material. The box includes new album-by-album liner notes written by Clinton Heylin with an introduction by Bill Flanagan. On the same date, Columbia released a compilation, The Very Best of Bob Dylan, which is available in both single CD and double CD formats. To publicize the 35 album box set, an innovative video of the song \"Like a Rolling Stone\" was released on Dylan's website. The interactive video, created by director Vania Heymann, allowed viewers to switch between 16 simulated TV channels, all featuring characters who are lip-synching the lyrics of the 48-year-old song.On February 2, 2014, Dylan appeared in a commercial for the Chrysler 200 car which was screened during the 2014 Super Bowl American football game. At the end of the commercial, Dylan says: \"So let Germany brew your beer, let Switzerland make your watch, let Asia assemble your phone. We will build your car.\" Dylan's Super Bowl commercial generated controversy and op-ed pieces discussing the protectionist implications of his words, and whether the singer had \"sold out\" to corporate interests.In 2013 and 2014, auction house sales demonstrated the high cultural value attached to Dylan's mid-1960s work, and the record prices that collectors were willing to pay for artefacts from this period. In December 2013, the Fender Stratocaster which Dylan had played at the 1965 Newport Folk Festival fetched $965,000, the second highest price paid for a guitar. In June 2014, Dylan's hand-written lyrics of \"Like a Rolling Stone\", his 1965 hit single, fetched $2 million dollars at auction, a record for a popular music manuscript.On October 28, 2014, Simon & Schuster published a massive 960 page, thirteen and a half This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_seq_length`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/utils.py:168: UserWarning: Could not find response key `### Response:` in the following instance: ### Instruction:In this task, you're given passages that contain mentions of names of people, places, or things. Some of these mentions refer to the same person, place, or thing. Your job is to write questions that evaluate one's understanding of such references. Good questions are expected to link pronouns (she, her, him, his, their, etc.) or other mentions to people, places, or things to which they may refer. Do not ask questions that can be answered correctly without understanding the paragraph or having multiple answers. Avoid questions that do not link phrases referring to the same entity. For each of your questions, the answer should be one or more phrases in the paragraph, and it should be unambiguous.### Input:Passage: Hubert Humphrey was a Minnesotan who became a nationally prominent politician. He first ran for mayor of Minneapolis in 1943, but lost the election to the Republican candidate by just a few thousand votes.  As a Democrat, Humphrey recognized that his best chance for political success was to obtain the support of the Minnesota Farmer-Labor Party.  Other members of the Farmer-Labor Party had been considering the idea, as encouraged by Franklin D. Roosevelt, but the merger only became reality after Humphrey traveled to Washington, D.C. to discuss the issue.  Rather than simply absorbing the Farmer-Labor party, with its constituency of 200,000 voters, Humphrey suggested calling the party the Minnesota Democratic-Farmer-Labor Party.  He was elected mayor of Minneapolis in 1945, and one of his first actions was to propose an ordinance making racial discrimination by employers subject to a fine.  This ordinance was adopted in 1947, and although few fines were issued, the city's banks and department stores realized that public relations would improve by hiring blacks in increasing numbers. Humphrey delivered an impassioned speech at the 1948 Democratic National Convention encouraging the party to adopt a civil rights plank in their platform.  He was elected to the United States Senate in 1948 and was re-elected in 1954 and 1960.In the early 1960s, the topic of civil rights was coming to national prominence with sit-ins and marches organized by Martin Luther King Jr. and other black leaders.  In 1963, President John F. Kennedy sent a comprehensive civil rights bill to Congress, based largely on the ideas that Humphrey had been placing before the Senate for the previous fifteen years.  The bill passed the House in early 1964, but passage through the Senate was more difficult, due to southern segregationists who filibustered for 75 days.  Finally, in June 1964, the Civil Rights Act of 1964 became law.  Humphrey called this his greatest achievement. Lyndon B. Johnson recruited Humphrey for his running mate in the 1964 presidential election, and Humphrey became Vice President of the United States.  Governor Karl Rolvaag (DFL) appointed Walter Mondale to fill Humphrey's Senate seat.  Humphrey voiced doubts about the 1965 bombings of North Vietnam, which alienated him from Johnson.  He later defended Johnson's conduct of the Vietnam War, alienating himself from liberals, who were beginning to oppose the war around 1967.  In the 1968 presidential election, Humphrey ran against Richard Nixon and Independent candidate George Wallace and lost the popular vote by only 0.7%.  Humphrey later returned to the Senate in 1971 after Eugene McCarthy left office.Eugene McCarthy (DFL) served in the United States House of Representatives from 1949 through 1959 and in the United States Senate from 1959 through 1971.  He gained a reputation as an intellectual with strong convictions and integrity.  In 1967, he challenged Lyndon B. Johnson for the presidential nomination, running on an anti-war platform in contrast to Johnson's policies.  His strong support in the New Hampshire primary convinced Johnson to leave the race.Democrat Walter Mondale also achieved national prominence as Vice President under Jimmy Carter.  He served in the Senate from his appointment in 1964 until becoming Vice President in 1977.  In 1984, he ran for President of the United States, choosing Geraldine Ferraro as his running mate. The election proved to be a landslide victory for popular incumbent Ronald Reagan.  In 2002, just 11 days before election day, when incumbent Senator Paul Wellstone was killed in a plane crash, Mondale stepped into the race as the Democratic candidate for the U.S. Senate.  He lost the bid by two percentage points to the Republican, Norm Coleman.In 1970, Wendell Anderson (DFL) was elected as governor of Minnesota.  He spent two years working with a split Minnesota Legislature to enact a tax and school finance reform package that shifted the source of public education funding from local property taxes to state sales taxes, as well as adding excise taxes to liquor and cigarettes.  This achievement, dubbed the \"Minnesota Miracle\", was immensely popular.  In the next few years, the Legislature enacted other facets of their \"new liberalism\", including ratification of the Equal Rights Amendment, strong environmental laws, increases in workers This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_seq_length`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-9-3435b262f1ae>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    359\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trl_activate_neftune\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 361\u001b[0;31m         \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    362\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    363\u001b[0m         \u001b[0;31m# After training we make sure to retrieve back the original forward pass method\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/trainer.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)\u001b[0m\n\u001b[1;32m   1857\u001b[0m                 \u001b[0mhf_hub_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menable_progress_bars\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1858\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1859\u001b[0;31m             return inner_training_loop(\n\u001b[0m\u001b[1;32m   1860\u001b[0m                 \u001b[0margs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1861\u001b[0m                 \u001b[0mresume_from_checkpoint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mresume_from_checkpoint\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/trainer.py\u001b[0m in \u001b[0;36m_inner_training_loop\u001b[0;34m(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)\u001b[0m\n\u001b[1;32m   2201\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2202\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccelerator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccumulate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2203\u001b[0;31m                     \u001b[0mtr_loss_step\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2204\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2205\u001b[0m                 if (\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/trainer.py\u001b[0m in \u001b[0;36mtraining_step\u001b[0;34m(self, model, inputs)\u001b[0m\n\u001b[1;32m   3136\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3137\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss_context_manager\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3138\u001b[0;31m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_loss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3139\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3140\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_gpu\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/trainer.py\u001b[0m in \u001b[0;36mcompute_loss\u001b[0;34m(self, model, inputs, return_outputs)\u001b[0m\n\u001b[1;32m   3159\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3160\u001b[0m             \u001b[0mlabels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3161\u001b[0;31m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3162\u001b[0m         \u001b[0;31m# Save past state if it exists\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3163\u001b[0m         \u001b[0;31m# TODO: this needs to be fixed and made cleaner later.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1509\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1510\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1511\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1518\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1519\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1520\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1521\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1522\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/peft/peft_model.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, task_ids, **kwargs)\u001b[0m\n\u001b[1;32m   1127\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_enable_peft_forward_hooks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1128\u001b[0m                 \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mk\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mspecial_peft_forward_args\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1129\u001b[0;31m                 return self.base_model(\n\u001b[0m\u001b[1;32m   1130\u001b[0m                     \u001b[0minput_ids\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1131\u001b[0m                     \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1509\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1510\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1511\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1518\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1519\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1520\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1521\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1522\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/peft/tuners/tuners_utils.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    159\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mabstractmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py\u001b[0m in \u001b[0;36mnew_forward\u001b[0;34m(module, *args, **kwargs)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_hf_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/models/phi/modeling_phi.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m   1167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1168\u001b[0m         \u001b[0;31m# decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1169\u001b[0;31m         outputs = self.model(\n\u001b[0m\u001b[1;32m   1170\u001b[0m             \u001b[0minput_ids\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1171\u001b[0m             \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1509\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1510\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1511\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1518\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1519\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1520\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1521\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1522\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py\u001b[0m in \u001b[0;36mnew_forward\u001b[0;34m(module, *args, **kwargs)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_hf_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/models/phi/modeling_phi.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m   1046\u001b[0m                 )\n\u001b[1;32m   1047\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1048\u001b[0;31m                 layer_outputs = decoder_layer(\n\u001b[0m\u001b[1;32m   1049\u001b[0m                     \u001b[0mhidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1050\u001b[0m                     \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1509\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1510\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1511\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1518\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1519\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1520\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1521\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1522\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py\u001b[0m in \u001b[0;36mnew_forward\u001b[0;34m(module, *args, **kwargs)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_hf_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/models/phi/modeling_phi.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, attention_mask, position_ids, output_attentions, use_cache, past_key_value)\u001b[0m\n\u001b[1;32m    774\u001b[0m         \u001b[0mresidual\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    775\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 776\u001b[0;31m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minput_layernorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    777\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    778\u001b[0m         \u001b[0;31m# Self Attention\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1509\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compiled_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[misc]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1510\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1511\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1513\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_call_impl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1518\u001b[0m                 \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_pre_hooks\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0m_global_backward_hooks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1519\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1520\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1521\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1522\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py\u001b[0m in \u001b[0;36mnew_forward\u001b[0;34m(module, *args, **kwargs)\u001b[0m\n\u001b[1;32m    164\u001b[0m                 \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_old_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_hf_hook\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpost_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/modules/normalization.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    199\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 201\u001b[0;31m         return F.layer_norm(\n\u001b[0m\u001b[1;32m    202\u001b[0m             input, self.normalized_shape, self.weight, self.bias, self.eps)\n\u001b[1;32m    203\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mlayer_norm\u001b[0;34m(input, normalized_shape, weight, bias, eps)\u001b[0m\n\u001b[1;32m   2544\u001b[0m             \u001b[0mlayer_norm\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnormalized_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2545\u001b[0m         )\n\u001b[0;32m-> 2546\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayer_norm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnormalized_shape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbias\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackends\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcudnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menabled\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2547\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2548\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4gNFHCSD8r0e"
      },
      "source": [
        "the runtime got deleted before i could use the trained model, im sorry"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "15CE9LYX8r0f"
      },
      "source": [
        "We know that fine-tuning LLMs on Colab or Kaggle notebooks can be a bit tricky, and fine-tuning phi-2 for this task may require more GPU hours. The main point of this question is to teach you how to train your model using HuggingFace packages. So, it's okay if your model doesn't produce optimal results. However, there are 5 additional points available if it can generate better results :)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zivgc23P8r0f"
      },
      "source": [
        "# RAG (50 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nemPt6Zx0re"
      },
      "source": [
        "If you have any further questions or concerns, contact the TA via email: alisalemi@ut.ac.ir"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lNHplrzc8r0f"
      },
      "source": [
        "## Install Requirements"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "iX-Lm3M58r0f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8804ac84-c1a6-4464-e192-824454c4209b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.9/302.9 kB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.0/121.0 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.5/142.5 kB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.9/9.9 MB\u001b[0m \u001b[31m27.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m171.5/171.5 kB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.0/542.0 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m22.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m18.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m401.2/401.2 kB\u001b[0m \u001b[31m28.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m85.5/85.5 MB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.1/81.1 kB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m15.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.8/15.8 MB\u001b[0m \u001b[31m49.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "%pip install -q langchain\n",
        "%pip install -q ctransformers\n",
        "%pip install -q sentence_transformers\n",
        "%pip install -q datasets\n",
        "%pip install -q rank_bm25\n",
        "%pip install -q faiss-gpu\n",
        "%pip install -q arxiv\n",
        "%pip install -q pymupdf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OiB9LEjb8r0f"
      },
      "source": [
        "## 1. An Overview of LangChain (10 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DPaEdN5E8r0g"
      },
      "source": [
        "LangChain is an open-source framework designed to simplify the creation of applications using LLMs. It provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\n",
        "\n",
        "In this overview, we will provide a step-by-step guide on how to construct a basic application using LangChain. This application will fetch country-related information from a Large Language Model. For this purpose, we will be utilizing the LLaMa 2 chat 7B as our base model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "64-XOX3M8r0g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 274,
          "referenced_widgets": [
            "c56c8819b6a34432a9d38ea212d5ae2e",
            "5b043b489c484e9abf4e90235d73156d",
            "24e57c6d42134d1cbcada8ce45e9ab37",
            "1ad4c3a4e56045c49fa4ae2ae27d6a26",
            "7ae8f9197d704917909c059e7170f0b0",
            "d315fc26469b45fcae77942d52f282f7",
            "5b9939c9dd814d58b20a26eb5a17cf7d",
            "8ce362ae2c8b4a1aaf818b29a8cb7f43",
            "3a8ec0a77fd446b393bc7b5016a956e8",
            "6d61f95ee5944829afceca7214c591ec",
            "f2d59795db0843efb504c75986b46829",
            "ee41dcc7c4874e5f92e0b7c908ef6d67",
            "7e2c82fd4c0144aa8c49fc501cf81fe1",
            "7a61c3599f104d23be9baac628f328b0",
            "f3fd3da0d4354e53a626d8ab7c7fa8d9",
            "69f0bf0e1cbb4255940139337a7d9322",
            "dc632adc0613406893c8d445c20ca47f",
            "3080d39f1c184b1c9487d6b4ae4e6218",
            "687964f103ac45b08c8a94dd95345a5f",
            "9fe0d604bad94c549f9428734e5356e4",
            "c5949e898465400a821e7dabc2b08d79",
            "1ad2eb5135554bf5a4515eda1cd0a1a0",
            "9325232b43974a068bff52cdaa9583b9",
            "837ee2ac429041599fba645ed1a11ea2",
            "2f57faad5e3141d181f0f910887827b2",
            "fe6da49d5618403fbb6cd8e6eae9c9fd",
            "f289ed2050504f468198a90402963ac8",
            "572b07ef47a640efa467f06e1b03da3d",
            "4e5ae24bc8ff4168b5b3cb6d6f7cb9c1",
            "efc46e9edc8b40c38f581ef7b0465c65",
            "61ce422754414fbeb0b152f4cfe73c84",
            "78af3d45ad724d1c9e4195cb8a985c29",
            "d5210899bec147389fe8621c5a686205",
            "eee96c4b44a647d0851a0bf9361f2c52",
            "71116db573064aa5859db7b7acc8debc",
            "c86d9fbcbb9847399a7f93c0dc38b011",
            "852da029c24f4761bb8a506f1f82a0cb",
            "7e82d6be6dfd49ebb7b0f930ff93627f",
            "d256afe34cb34aac8fb99233c414aa4a",
            "ff040a214df34fc4951cce30741303bc",
            "e3550dfac171450c907a51cdfff8d403",
            "2c3dc36f449146ee977ce9829089c779",
            "8fca73019b6f41af8c17aa509ef15c79",
            "82569ad40e5e408dae31fa614892bb4a"
          ]
        },
        "outputId": "f7389adc-ad42-4f53-b5b3-e4d0d52a57bc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c56c8819b6a34432a9d38ea212d5ae2e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/29.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ee41dcc7c4874e5f92e0b7c908ef6d67"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9325232b43974a068bff52cdaa9583b9"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "llama-2-7b-chat.Q8_0.gguf:   0%|          | 0.00/7.16G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "eee96c4b44a647d0851a0bf9361f2c52"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from langchain_community.llms import CTransformers\n",
        "\n",
        "model = CTransformers(\n",
        "  model=\"TheBloke/Llama-2-7B-Chat-GGUF\",\n",
        "  model_file=\"llama-2-7b-chat.Q8_0.gguf\",\n",
        "  model_type=\"llama\",\n",
        "  config={\n",
        "    \"gpu_layers\": 50\n",
        "  }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lM2p2PkT8r0g"
      },
      "source": [
        "### 1.1 GGUF Format (3 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qT__Ll4U8r0g"
      },
      "source": [
        "Write a brief paragraph discussing the GGUF format and its benefits. Compare it with transformers library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DDo02QPe8r0g"
      },
      "source": [
        "there is a comparision between GGLM ( transformers library ) and GGUF formats\n",
        "source: https://medium.com/@sandyeep70/ggml-to-gguf-a-leap-in-language-model-file-formats-cd5d3a6058f9\n",
        "\n",
        "Difference and Preferences:\n",
        "GGML (Pros and Cons):\n",
        "\n",
        "Good:\n",
        "\n",
        "Tried to make a file format for GPT models early on.\n",
        "Made it easy to share models in one file.\n",
        "Models could run on regular computers, which was good for more people.\n",
        "Not So Good:\n",
        "\n",
        "Lack of flexibility hindered additional model information.\n",
        "Compatibility issues arose with new features.\n",
        "Manual adjustments are required for settings.\n",
        "GGUF (Advancements and Considerations):\n",
        "\n",
        "Improved:\n",
        "\n",
        "It’s like the next version after GGML, fixing its problems.\n",
        "You can easily add new things to it without causing problems for old models.\n",
        "It can work with different models, not just llama.cpp."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "joCaX8tY8r0g"
      },
      "source": [
        "### 1.2 Simple Chain (2 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sdkMm3Mz8r0h"
      },
      "source": [
        "Complete the next cell to create a simple chain that takes the name of a country as input and outputs its capital. To accomplish this, you should utilize the `HumanMessagePromptTemplate` and `AIMessagePromptTemplate` classes to formulate an effective prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gMEB4D4z8r0h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c13f6b6-8e5c-4b3c-bee7-1a4251bce058"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " Tehran.\n"
          ]
        }
      ],
      "source": [
        "from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate, AIMessagePromptTemplate, SystemMessagePromptTemplate\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "  HumanMessagePromptTemplate.from_template(\"what is the capital of {country}\"),\n",
        "  AIMessagePromptTemplate.from_template(\"The capital of {country} is\")\n",
        "])\n",
        "\n",
        "output_parser = StrOutputParser()\n",
        "\n",
        "simple_chain = prompt | model | output_parser\n",
        "\n",
        "answer = simple_chain.invoke({\"country\": \"Iran\"})\n",
        "\n",
        "print(answer)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hq8wDeKK8r0h"
      },
      "source": [
        "Write about the objectives behind the creation of `HumanMessagePromptTemplate` and `AIMessagePromptTemplate` classes. What they actually do? Write a brief description."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_V823OtC8r0h"
      },
      "source": [
        "we can set out prompts by \" HumanMessagePromptTemplate \" and control the output generated by llm with \" AIMessagePromptTemplate \""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Spwbg5wT8r0h"
      },
      "source": [
        "What is the purpose of adding an empty `AIMessagePromptTemplate` at the end of prompt? What is the consequences of omitting it?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u4ABlTkX8r0h"
      },
      "source": [
        "if there is no template provided, the model is free to generate more than just one or limited outputs and it can go wrong sometimes because of its tempreture and sampling methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MoN6LFaW8r0h"
      },
      "source": [
        "### 1.3 JSON Chain (5 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F56RIBhk8r0i"
      },
      "source": [
        "Now we want to improve the chain to extract data from the model response. Modify the existing prompt to request information about a country's name, population, and major cities in addition to the capital. Additionally, incorporate a `SystemMessagePromptTemplate` to ensure the model's response is structured in JSON format. Keep in mind that a distinct parser is required to parse the JSON output."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def post_proccess(answer):\n",
        "    index = answer.find('}')\n",
        "    if index != -1:\n",
        "        return answer[:index+1]\n",
        "    else:\n",
        "        print(\"The '}' character is not found in the string.\")\n",
        "        return 0"
      ],
      "metadata": {
        "id": "jhv3Aw3ovbkC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ztIkrpmk8r0i"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "system_message_prompt = SystemMessagePromptTemplate.from_template(\n",
        "    \"\"\"You are an assistant that provides information about countries.\n",
        "    you are given some questions and a template for answering question in json format\n",
        "    write the json data and replace '%GENERATED_TEXT' with the appropriate information for each field in JSON format.\n",
        "    for the output, just write the data once and dont add anything else.\"\"\"\n",
        ")\n",
        "\n",
        "human_message_prompt = HumanMessagePromptTemplate.from_template(\"1. What is the capital of {country}?\\n2. What is the population of {country}?\\n3. What are the important cities of {country}?\")\n",
        "\n",
        "ai_message_prompt = AIMessagePromptTemplate.from_template(\n",
        "    '''\n",
        "    Answer template in json format:\n",
        "    {{\"country\": {country},\"capital\": \"%GENERATED_TEXT\",\"population\": \"%GENERATED_TEXT\",\"cities\": [\"%GENERATED_TEXT\"]}}\n",
        "    Answer template after filling %GENERATED_TEXT:\n",
        "    '''\n",
        ")\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "  SystemMessagePromptTemplate,\n",
        "  human_message_prompt,\n",
        "  ai_message_prompt\n",
        "])\n",
        "output_parser = StrOutputParser()\n",
        "\n",
        "json_chain = prompt | model | output_parser\n",
        "\n",
        "answers = json_chain.batch([\n",
        "  {\"country\": \"Iran\"},\n",
        "  {\"country\": \"USA\"},\n",
        "  {\"country\": \"Japon\"},\n",
        "  {\"country\": \"Nigeria\"}\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for ans in answers:\n",
        "  if post_proccess(ans):\n",
        "    ans = post_proccess(ans)\n",
        "    ans = json.loads(ans)\n",
        "    print(f\"{ans['country']}:\")\n",
        "    print(f\"  capital: {ans['capital']}\")\n",
        "    print(f\"  population: {ans['population']}\")\n",
        "    print(f\"  important cities: {ans['cities']}\")\n",
        "    print(150*'-')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YVF1IPLvw6MB",
        "outputId": "a52e0a66-3889-445b-fa79-f70334321b5f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Iran:\n",
            "  capital: Tehran\n",
            "  population: 834100697\n",
            "  important cities: ['Tehran', 'Mashhad', 'Isfahan', 'Karaj', 'Qom']\n",
            "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
            "USA:\n",
            "  capital: Washington D.C.\n",
            "  population: 329460865 (estimated 2020)\n",
            "  important cities: ['New York City', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix', 'Philadelphia', 'San Antonio', 'San Diego', 'Dallas', 'San Jose', 'Austin']\n",
            "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
            "Japon:\n",
            "  capital: Tokyo is the capital of Japan.\n",
            "  population: The population of Japan is approximately 128 million people.\n",
            "  important cities: ['Tokyo', 'Osaka', 'Kobe', 'Nagoya', 'Sapporo']\n",
            "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
            "Nigeria:\n",
            "  capital: Abuja\n",
            "  population: 235890741\n",
            "  important cities: ['Lagos', 'Kano', 'Ibadan', 'Benin City', 'Port Harcourt', 'Enugu', 'Owerri']\n",
            "------------------------------------------------------------------------------------------------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HS8as30J8r0i"
      },
      "source": [
        "## 2. Different Types of Retrievers (15 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aM-prY48r0i"
      },
      "source": [
        "In this section, We use mini-bioasq dataset to evalute different types of retrivers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dmjlX1Go8r0i",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "da4a4140-8057-4e0d-92cf-b5b412c679e2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset({\n",
            "    features: ['passage', 'id'],\n",
            "    num_rows: 40221\n",
            "})\n",
            "Dataset({\n",
            "    features: ['question', 'answer', 'relevant_passage_ids', 'id'],\n",
            "    num_rows: 100\n",
            "})\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "from datasets import load_dataset\n",
        "\n",
        "corpus = load_dataset(\"rag-datasets/mini-bioasq\", \"text-corpus\", split=\"passages\")\n",
        "qa_dataset = load_dataset(\"rag-datasets/mini-bioasq\", \"question-answer-passages\", split=\"test[:100]\")\n",
        "\n",
        "qa_dataset = qa_dataset.map(lambda data: {\n",
        "  \"relevant_passage_ids\": json.loads(data[\"relevant_passage_ids\"])\n",
        "})\n",
        "\n",
        "print(corpus)\n",
        "print(qa_dataset)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bafD9h0d8r0i"
      },
      "source": [
        "### 2.1 Evaluate Retriever (4 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9OnYzmxC8r0j"
      },
      "source": [
        "To effectively compare various retrieval systems, we must define a metric. Complete the `evaluate_retriever` function to measure the accuracy of the retrieved documents. Consider the `relevant_passage_ids` column as the expected documents to be retrieved."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GiQ0ouC68r0j"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "import numpy as np\n",
        "\n",
        "def count_correct_labels(y_trues, y_preds):\n",
        "    correct_count = 0\n",
        "\n",
        "    for y_pred in y_preds:\n",
        "        if y_pred in y_trues:\n",
        "            correct_count += 1\n",
        "    return correct_count\n",
        "\n",
        "def evaluate_retriever(retriever):\n",
        "\n",
        "    accuracies = []\n",
        "\n",
        "    for index in range(len(qa_dataset)):\n",
        "        question = qa_dataset['question'][index]\n",
        "        y_trues = qa_dataset['relevant_passage_ids'][index]\n",
        "        y_preds = retriever(question)\n",
        "\n",
        "        correct_count = count_correct_labels(y_trues, y_preds)\n",
        "        accuracies.append(correct_count/len(y_preds))\n",
        "\n",
        "    mean_accuracy = np.mean(np.array(accuracies))\n",
        "    return mean_accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qvl1X3vl8r0j"
      },
      "source": [
        "### 2.2 TF-IDF Retriever (3 pt)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "class TfidfRetriever:\n",
        "    def __init__(self, corpus, passage_ids):\n",
        "        \"\"\"\n",
        "        Initializes the TF-IDF retriever with the given corpus and passage IDs.\n",
        "\n",
        "        Parameters:\n",
        "        - corpus (List[str]): A list of documents/passages.\n",
        "        - passage_ids (List[str]): A list of IDs corresponding to each document in the corpus.\n",
        "        \"\"\"\n",
        "        self.vectorizer = TfidfVectorizer()\n",
        "        self.tfidf_matrix = self.vectorizer.fit_transform(corpus)\n",
        "        self.passage_ids = passage_ids\n",
        "\n",
        "    def retrieve(self, query, top_k = 5):\n",
        "        \"\"\"\n",
        "        Retrieves the top-k most relevant passage IDs for the given query.\n",
        "\n",
        "        Parameters:\n",
        "        - query (str): The query string.\n",
        "        - top_k (int): The number of top documents to retrieve.\n",
        "\n",
        "        Returns:\n",
        "        - List[str]: A list of top-k relevant passage IDs.\n",
        "        \"\"\"\n",
        "        query_vec = self.vectorizer.transform([query])\n",
        "        similarity_scores = cosine_similarity(query_vec, self.tfidf_matrix).flatten()\n",
        "        top_indices = similarity_scores.argsort()[-top_k:][::-1]\n",
        "        return [self.passage_ids[i] for i in top_indices]\n"
      ],
      "metadata": {
        "id": "-qpzlL210yb9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tped6bPN8r0j"
      },
      "source": [
        "Create a TF-IDF retriever and configure it to returns the top 5 relevant documents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5R1AoV-e8r0j"
      },
      "outputs": [],
      "source": [
        "tfidf_retriever = TfidfRetriever(corpus['passage'], corpus['id'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0qnbeJnz8r0k"
      },
      "source": [
        "### 2.3 Semantic Retriever (5 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YgEphnmn8r0k"
      },
      "source": [
        "Semantic retrievers operate by retrieving documents through embeddings. These systems require an embedding model to convert documents into a vector space, and a vector database to find the closest documents to a query. Construct a semantic retriever that utilizes [`intfloat/e5-base`](https://huggingface.co/intfloat/e5-base) as the embedding model and FAISS for the vector database."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import faiss\n",
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "\n",
        "class SemanticRetriever:\n",
        "    def __init__(self, model_name, corpus, passage_ids, batch_size=16):\n",
        "        \"\"\"\n",
        "        Initializes the Semantic retriever with the given model, corpus, and passage IDs.\n",
        "\n",
        "        Parameters:\n",
        "        - model_name (str): The name of the embedding model to use.\n",
        "        - corpus (list): A list of documents/passages.\n",
        "        - passage_ids (list): A list of IDs corresponding to each document in the corpus.\n",
        "        - batch_size (int): The batch size to use for embedding the corpus.\n",
        "        \"\"\"\n",
        "        self.device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "        self.tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "        self.model = AutoModel.from_pretrained(model_name).to(self.device)\n",
        "        self.passage_ids = passage_ids\n",
        "        self.batch_size = batch_size\n",
        "        self.index = self._create_faiss_index(corpus)\n",
        "\n",
        "    def _embed(self, texts):\n",
        "        \"\"\"\n",
        "        Converts a list of texts to embeddings.\n",
        "\n",
        "        Parameters:\n",
        "        - texts (list): A list of texts to embed.\n",
        "\n",
        "        Returns:\n",
        "        - np.ndarray: The embeddings of the texts.\n",
        "        \"\"\"\n",
        "        all_embeddings = []\n",
        "        for i in range(0, len(texts), self.batch_size):\n",
        "            batch_texts = texts[i:i + self.batch_size]\n",
        "            inputs = self.tokenizer(batch_texts, padding=True, truncation=True, return_tensors=\"pt\").to(self.device)\n",
        "            with torch.no_grad():\n",
        "                outputs = self.model(**inputs)\n",
        "            embeddings = outputs.last_hidden_state.mean(dim=1).cpu().numpy()\n",
        "            all_embeddings.append(embeddings)\n",
        "        return np.vstack(all_embeddings)\n",
        "\n",
        "    def _create_faiss_index(self, corpus):\n",
        "        \"\"\"\n",
        "        Creates a FAISS index from the given corpus.\n",
        "\n",
        "        Parameters:\n",
        "        - corpus (list): A list of documents/passages.\n",
        "\n",
        "        Returns:\n",
        "        - faiss.IndexFlatL2: The FAISS index of the document embeddings.\n",
        "        \"\"\"\n",
        "        embeddings = self._embed(corpus)\n",
        "        index = faiss.IndexFlatL2(embeddings.shape[1])\n",
        "        index.add(embeddings)\n",
        "        return index\n",
        "\n",
        "    def retrieve(self, query, top_k=5):\n",
        "        \"\"\"\n",
        "        Retrieves the top-k most relevant passage IDs for the given query.\n",
        "\n",
        "        Parameters:\n",
        "        - query (str): The query string.\n",
        "        - top_k (int): The number of top documents to retrieve.\n",
        "\n",
        "        Returns:\n",
        "        - list: A list of top-k relevant passage IDs.\n",
        "        \"\"\"\n",
        "        query_embedding = self._embed([query])\n",
        "        distances, indices = self.index.search(query_embedding, top_k)\n",
        "        return [self.passage_ids[idx] for idx in indices[0]]"
      ],
      "metadata": {
        "id": "Yb4sFo3X5RuX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V6NzUEAs8r0k"
      },
      "outputs": [],
      "source": [
        "semantic_retriever = SemanticRetriever(\"intfloat/e5-base\", corpus['passage'], corpus['id'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mqfYAU2A8r0k"
      },
      "source": [
        "### 2.4 Compare Retrivers (3 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "onPepLjZ8r0k"
      },
      "source": [
        "Calculate the score for each retriever using `evaluate_retriever` you previously writed. In this question, which one outperforms the other? Illustrate a scenario for each retriver that it outperforms the other."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSDigftX8r0k"
      },
      "source": [
        "average accuracy for semantic retriever is better, because it uses dense embedding and its better for capturing word meanings than a spare embedding like tfidf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sc8qcfmA8r0k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2b071c3e-c2c7-4a15-c2e4-ade829bbbec9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF-IDF accuracy: 0.46\n",
            "semantic accuracy: 0.51\n"
          ]
        }
      ],
      "source": [
        "tfidf_acc = evaluate_retriever(tfidf_retriever.retrieve)\n",
        "semantic_acc = evaluate_retriever(semantic_retriever.retrieve)\n",
        "\n",
        "print(f\"TF-IDF accuracy: {tfidf_acc:.2f}\")\n",
        "print(f\"semantic accuracy: {semantic_acc:.2f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Rg5YG0m8r0l"
      },
      "source": [
        "## 3. RAG (25 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "By_xoYn_8r0l"
      },
      "source": [
        "In this section, you should use all the concepts you've learned until now to create a complete RAG chain."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uu-cD6OH8r0l"
      },
      "source": [
        "### 3.1 Load Documents (2 pt)\n",
        "\n",
        "Load [RAFT](https://arxiv.org/abs/2403.10131) and [DSPy](https://arxiv.org/abs/2401.12178) papers. You can use `ArxivLoader` to get documents from arXiv.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "n4mFsjna8r0l",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b764d9c4-3805-48e2-afab-45f92ee961a4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "RAFT Paper Content:\n",
            "[Document(page_content='RAFT: Adapting Language Model to Domain Specific RAG\\nTianjun Zhang Shishir G. Patil Naman Jain Sheng Shen\\nMatei Zaharia Ion Stoica Joseph E. Gonzalez\\ntianjunz@berkeley.edu, shishirpatil@berkeley.edu\\nUC Berkeley\\nAbstract\\nPretraining Large Language Models (LLMs) on\\nlarge corpora of textual data is now a standard\\nparadigm. When using these LLMs for many\\ndownstream applications, it is common to ad-\\nditionally bake in new knowledge (e.g., time-\\ncritical news, or private domain knowledge) into\\nthe pretrained model either through RAG-based-\\nprompting, or finetuning. However, the optimal\\nmethodology for the model to gain such new\\nknowledge remains an open question. In this pa-\\nper, we present Retrieval Augmented Fine Tun-\\ning (RAFT), a training recipe that improves the\\nmodel’s ability to answer questions in an \"open-\\nbook\" in-domain setting. In RAFT, given a ques-\\ntion, and a set of retrieved documents, we train\\nthe model to ignore those documents that don’t\\nhelp in answering the question, which we call,\\ndistractor documents. RAFT accomplishes this\\nby citing verbatim the right sequence from the rel-\\nevant document that would help answer the ques-\\ntion. This coupled with RAFT’s chain-of-thought-\\nstyle response helps improve the model’s ability\\nto reason. In domain specific RAG, RAFT consis-\\ntently improves the model’s performance across\\nPubMed, HotpotQA, and Gorilla datasets, present-\\ning a post-training recipe to improve pre-trained\\nLLMs to in-domain RAG. RAFT’s code and demo\\nare open-sourced at https://github.com/\\nShishirPatil/gorilla\\n1. Introduction\\nTrained on vast quantities of public data, Large Language\\nModels LLMs have achieved significant advances in a wide\\nrange of general knowledge reasoning tasks (Brown et al.,\\n2020; Wei et al., 2022).\\nHowever, increasingly LLMs are being employed in special-\\nized domains to support tasks ranging from code completion\\nfor specific software frameworks to question answering on\\nspecific document collections (e.g., legal or medical docu-\\nments). In these settings, general knowledge reasoning is\\nless critical but instead, the primary goal is to maximize ac-\\ncuracy based on a given set of documents. Indeed, adapting\\nLLMs to the specialized domains (e.g., recent news, enter-\\nprise private documents, or program resources constructed\\nafter the training cutoff) is essential to many emerging ap-\\nplications (Vu et al., 2023; Lazaridou et al., 2022) and is the\\nfocus of this work.\\nThis paper studies the following question – How to adapt\\npre-trained LLMs for Retrieval Augmented Generation\\n(RAG) in specialized domains?\\nWhen it comes to adapting LLMs to specialized domains,\\nwe consider the following two candidates: in-context learn-\\ning through Retrieval-Augmented Generation (RAG) and\\nsupervised fine-tuning. RAG-based methods allow the LLM\\nto reference the documents when answering questions. How-\\never, these methods fail to leverage the learning opportunity\\nafforded by the fixed domain setting and early access to\\nthe test documents. Alternatively, supervised fine-tuning\\noffers the opportunity to learn more general patterns in the\\ndocuments and better align to end tasks and user prefer-\\nences (Zhou et al., 2023a). However, existing fine-tuning\\nbased approaches either fail to leverage the documents at\\ntest time (don’t incorporate RAG) or fail to account for the\\nimperfections in the retrieval process during training.\\nWe can draw an analogy to an open-book exam. Exist-\\ning in-context retrieval methods are equivalent to taking an\\nopen-book exam without studying. Alternatively, existing\\nfine-tuning based approaches implement “studying\" by ei-\\nther directly “memorizing\" (Xiong et al., 2023) the input\\ndocuments or answering practice questions (Wang et al.,\\n2022) without referencing the documents. While these ap-\\nproaches leverage in-domain learning they fail to prepare\\nfor the open-book nature of test setting.\\nIn this paper, we study how to combine supervised\\nfine-tuning (SFT) with retrieval augmented generation\\n(RAG). We propose a novel adaptation strategy – Retrieval-\\nAugmented Fine Tuning (RAFT). RAFT specifically ad-\\ndresses the challenge of fine-tuning LLMs to incorporate\\ndomain knowledge while also improving in-domain RAG\\n1\\narXiv:2403.10131v1  [cs.CL]  15 Mar 2024\\nRAFT: Adapting Language Model to Domain Specific RAG\\n“Open book”\\nquery\\nanswer\\n“Closed book”\\nquery\\nanswer\\nBake in Knowledge \\nat Train Time\\nModel can use \\nExternal Docs at Test\\nRAFT (Proposed)\\nquery\\nanswer\\nTeach Model to\\n use External Docs at Test\\nFigure 1: How best to prepare for an Exam?(a) Fine-tuning based approaches implement \"studying\" by either directly\\n\"memorizing\" the input documents or answering practice QA without referencing the documents. (b) Alternatively, in-\\ncontext retrieval methods fail to leverage the learning opportunity afforded by the fixed domain and are equivalent to\\ntaking an open-book exam without studying. While these approaches leverage in-domain learning, they fail to prepare for\\nopen-book tests. In contrast, our approach (c) RAFT leverages fine-tuning with question-answer pairs while referencing the\\ndocuments in a simulated imperfect retrieval setting — thereby effectively preparing for the open-book exam setting.\\nperformance. RAFT aims to not only enable models to learn\\ndomain specific knowledge through fine-tuning, but also\\nto ensure robustness against inaccurate retrievals. This is\\nachieved by training the models to understand the dynamics\\nbetween the question posed (prompt), the domain specific\\ndocuments retrieved, and the appropriate answer. Going\\nback to our analogy, our approach is analogous to study-\\ning for an open-book exam by recognizing relevant, and\\nirrelevant retrieved documents.\\nIn RAFT, we train the model to answer the question (Q)\\nfrom Document(s) (D*) to generate an answer (A*), where\\nA* includes chain-of-thought (Wei et al., 2022; Anthropic,\\n2023), and in the presence of distractor documents (Dk).\\nWe explain the methodology in detail in Section 3 and ana-\\nlyze the sensitivity to the number of distractor documents\\n(k) at train- and test- time in Section 5. RAFT consis-\\ntently outperforms Supervised-finetuning both with- and\\nwithout- RAG across PubMed (Dernoncourt & Lee, 2017),\\nHotpotQA (Yang et al., 2018), and HuggingFace Hub, Torch\\nHub, and Tensorflow Hub Gorilla datasets (Patil et al., 2023),\\npresenting a novel, yet simple technique to improve pre-\\ntrained LLMs for in-domain RAG.\\n2. LLMs for Open-Book Exam\\nTo understand our goal better, we expand on our analogy be-\\ntween training an LLM in the real-world setting of preparing\\nfor an exam.\\nClosed-Book Exam\\nA closed book exam often refers to\\na scenario where the LLMs do not have access to any ad-\\nditional documents or references to answer the questions\\nduring the exam. For LLMs, this is equivalent to the sce-\\nnario, for example, in which the LLM is used as a chatbot.\\nIn this scenario, the LLM draws from the knowledge baked\\nin during pre-training and supervised finetuning to respond\\nto the prompt.\\nOpen Book Exam\\nIn contrast, we liken the open-book\\nexam setting to the scenario in which the LLM can re-\\nfer to external sources of information (e.g., a website or\\na book chapter). In such scenarios, typically, the LLM is\\npaired with a retriever which retrieves ‘k’ documents (or\\nspecific segments of the document) which are appended to\\nthe prompt. It is only through these documents retrieved that\\nthe LLM gains access to “new knowledge”. As a result, we\\nargue that the LLM’s performance in these settings, where it\\nis trained as a general-purpose LLM is largely dependent on\\nthe quality of the retriever and how accurately the retriever\\ncan identify the most relevant piece of information.\\nDomain Specific Open-Book Exam\\nIn this paper, we\\nfocused on a narrower but increasingly popular domain than\\nthe general open book exam, called the domain specific\\nopen book exam. In domain specific open book exams, we\\nknow apriori the domain in which the LLM will be tested\\n– used for inference. The LLM can respond to the prompt\\nusing use any and all information from this specific domain,\\nwhich it has been fine-tuned on. Examples of domain spe-\\ncific examples include enterprise documents, latest news,\\ncode repositories belonging to an organization, etc. In all\\nthese scenarios, the LLM will be used to respond to the\\nquestions, whose answers can be found within a collection\\nof documents (a small practical domain). The retrieval tech-\\nnique itself has little to no impact on the mechanism (though\\nit may impact the accuracy). This paper mainly studies this,\\ndomain specific open-book setting and how to adapt a pre-\\ntrained LLM to this specific domain, including how to make\\nit more robust to a varying number of retrieved documents\\nand distractors.\\n3. RAFT\\nIn this section, we present RAFT, a novel way of training\\nLLMs for domain specific open-book exams. We first intro-\\nduce the classical technique of supervised fine-tuning, fol-\\n2\\nRAFT: Adapting Language Model to Domain Specific RAG\\nFigure 2: Overview of our RAFT method. The top-left figure depicts our approach of adapting LLMs to reading solution\\nfrom a set of positive and negative documents in contrast to standard RAG setup where models are trained based on the\\nretriever outputs, which is a mixture of both memorization and reading. At test time, all methods follow the standard RAG\\nsetting, provided with a top-k retrieved documents in the context.\\nlowed by the key takeaways from our experiments. Then, we\\nintroduce RAFT , a modified version of general instruction\\ntuning. Lastly, we provide an overview of the experiments\\nto expect in the later sections.\\nSupervised Finetuning\\nConsider the supervised fine-tuning (SFT) setting for a\\nQuestion-Answer dataset. The formulation consists of the\\nDataset (D) from which a set of Question (Q) and corre-\\nsponding answer (A) pairs are derived or already available.\\nIn the classical SFT setting, the model is trained to improve\\nits ability to answer the questions based on its knowledge -\\nobtained either during pre-training, or during the SFT train-\\ning phase. The model so trained can also be used at test-time\\nwith the Retrieval Augmented Generation (RAG) setting,\\nwhere additional documents can be introduced in the prompt\\nto help the model answer the question. This can be repre-\\nsented as follows:\\n• Train: Q →A\\n• 0-shot Inference: Q →A\\n• RAG Inference: Q + D →A\\nRAFT\\nRetrieval Aware Fine-Tuning (RAFT), presents a novel\\nrecipe to prepare fine-tuning data to tailor the models for\\ndomain specific open-book settings, equivalent to in-domain\\nRAG In RAFT, we prepare the training data such that each\\ndata point contains a question (Q), a set of documents (Dk),\\nand a corresponding Chain-of-though style answer (A∗)\\ngenerated from one of the document (D∗). We differentiate\\nbetween two types of documents: ‘oracle’ documents (D∗)\\ni.e. the documents from which the answer to the question\\ncan be deduced, and ‘distractor’ documents (Di) that do not\\ncontain answer-relevant information. As an implementation\\ndetail, the ‘oracle’ document doesn’t need to be a single doc-\\nument, but can be more than one document, as is the case in\\nHotpotQA (Yang et al., 2018). Then, for P fraction of the\\nquestions (qi) in the dataset, we retain the oracle document\\n(d∗\\ni ) along with distractor documents (dk−1). For (1 −P)\\nfraction of the questions (qi) in the dataset, we include no\\noracle document and only include distractor documents (dk).\\nWe then fine-tune the language model using the standard\\nsupervised training (SFT) technique, training it to generate\\nanswers from the provided documents and questions. Fig. 2\\nillustrates the high-level design principal for RAFT .\\nWe demonstrate that our approach trains the model to per-\\nform better RAG on the set of documents it is trained on i.e.,\\nin-domain. By removing the oracle documents in some in-\\nstances, we are compelling the model to memorize answers\\ninstead of deriving them from the context. The training data\\nfor RAFT is as follows, and an example of training data can\\nbe seen in Fig. 3:\\n• P % of data: Q + D∗+ D2 + . . . + Dk →A∗\\n• (1 −P) % of data: Q + D1 + D2 + . . . + Dk →A∗\\nSubsequently, for the test scenario, the model is provided\\nwith the Q and top-k documents retrieved by the RAG\\npipeline. Note that RAFT is independent of the retriever\\nused.\\nA key factor in enhancing training quality is the genera-\\n3\\nRAFT: Adapting Language Model to Domain Specific RAG\\ntion of a reasoning process, such as Chain-of-Thought, to\\nexplain the provided answers.RAFT approach is similar:\\nwe demonstrate that creating a full reasoning chain and in\\naddition, clearly citing sources enhances the model’s accu-\\nracy in answering questions. In Fig. 3, we illustrate this\\nset-up. Generating the training data in this fashion, involves\\npresenting the model with a question, context, and verified\\nanswers, and then requesting it to form a reasoning chain\\nthat appropriately references the original context.\\nFor all the datasets in our experiments, we generate the\\nanswers using the technique described above. Note that the\\nGorilla APIBench dataset, already includes reasoning in\\nthe answers. We provide an example of the generation step\\nin Fig. 3, the detailed reasoning answer includes a citation\\nfrom the original context inside ##begin_quote## and\\n##end_quote## as well as the detailed explanation on\\nhow to reach the conclusion based on the citations. We\\ndemonstrate that adding detailed reasoning paragraphs helps\\nboost the model’s performance in our experiment section.\\n4. Evaluation\\nWe design our experiments to study how well RAFT per-\\nforms compared to various baselines. We find that the RAFT-\\n7B model (a finetuned version of LlaMA-2) is better at read-\\ning and extracting information from in-domain documents,\\nthan domain specific finetuned model, and general-purpose\\nmodel with RAG. As an ablation, we also demonstrate how\\nimportant it is for the model to learn with Chain-of-Thought\\nresponses. In this section, we will first introduce all the\\ndatasets we used in the experiments, then all the baseline\\nmodel/fine-tuning techniques that we benchmark against.\\n4.1. Datasets\\nIn our experiments, we use the following datasets to evaluate\\nour model and all baselines. We selected these datasets\\nto represent both popular and diverse domains including\\nWikipedia, Coding/API documents, and question-answering\\non medical documents.\\n• Natural Questions (NQ) (Kwiatkowski et al., 2019),\\nTrivia QA (Joshi et al., 2017) and HotpotQA (Yang\\net al., 2018) are the open-domain question-answers\\nbased on Wikipedia, mainly focused on common\\nknowledge (e.g., movies, sports, etc).\\n• HuggingFace, Torch Hub, and TensorFlow Hub are\\nfrom the APIBench (Patil et al., 2023) proposed in\\nthe Gorilla paper. These benchmarks measure how to\\ngenerate the correct, functional, and executable API\\ncalls based on the documentation.\\n• PubMed QA (Jin et al., 2019) is a question-answering\\ndataset tailored only for biomedical-research question-\\nanswering. It mainly focuses on answering medical\\nand biology questions based on a given set of docu-\\nments.\\nNote that the first category of dataset (NQ, Trivia QA, and\\nHotpotQA) is a relatively general domain whereas the latter\\ntwo domains are on very domain specific documents.\\nBaselines\\nWe consider the following baselines for our\\nexperiments:\\n• LlaMA2-7B-chat model with 0-shot prompting: this is\\nthe commonly used instruction-finetuned model for QA\\ntasks, where we provide clearly written instructions,\\nbut no reference documentation.\\n• LlaMA2-7B-chat model with RAG (Llama2 + RAG):\\nsimilar to the previous setting, except here we include\\nreference documents. This is a popular technique when\\ndealing with domain specific QA tasks.\\n• domain specific Finetuning with 0-shot prompting\\n(DSF): Performing standard supervised finetuning,\\nwithout documents in context. We find that it mostly\\nuseful to align the answering style of the model as well\\nas get familiar with the domain context.\\n• domain specific Finetuning with RAG (DSF + RAG):\\nEquip a domain specific finetuned model with external\\nknowledge using RAG. So, for the “knowledge” the\\nmodel does not know, it can still refer to the context.\\n4.2. Results\\nUsing the above datasets and baselines, we evaluate our\\nmodel RAFT and demonstrate the effectiveness of RAFT in\\nTab. 1. We see that RAFT consistently and significantly\\noutperforms the baselines. Compared with the base Llama-\\n2 instruction-tuned model, RAFT with RAG does much\\nbetter in terms of extracting information as well as being\\nrobust towards distractors. The gain can be as big as 35.25%\\non Hotpot QA and 76.35% on Torch Hub evaluation. Com-\\npared with DSF on the specific dataset, our model does bet-\\nter at relying on the provided context to solve the problem.\\nRAFT does much better on tasks like HotpotQA and Hug-\\ngingFace datasets (30.87% on HotpotQA and 31.41% on\\nHuggingFace). Note that for PubMed QA, since it is a binary\\nyes/no question, we don’t observe significant gains when we\\ncompare our model with DSF + RAG. Even compared with\\na much larger and better model GPT-3.5, RAFT demon-\\nstrates significant advantages.\\nOverall, the LLaMA-7B model, both with and without the\\nRAG, performs poorly due to its answering style not align-\\ning with the ground truth. By applying domain specific\\n4\\nRAFT: Adapting Language Model to Domain Specific RAG\\nQuestion: The Oberoi family is part of a hotel company that has a head office in what\\ncity?\\ncontext: [The Oberoi family is an Indian family that is famous for its involvement in\\nhotels, namely through The Oberoi Group]...[It is located in city center of Jakarta,\\nnear Mega Kuningan, adjacent to the sister JW Marriott Hotel. It is operated by The\\nRitz-Carlton Hotel Company. The complex has two towers that comprises a hotel and the\\nAirlangga Apartment respectively]...[The Oberoi Group is a hotel company with its head\\noffice in Delhi.]\\nInstruction: Given the question, context and answer above, provide a logical reasoning\\nfor that answer. Please use the format of: ##Reason: {reason} ##Answer: {answer}.\\n--------------------------------------------------------------------------------------\\nCoT Answer: ##Reason: The document ##begin_quote## The Oberoi family is an Indian\\nfamily that is famous for its involvement in hotels, namely through The Oberoi Group.\\n##end_quote## establishes that the Oberoi family is involved in the Oberoi group, and\\nthe document ##begin_quote## The Oberoi Group is a hotel company with its head office\\nin Delhi. ##end_quote## establishes the head office of The Oberoi Group. Therefore,\\nthe Oberoi family is part of a hotel company whose head office is in Delhi. ##Answer:\\nDelhi\\nFigure 3: RAFT prompt to help LLM evaluate its own generated reasoning and answers, contrasting them with the correct\\nreasoning and answers. The LLM is prompted to identify errors in its reasoning and extract key insights for improvement.\\nThis figure specifically represents the ‘GenerateExplanation‘ step in the RAFT algorithm (Section 3).\\nTable 1: RAFT improves RAG performance forall specialized domains: Across PubMed, HotpotQA, HuggingFace,\\nTorch Hub, and Tensorflow Hub, we see that domain specific Finetuning improves significantly of the performance of the\\nbase model, but RAFT consistently outperforms the existing domain specific finetuning method with or without RAG. This\\nsuggests the need to train the model with context. We compare our model with LLaMA finetuning receipes, and provide\\nGPT-3.5 for reference.\\nPubMed\\nHotpotQA\\nHuggingFace\\nTorch Hub\\nTensorFlow Hub\\nGPT-3.5 + RAG\\n71.60\\n41.5\\n29.08\\n60.21\\n65.59\\nLLaMA2-7B\\n56.5\\n0.54\\n0.22\\n0\\n0\\nLLaMA2-7B + RAG\\n58.8\\n0.03\\n26.43\\n08.60\\n43.06\\nDSF\\n59.7\\n6.38\\n61.06\\n84.94\\n86.56\\nDSF + RAG\\n71.6\\n4.41\\n42.59\\n82.80\\n60.29\\nRAFT (LLaMA2-7B)\\n73.30\\n35.28\\n74.00\\n84.95\\n86.86\\ntuning, we significantly enhance its performance. This pro-\\ncess enables the model to learn and adopt the appropriate\\nstyle of answering. However, introducing RAG to a domain-\\nspecifically fine-tuned (DSF) model doesn’t invariably lead\\nto better outcomes. This might indicate that the model lacks\\ntraining in context processing and extracting useful infor-\\nmation from it. By incorporating our method, RAFT , we\\ntrain the model not only to match its answering style with\\nthat required but also to improve its document processing\\ncapabilities. Consequently, our approach outperforms all\\nothers.\\n4.3. Effect of CoT\\nWe also conduct an analysis to evaluate the effectiveness of\\nthe Chain-of-Thought approach in enhancing the model’s\\nperformance. As indicated in Table 2, simply providing\\nthe answer to a question may not always be adequate. This\\napproach can lead to a rapid decrease in loss, resulting in\\nthe training process to diverge. Incorporating a reasoning\\nchain that not only guides the model to the answer but also\\nenriches the model’s understanding can improve the over-\\nall accuracy. In our experiments, integrating the Chain-of-\\nThought significantly enhances training robustness. We em-\\nploy GPT-4-1106 to generate our Chain-of-Thought prompts\\n5\\nRAFT: Adapting Language Model to Domain Specific RAG\\nand include an example of the prompt we used in Figure 3.\\n4.4. Qualitative Analysis\\nTo illustrate the potential advantages of RAFT over the\\ndomain-specifically fine-tuned (DSF) approach, we present\\na comparative example in Figure 4. This example qual-\\nitatively demonstrates a scenario where the DSF model\\nbecomes confused by a question asking for the identity of\\na screenwriter. Instead of providing the correct name, it\\nmistakenly cites one of the films written by the screenwriter.\\nIn contrast, the RAFT model accurately answers the ques-\\ntion. This discrepancy suggests that training a model solely\\nwith question-answer pairs may impair its ability to derive\\nrelevant context from provided documents. The comparison\\nunderscores the importance of incorporating both standard\\ninstructional tuning and context comprehension into the\\ntraining dataset to preserve and enhance the model’s ability\\nto process text effectively.\\n4.5. Should we train the LLM always with the oracle\\ncontext for RAG?\\nIn our exploration of whether large language models\\n(LLMs) should always be trained with the oracle context for\\nRetrieval-Augmented Generation (RAG), we address a key\\nquestion: what proportion (p%) of the training data should\\ninclude oracle documents? Intuitively, one might assume\\nthat for effective training in reading and extracting informa-\\ntion from context (e.g., RAG tasks), the oracle document\\nshould always be included during training (P = 100%). How-\\never, our findings challenge this assumption: incorporating\\na portion of the training data without the oracle document\\nin the context (P = 80%) appears to enhance the model’s\\nperformance on RAG tasks.\\nFig. 5 presents our investigation into the hyperparameter\\nP%, which represents the percentage of training instances\\nthat should include oracle documents. Our analysis reveals\\nthat the optimal proportion varies across datasets, with fig-\\nures ranging from 40%, 60%, and 100%. This indicates\\nthat training your LLM without the correct corresponding\\ncontext at times can be beneficial for the downstream task of\\nanswering questions related to the documents. In our train-\\ning setup, we include four distractor documents alongside\\nthe oracle document, and at test time, we maintain this for-\\nmat by providing the oracle document with four distractors.\\nOur findings suggest that, for domain specific RAG tasks,\\nincluding a certain percentage of training data without the\\noracle documents in the context proves to be advantageous.\\n5. RAFT Generalizes to Top-K RAG\\nAfter demonstrating the performance of RAFT on vari-\\nous benchmarks, we now study another important problem:\\nHow does the number of distractor documents in RAFT af-\\nfect the model’s performance when augmented with top-k\\nretriever augmented generation (RAG) result during the eval-\\nuation? Previous research has highlighted the vulnerability\\nof LLMs to irrelevant text (see studies (Shi et al., 2023a;\\nWeston & Sukhbaatar, 2023; Liu et al., 2023b)). This issue\\nis particularly critical for LLMs + RAG since top-k RAG\\nis frequently employed at test time to ensure high recall.\\nSuch a scenario necessitates the model to have the ability to\\ndiscern and disregard irrelevant content, focusing solely on\\npertinent information.\\n5.1. Making Model Robust to top-K RAG\\nTo tackle the challenge of enhancing large language mod-\\nels’ (LLMs) ability to sift through irrelevant text within the\\nretrieval pipeline, our analysis revealed that training solely\\nwith oracle (highly relevant) documents can inadvertently di-\\nminish the model’s ability to discern and disregard irrelevant\\ninformation. To address this, our algorithm, RAFT , adopts\\na strategy that integrates oracle documents with a mix of\\nirrelevant ones. This methodology prompts us to investigate\\nthe ideal fraction of negative (irrelevant) documents to in-\\ncorporate throughout the training process and to assess how\\nwell this training approach adapts to different volumes of\\ndocuments encountered by the Retrieval-Augmented Gen-\\neration (RAG) during the test phase. Our aim is to refine\\nthe balance between relevant and irrelevant information to\\nstrengthen the model’s efficiency in identifying and utilizing\\npertinent content. Notice that Sec 4.5 looked at what P% of\\ntraining data should include distractors, while in this section,\\nwe study test-time scenarios.\\nTraining with Negative Documents To enhance the robust-\\nness of large language models (LLMs) against irrelevant\\ntext in retrieved documents, we adopted a finetuning ap-\\nproach that incorporates both golden (highly relevant) docu-\\nments and distractor (irrelevant) documents. The model was\\ntrained with varying numbers of distractor documents, but\\nconsistently evaluated using the top-k documents obtained\\nfrom the retriever - not to be confused with p.\\nOur findings, detailed in Fig. 6, reveal that finetuning with\\nonly the oracle document frequently results in inferior per-\\nformance compared to configurations that include a greater\\nnumber of distractor documents. As we can see in the fig-\\nure, the better performance for Natural Questions is training\\nwith D∗+ 3D and it is D∗+ 1D documents with Hotpot\\nQA. This insight has been particularly beneficial for our\\nalgorithm, RAFT . In our experiments, we typically employ\\na training setup consisting of one oracle document alongside\\nfour distractor documents. This approach strikes a balance,\\nensuring the model is not overwhelmed by distractors while\\nstill gaining the ability to effectively discern and prioritize\\nrelevant information.\\n6\\nRAFT: Adapting Language Model to Domain Specific RAG\\nTable 2: Ablation on Chain-of-Thought: The numbers of RAFT and RAFT without CoT. Results on various datasets\\nshow that adding CoT can significantly improve the performance of the finetuned model. With a gain of 9.66% and 14.93%\\non the Hotpot QA and HuggingFace datasets respectively.\\nPubMed\\nHotpotQA\\nHuggingFace\\nTorch Hub\\nTensorFlow Hub\\nRAFT w.o CoT\\n68.30\\n25.62\\n59.07\\n86.56\\n83.21\\nRAFT\\n73.30\\n35.28\\n74.00\\n84.95\\n86.86\\nHotPot QA\\nQuestion: What screenwriter with credits for “Evolution”[0/1879]e a film starring Nicolas Cage and Téa Leoni?\\nDocuments: . . . David Weissman is a screenwriter and director. His film credits include “The Family Man” (2000),\\n“Evolution” (2001), and “When in Rome” (2010).\\n. . .\\nThe Family Man is a 2000 American romantic comedy-drama film directed by Brett Ratner, written by David Diamond\\nand David Weissman, and starring Nicolas Cage and Téa Leoni. ...\\n——————————————————————————————————————————————–\\nRAFT: ##Reason: The screenwriter with credits for the film “Evolution,” starring Nicolas Cage and Téa Leoni, is\\nDavid Weissman. This information is provided in the reference documents which mention David Weissman as a\\nscreenwriter with film credits including “The Family Man” (2000), “Evolution” (2001), and “When in Rome” (2010).\\nTherefore, the screenwriter for “Evolution” is David Weissman. ##Answer: David Weissman ✓\\nDSF: The Family Man ✗\\nFigure 4: Comparison of RAFT and DSF: We prompt RAFT and DSF fine-tuned models on the HotpotQA dataset. We\\ncan see that the DSF model extracts the wrong information from the context. For the question, who is the screenwriter, it\\nresponds with a film name. RAFT manages to get the result correctly .\\nGeneralization to a variable number of test-time docu-\\nments. We extended our research to examine the impact of\\ndifferent quantities of test-time documents on the model’s\\nperformance. Specifically, our experiments focused on as-\\nsessing how models, trained with varying numbers of dis-\\ntractor documents, respond to changes in the number of\\ndocuments presented at test time.\\nThe results, illustrated in Fig. 6, confirm that the inclusion\\nof distractor documents during training indeed makes the\\nmodel more resilient to fluctuations in the number of docu-\\nments encountered during testing. This ability to maintain\\nconsistent performance despite variations in test-time doc-\\nument numbers further validates the robustness of our ap-\\nproach, RAFT . This finding underscores the importance of\\na well-calibrated training environment to prepare the model\\nfor a range of scenarios it may encounter in real-world ap-\\nplications.\\n6. Related Works\\nRetrieval-Augmented Language Models RAG enhances\\nlanguage models by integrating a retrieval module that\\nsources relevant information from external knowledge bases,\\nsignificantly improving performance across various NLP\\ntasks, including language modeling (Guu et al., 2020;\\nBorgeaud et al., 2022; Khandelwal et al., 2019; Shi et al.,\\n2023d; Lin et al., 2023b; Shi et al., 2023c; Asai et al., 2023;\\nXu et al., 2023; Wang et al., 2023) and open-domain ques-\\ntion answering (Izacard et al., 2023; Lewis et al., 2020). This\\nintegration follows a “retrieve-and-read\" paradigm where\\nthe retrieval module provides additional context from exter-\\nnal sources, which the LM then uses to generate the final out-\\nput. The retrieval process involves using the input as a query\\nto fetch documents, which the LM incorporates for final pre-\\ndictions. For instance, Atlas (Izacard et al., 2023) fine-tunes\\nT5 models with the retriever, treating documents as latent\\nvariables, while RETRO (Borgeaud et al., 2022) modifies\\nthe decoder-only architecture to include retrieved texts and\\nconducts pre-training from scratch. kNN-LM (Khandelwal\\net al., 2019) interpolates between the LM’s next token distri-\\nbution and distributions computed from retrieved tokens at\\ninference. (Shi et al., 2023d; Ram et al., 2023) assume black-\\nbox access to an LM and combine it with either off-the-shelf\\nor fine-tuned retriever.\\nMemorization A key question around large neural language\\nmodels is whether they truly “understand” text (Feldman,\\n7\\nRAFT: Adapting Language Model to Domain Specific RAG\\n0\\n20\\n40\\n60\\n80\\n100\\nP % Golden Retrieved Context at Training\\n0.25\\n0.30\\n0.35\\n0.40\\n0.45\\nFinal Accuracy\\nT\\nest Domain: NQ\\n0\\n20\\n40\\n60\\n80\\n100\\n% Golden Retrieved Context at Training\\n0.50\\n0.55\\n0.60\\n0.65\\nFinal Accuracy\\nT\\nest Domain: TQA\\n0\\n20\\n40\\n60\\n80\\n100\\nP % Golden Retrieved Context at Training\\n0.40\\n0.45\\n0.50\\n0.55\\n0.60\\nFinal Accuracy\\nT\\nest Domain: Hotpot QA\\nFigure 5: How many golden documents to involve? We study the hyperparameter P% which indicates what fraction of\\nthe training data contains the oracle document(s) in its context. Results on NQ, TQA and HotpotQA suggest that mixing a\\nfraction of data that does not have the oracle document in its context is helpful for in-domain RAG.\\n2\\n4\\n6\\n8\\n10\\n# T\\nest Documents (T\\nop-k)\\n0.22\\n0.24\\n0.26\\n0.28\\n0.30\\n0.32\\nFinal Accuracy\\nNatural Questions\\nTrain D*\\nTrain D* + 1D\\nTrain D* + 2D\\nTrain D* + 3D\\n2\\n4\\n6\\n8\\n10\\n# T\\nest Documents (T\\nop-k)\\n0.125\\n0.150\\n0.175\\n0.200\\n0.225\\n0.250\\nFinal Accuracy\\nHotpot QA\\nTrain D*\\nTrain D* + 1D\\nTrain D* + 2D\\nTrain D* + 3D\\nFigure 6: Test-Time Documents Varying: We study how robust RAFT is to varying numbers of test-time documents\\nthat a retriever might provide. In NQ, we find that training with 4 documents leads to the best performance, but training\\nwith 2 documents is optimal for HotpotQA. However, across both datasets, training with all datasets consisting of oracle\\ndocuments hurts performance.\\n2020; Power et al., 2022) or simply rely on surface pattern\\nmemorization (Carlini et al., 2019; Tänzer et al., 2022).\\n(Feldman, 2020; Carlini et al., 2019; 2022) develop method-\\nologies to quantify the extent of memorization in neural\\nmodels. (Brown et al., 2020; Power et al., 2022; Liu et al.,\\n2022b) further explored how memorization impacts the mod-\\nels’ generalization capabilities. Recently, a seminal work\\nby (Carlini et al., 2021; Shi et al., 2023b) demonstrated\\nthe ability of language models to memorize and regurgitate\\ntraining data, raising significant privacy concerns (Kandpal\\net al., 2022; Pan et al., 2020).\\nFinetuning of LLMs Recent years have seen rapid progress\\nin developing large-scale language models (LLMs) (Brown\\net al., 2020; OpenAI, 2023; Workshop et al., 2022; Tou-\\nvron et al., 2023;?; Anil et al., 2023). To adapt these foun-\\ndation models to downstream tasks, fine-tuning (Mishra\\net al., 2021; Sanh et al., 2021; Chung et al., 2022; Muen-\\nnighoff et al., 2023; Zhou et al., 2023b; Lin et al., 2023b;\\nJi et al., 2024) has become a prevalent approach. Tradi-\\ntional supervised fine-tuning may be limited by the cost\\nand compute required for adapating LLMs. Addressing\\nthese challenges, research in the realm of parameter-efficient\\nfine-tuning (Houlsby et al., 2019), such as Prompt Tuning\\n(Lester et al., 2021), Prefix-Tuning (Li & Liang, 2021),\\nP-Tuning (Liu et al., 2022a) and Low-Rank based fine-\\ntuning (Hu et al., 2021), has gained traction. These methods\\nenable LLMs to acquire domain-specific knowledge and\\nadapt to specialized tasks such as question answering, sum-\\nmarization, and dialogue generation. Another branch of\\nfinetuning is through RLHF (Ouyang et al., 2022; Rafailov\\net al., 2023; Liu et al., 2023a; Zhang et al., 2023), which\\nadopts RL to align LLM’s preference with human.\\nFinetuning for RAG More recently, several papers have\\nbeen exploring the idea of finetuning a pretrained LLM to\\nbe better at RAG tasks (Lin et al., 2023a; Wang et al., 2023;\\nXu et al., 2023; Liu et al., 2024). These works focus on con-\\nstructing a combination of finetuning dataset for RAG and\\ntrain a model to perform well on these tasks. In particular, in\\n8\\nRAFT: Adapting Language Model to Domain Specific RAG\\ntheir settings, at test time, the domain or documents can be\\ndifferent than the training time; whereas our paper studies a\\nslightly opposite scenario where we only care about testing\\nthe LLM on the same set of documents.\\n7. Conclusion\\nRAFT is a training strategy designed to enhance the model’s\\nperformance in answering questions within a specific do-\\nmain, in \"open-book\" settings. This technique demonstrates\\na fine-tuning recipe for LLMs for question-answering tasks\\nbased on a selected collection of documents. We have pin-\\npointed several crucial design decisions, such as training\\nthe model alongside distractor documents, organizing the\\ndataset so a portion lacks oracle documents in their con-\\ntext, and formulating answers in a chain-of-thought manner\\nwith direct quotations from the relevant text. Our evalua-\\ntions on PubMed, HotpotQA, and Gorilla API Bench un-\\nderline RAFT’s significant potential. Looking forward, we\\nanticipate that in-domain Retrieval-Augmented Generation\\n(RAG) will continue to gain interest within both industrial\\nand academic spheres. Unlike general-RAG, our work ad-\\ndresses practical scenarios where LLMs are tasked with an-\\nswering questions using domain-specific knowledge. Align-\\ning with current trends, our findings suggest that smaller,\\nfine-tuned models are capable of performing comparably\\nwell in domain-specific question-answering tasks, in con-\\ntrast to their generic LLM counterparts.\\nReferences\\nAnil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin,\\nD., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen,\\nZ., et al.\\nPalm 2 technical report.\\narXiv preprint\\narXiv:2305.10403, 2023.\\nAnthropic. Prompt engineering for claude’s long context\\nwindow. 2023.\\nAsai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-\\nrag: Learning to retrieve, generate, and critique through\\nself-reflection. arXiv preprint arXiv:2310.11511, 2023.\\nBorgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford,\\nE., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B.,\\nDamoc, B., Clark, A., et al. Improving language models\\nby retrieving from trillions of tokens. In International\\nconference on machine learning, pp. 2206–2240. PMLR,\\n2022.\\nBrown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,\\nDhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,\\nAskell, A., et al. Language models are few-shot learners.\\nAdvances in neural information processing systems, 33:\\n1877–1901, 2020.\\nCarlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song,\\nD. The secret sharer: Evaluating and testing unintended\\nmemorization in neural networks. In 28th USENIX Se-\\ncurity Symposium (USENIX Security 19), pp. 267–284,\\n2019.\\nCarlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-\\nVoss, A., Lee, K., Roberts, A., Brown, T., Song, D.,\\nErlingsson, U., et al. Extracting training data from large\\nlanguage models. In 30th USENIX Security Symposium\\n(USENIX Security 21), pp. 2633–2650, 2021.\\nCarlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F.,\\nand Zhang, C. Quantifying memorization across neural\\nlanguage models. In The Eleventh International Confer-\\nence on Learning Representations, 2022.\\nChung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y.,\\nFedus, W., Li, Y., Wang, X., Dehghani, M., Brahma,\\nS., et al. Scaling instruction-finetuned language models.\\narXiv preprint arXiv:2210.11416, 2022.\\nDernoncourt, F. and Lee, J. Y. Pubmed 200k rct: a dataset\\nfor sequential sentence classification in medical abstracts.\\narXiv preprint arXiv:1710.06071, 2017.\\nFeldman, V. Does learning require memorization? a short\\ntale about a long tail. In Proceedings of the 52nd Annual\\nACM SIGACT Symposium on Theory of Computing, pp.\\n954–959, 2020.\\nGuu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.\\nRetrieval augmented language model pre-training. In\\nInternational conference on machine learning, pp. 3929–\\n3938. PMLR, 2020.\\nHoulsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B.,\\nDe Laroussilhe, Q., Gesmundo, A., Attariyan, M., and\\nGelly, S. Parameter-efficient transfer learning for nlp.\\nIn International Conference on Machine Learning, pp.\\n2790–2799. PMLR, 2019.\\nHu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang,\\nS., Wang, L., and Chen, W. Lora: Low-rank adaptation of\\nlarge language models. arXiv preprint arXiv:2106.09685,\\n2021.\\nIzacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni,\\nF., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and\\nGrave, E. Atlas: Few-shot learning with retrieval aug-\\nmented language models. Journal of Machine Learning\\nResearch, 24(251):1–43, 2023. URL http://jmlr.\\norg/papers/v24/23-0037.html.\\nJi, C. C.-J., Mao, H., Yan, F., Shishir G. Patil, T. Z., Stoica,\\nI., and Gonzalez, J. E. Gorilla openfunctions v2. 2024.\\n9\\nRAFT: Adapting Language Model to Domain Specific RAG\\nJin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X.\\nPubmedqa: A dataset for biomedical research question\\nanswering. arXiv preprint arXiv:1909.06146, 2019.\\nJoshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.\\nTriviaqa: A large scale distantly supervised challenge\\ndataset for reading comprehension.\\narXiv preprint\\narXiv:1705.03551, 2017.\\nKandpal, N., Wallace, E., and Raffel, C. Deduplicating\\ntraining data mitigates privacy risks in language models.\\nIn International Conference on Machine Learning, pp.\\n10697–10707. PMLR, 2022.\\nKhandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L.,\\nand Lewis, M. Generalization through memorization:\\nNearest neighbor language models.\\narXiv preprint\\narXiv:1911.00172, 2019.\\nKwiatkowski, T., Palomaki, J., Redfield, O., Collins, M.,\\nParikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin,\\nJ., Lee, K., et al. Natural questions: a benchmark for ques-\\ntion answering research. Transactions of the Association\\nfor Computational Linguistics, 7:453–466, 2019.\\nLazaridou, A., Gribovskaya, E., Stokowiec, W., and Grig-\\norev, N. Internet-augmented language models through\\nfew-shot prompting for open-domain question answering.\\narXiv preprint arXiv:2203.05115, 2022.\\nLester, B., Al-Rfou, R., and Constant, N. The power of scale\\nfor parameter-efficient prompt tuning. arXiv preprint\\narXiv:2104.08691, 2021.\\nLewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V.,\\nGoyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel,\\nT., et al. Retrieval-augmented generation for knowledge-\\nintensive nlp tasks. Advances in Neural Information Pro-\\ncessing Systems, 33:9459–9474, 2020.\\nLi, X. L. and Liang, P. Prefix-tuning: Optimizing continuous\\nprompts for generation. arXiv preprint arXiv:2101.00190,\\n2021.\\nLin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James,\\nR., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M.,\\net al. Ra-dit: Retrieval-augmented dual instruction tuning.\\narXiv preprint arXiv:2310.01352, 2023a.\\nLin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James,\\nR., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M.,\\net al. Ra-dit: Retrieval-augmented dual instruction tuning.\\narXiv preprint arXiv:2310.01352, 2023b.\\nLiu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight\\naligns language models with feedback. arXiv preprint\\narXiv:2302.02676, 3, 2023a.\\nLiu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilac-\\nqua, M., Petroni, F., and Liang, P. Lost in the middle:\\nHow language models use long contexts. arXiv preprint\\narXiv:2307.03172, 2023b.\\nLiu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang,\\nJ. P-tuning: Prompt tuning can be comparable to fine-\\ntuning across scales and tasks. In Proceedings of the 60th\\nAnnual Meeting of the Association for Computational\\nLinguistics (Volume 2: Short Papers), pp. 61–68, 2022a.\\nLiu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark,\\nM., and Williams, M. Towards understanding grokking:\\nAn effective theory of representation learning. Advances\\nin Neural Information Processing Systems, 35:34651–\\n34663, 2022b.\\nLiu, Z., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catan-\\nzaro, B. Chatqa: Building gpt-4 level conversational qa\\nmodels. arXiv preprint arXiv:2401.10225, 2024.\\nMishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross-\\ntask generalization via natural language crowdsourcing\\ninstructions. arXiv preprint arXiv:2104.08773, 2021.\\nMuennighoff, N., Wang, T., Sutawika, L., Roberts, A., Bi-\\nderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong,\\nZ. X., Schoelkopf, H., Tang, X., Radev, D., Aji, A. F., Al-\\nmubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff,\\nE., and Raffel, C. Crosslingual generalization through\\nmultitask finetuning. In Rogers, A., Boyd-Graber, J., and\\nOkazaki, N. (eds.), Proceedings of the 61st Annual Meet-\\ning of the Association for Computational Linguistics (Vol-\\nume 1: Long Papers), pp. 15991–16111, Toronto, Canada,\\nJuly 2023. Association for Computational Linguistics.\\ndoi: 10.18653/v1/2023.acl-long.891.\\nURL https:\\n//aclanthology.org/2023.acl-long.891.\\nOpenAI. Gpt-4 technical report, 2023.\\nOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,\\nMishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,\\net al. Training language models to follow instructions\\nwith human feedback. Advances in Neural Information\\nProcessing Systems, 35:27730–27744, 2022.\\nPan, X., Zhang, M., Ji, S., and Yang, M. Privacy risks of\\ngeneral-purpose language models. In 2020 IEEE Sympo-\\nsium on Security and Privacy (SP), pp. 1314–1331. IEEE,\\n2020.\\nPatil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla:\\nLarge language model connected with massive apis. arXiv\\npreprint arXiv:2305.15334, 2023.\\nPower, A., Burda, Y., Edwards, H., Babuschkin, I., and\\nMisra, V.\\nGrokking: Generalization beyond overfit-\\nting on small algorithmic datasets.\\narXiv preprint\\narXiv:2201.02177, 2022.\\n10\\nRAFT: Adapting Language Model to Domain Specific RAG\\nRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning,\\nC. D., and Finn, C. Direct preference optimization: Your\\nlanguage model is secretly a reward model. arXiv preprint\\narXiv:2305.18290, 2023.\\nRam, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua,\\nA., Leyton-Brown, K., and Shoham, Y.\\nIn-context\\nretrieval-augmented language models. arXiv preprint\\narXiv:2302.00083, 2023.\\nSanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L.,\\nAlyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja,\\nA., et al. Multitask prompted training enables zero-shot\\ntask generalization. arXiv preprint arXiv:2110.08207,\\n2021.\\nShi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi,\\nE. H., Schärli, N., and Zhou, D. Large language models\\ncan be easily distracted by irrelevant context. In Inter-\\nnational Conference on Machine Learning, pp. 31210–\\n31227. PMLR, 2023a.\\nShi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins,\\nT., Chen, D., and Zettlemoyer, L. Detecting pretrain-\\ning data from large language models. arXiv preprint\\narXiv:2310.16789, 2023b.\\nShi, W., Min, S., Lomeli, M., Zhou, C., Li, M., Lin, V.,\\nSmith, N. A., Zettlemoyer, L., Yih, S., and Lewis, M.\\nIn-context pretraining: Language modeling beyond doc-\\nument boundaries.\\narXiv preprint arXiv:2310.10638,\\n2023c.\\nShi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis,\\nM., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-\\naugmented black-box language models. arXiv preprint\\narXiv:2301.12652, 2023d.\\nTänzer, M., Ruder, S., and Rei, M. Memorisation versus\\ngeneralisation in pre-trained language models. In Pro-\\nceedings of the 60th Annual Meeting of the Association\\nfor Computational Linguistics (Volume 1: Long Papers),\\npp. 7564–7578, 2022.\\nTouvron, H., Martin, L., Stone, K., Albert, P., Almahairi,\\nA., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,\\nBhosale, S., et al. Llama 2: Open foundation and fine-\\ntuned chat models. arXiv preprint arXiv:2307.09288,\\n2023.\\nVu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J.,\\nTar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. Freshllms:\\nRefreshing large language models with search engine\\naugmentation. arXiv preprint arXiv:2310.03214, 2023.\\nWang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi,\\nM., and Catanzaro, B.\\nInstructretro: Instruction tun-\\ning post retrieval-augmented pretraining. arXiv preprint\\narXiv:2310.07713, 2023.\\nWang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A.,\\nKhashabi, D., and Hajishirzi, H. Self-instruct: Aligning\\nlanguage models with self-generated instructions. arXiv\\npreprint arXiv:2212.10560, 2022.\\nWei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F.,\\nChi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought\\nprompting elicits reasoning in large language models.\\nAdvances in Neural Information Processing Systems, 35:\\n24824–24837, 2022.\\nWeston, J. and Sukhbaatar, S.\\nSystem 2 attention\\n(is something you might need too).\\narXiv preprint\\narXiv:2311.11829, 2023.\\nWorkshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick, E.,\\nIli´\\nc, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon,\\nF., et al. Bloom: A 176b-parameter open-access multilin-\\ngual language model. arXiv preprint arXiv:2211.05100,\\n2022.\\nXiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P.,\\nHou, R., Martin, L., Rungta, R., Sankararaman, K. A.,\\nOguz, B., et al. Effective long-context scaling of founda-\\ntion models. arXiv preprint arXiv:2309.16039, 2023.\\nXu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Sub-\\nramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro,\\nB. Retrieval meets long context large language models.\\narXiv preprint arXiv:2310.03025, 2023.\\nYang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W.,\\nSalakhutdinov, R., and Manning, C. D. Hotpotqa: A\\ndataset for diverse, explainable multi-hop question an-\\nswering. arXiv preprint arXiv:1809.09600, 2018.\\nZhang, T., Liu, F., Wong, J., Abbeel, P., and Gonzalez, J. E.\\nThe wisdom of hindsight makes language models better\\ninstruction followers. arXiv preprint arXiv:2302.05206,\\n2023.\\nZhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X.,\\nEfrat, A., Yu, P., Yu, L., et al. Lima: Less is more for\\nalignment. arXiv preprint arXiv:2305.11206, 2023a.\\nZhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X.,\\nEfrat, A., Yu, P., Yu, L., et al. Lima: Less is more for\\nalignment. arXiv preprint arXiv:2305.11206, 2023b.\\n11\\n', metadata={'Published': '2024-03-15', 'Title': 'RAFT: Adapting Language Model to Domain Specific RAG', 'Authors': 'Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, Joseph E. Gonzalez', 'Summary': 'Pretraining Large Language Models (LLMs) on large corpora of textual data is\\nnow a standard paradigm. When using these LLMs for many downstream\\napplications, it is common to additionally bake in new knowledge (e.g.,\\ntime-critical news, or private domain knowledge) into the pretrained model\\neither through RAG-based-prompting, or fine-tuning. However, the optimal\\nmethodology for the model to gain such new knowledge remains an open question.\\nIn this paper, we present Retrieval Augmented FineTuning (RAFT), a training\\nrecipe that improves the model\\'s ability to answer questions in a \"open-book\"\\nin-domain settings. In RAFT, given a question, and a set of retrieved\\ndocuments, we train the model to ignore those documents that don\\'t help in\\nanswering the question, which we call, distractor documents. RAFT accomplishes\\nthis by citing verbatim the right sequence from the relevant document that\\nwould help answer the question. This coupled with RAFT\\'s chain-of-thought-style\\nresponse helps improve the model\\'s ability to reason. In domain-specific RAG,\\nRAFT consistently improves the model\\'s performance across PubMed, HotpotQA, and\\nGorilla datasets, presenting a post-training recipe to improve pre-trained LLMs\\nto in-domain RAG. RAFT\\'s code and demo are open-sourced at\\ngithub.com/ShishirPatil/gorilla.'})]\n",
            "\n",
            "DSPy Paper Content:\n",
            "[Document(page_content='In-Context Learning for Extreme Multi-Label Classification\\nKarel D’Oosterlinck1,2,∗, Omar Khattab2, François Remy1,\\nThomas Demeester1, Chris Develder1, Christopher Potts2\\n1Ghent University – imec\\n2Stanford University\\n∗karel.doosterlinck@ugent.be\\nAbstract\\nMulti-label classification problems with thou-\\nsands of classes are hard to solve with in-\\ncontext learning alone, as language models\\n(LMs) might lack prior knowledge about the\\nprecise classes or how to assign them, and\\nit is generally infeasible to demonstrate ev-\\nery class in a prompt. We propose a general\\nprogram, Infer–Retrieve–Rank, that defines\\nmulti-step interactions between LMs and re-\\ntrievers to efficiently tackle such problems. We\\nimplement this program using the DSPy pro-\\ngramming model, which specifies in-context\\nsystems in a declarative manner, and use DSPy\\noptimizers to tune it towards specific datasets\\nby bootstrapping only tens of few-shot exam-\\nples. Our primary extreme classification pro-\\ngram, optimized separately for each task, at-\\ntains state-of-the-art results across three bench-\\nmarks (HOUSE, TECH, TECHWOLF). We\\napply the same program to a benchmark with\\nvastly different characteristics and attain com-\\npetitive performance as well (BioDEX). Unlike\\nprior work, our proposed solution requires no\\nfinetuning, is easily applicable to new tasks, al-\\nleviates prompt engineering, and requires only\\ntens of labeled examples. Our code is public at\\nhttps://github.com/KarelDO/xmc.dspy.\\n1\\nIntroduction\\nExtreme multi-label classification (XMC) tasks are\\nhard to solve with in-context learning alone. Lan-\\nguage models (LMs) might lack prior knowledge\\nabout the precise classes, and the sheer number\\nof available classes—often upwards of 10,000—\\ngenerally means it is infeasible even to demon-\\nstrate every class in a prompt. To deal with this,\\nsome recent efforts make multiple LM calls at in-\\nference time (Zhu and Zamani, 2023), while others\\nprompt LMs to generate synthetic data for finetun-\\ning (Decorte et al., 2023; Clavié and Soulié, 2023).\\nThese methods can be configured to work well,\\nbut they all have manual “knobs” like prompts and\\nother hyperparameters that make applying them to\\nnew datasets, metrics, or LMs challenging.\\nIn this paper, we show that simple programs\\nwritten using the DSPy programming model (Khat-\\ntab et al., 2023) support powerful, highly gen-\\neral approaches to XMC tasks.\\nDSPy allows\\nus to separately specify the modular program\\nof our method and how it should be optimized\\ntowards different datasets.\\nWe propose a sim-\\nple in-context program for XMC tasks called\\nInfer–Retrieve–Rank (IReRa, Figure 1, Step 1).\\nFirst, an LM processes the input document and\\nguesses a set of applicable terms (Infer). Then, a\\nretriever relates each predicted term to the actual\\nlabel space (Retrieve). Finally, an LM is used to\\nrerank retrieved labels (Rank). Crucially, we use a\\nfrozen retriever and frozen LMs. The key insights\\nof Infer–Retrieve–Rank is that such a frozen re-\\ntriever can be made much more flexible if the LM\\nlearns in-context how to predict relevant queries\\nand interpret the retrieved results.\\nThe underlying LMs, retriever, and prompts are\\nconsidered hyperparameters of the IReRa program,\\nwhich can be tuned automatically or easily con-\\nfigured. Using only 10 unlabeled training inputs,\\nand ≈50 labeled validation examples, we bootstrap\\na few-shot prompt for our two LM components,\\nusing a zero-shot teacher LM with a minimal seed-\\nprompt (Figure 1, Step 2). DSPy’s compilation ab-\\nstraction handles this nicely; it takes the program\\nlogic we’ve already defined, instantiates it with a\\nteacher LM, processes the unlabeled training ex-\\namples, generates zero-shot labels for each of the\\nprogram steps, and picks the best labels to put in a\\nfew-shot prompt based on validation performance.\\nBecause our program consists of two in-context\\nmodules, we propose to bootstrap them sequen-\\ntially (Figure 1, Step 3).\\nIn our experiments, we instantiate the Infer\\nmodule with a Llama-2-7b-chat model (Touvron\\net al., 2023), while the teacher model used for boot-\\narXiv:2401.12178v1  [cs.CL]  22 Jan 2024\\nFigure 1: We propose Infer-Retrieve-Rank, an efficient in-context learning program for multi-label classification\\nwith an extreme amount of classes (≥10,000). Given an input, a first in-context learning module predicts queries\\nwhich route to a frozen retriever. The retrieved documents are re-ranked by a second in-context module (Step 1).\\nGiven a minimal prompt (Step 2), a zero-shot Teacher LM bootstraps demonstrations to optimize the few-shot\\nStudent LM (Step 3). Optimization using ≈50 labeled inputs can yield state-of-the-art results, using only ≈20\\nTeacher and ≈1,500 Student calls. The (optimization) logic is expressed using the DSPy programming model.\\nstrapping is GPT-3.5. The Rank module is instanti-\\nated and bootstrapped both by a GPT-4 model.\\nAdapting Infer–Retrieve–Rank to a new\\ndataset can be as simple as (i) writing a new mini-\\nmal zero-shot prompt, (ii) configuring which LMs\\nto use, and (iii) running the optimization proce-\\ndure. We optimize this program separately towards\\n4 XMC datasets: one dataset involving the extract-\\ning and coding of adverse drug events expressed in\\nbiomedical literature (BioDEX; D’Oosterlinck et al.\\n2023) and three datasets involving the labeling of\\njob vacancy snippets with the required competen-\\ncies they express (HOUSE, TECH, TECHWOLF;\\nZhang et al. 2022; Decorte et al. 2022, 2023). Our\\nprogram attains state-of-the-art results on the job\\nvacancy datasets, and gets meaningful traction on\\nthe harder biomedical task—without finetuning,\\nwithout prompt engineering, and by using only ≈50\\nlabeled examples. We find that the optimization is\\na consistent driver of performance across tasks.\\n2\\nRelated Work\\nThe canonical way of tackling an extreme classi-\\nfication problem involves either finetuning a spe-\\ncialized retriever over the label space or finetuning\\none binary classifier per class (Decorte et al., 2022,\\n2023; Clavié and Soulié, 2023). These methods\\nrequire a lot of data, since every one of the many\\nclasses requires at least a few labeled examples.\\nTo avoid manual data labeling, researchers use dis-\\ntant supervision (Decorte et al., 2022), bootstrap\\nsynthetic data using LLMs (Decorte et al., 2023;\\nClavié and Soulié, 2023; De Raedt et al., 2023), or\\nfinetune retrievers on adjacent problems where data\\nis available (Remy et al., 2022). At inference time,\\nan additional LLM call can be used to rerank a list\\nof generated candidate labels to further increase\\nperformance (Clavié and Soulié, 2023).\\nZhu and Zamani (2023) use multiple GPT-3.5\\ncalls combined with retrieval at inference-time to\\nbootstrap a synthetic prompt per input, infer labels,\\nand rerank them. While they do not use any fine-\\ntuning, they require many LLM and retrieval calls\\nper input. They evaluate on two recommendation\\ntasks, where inputs and outputs are the same type\\nof documents. Bhatia et al. (2016) formulate many\\nrecommendation tasks under the XMC setting. In-\\nstead, we consider XMC tasks where inputs and\\noutputs are not of similar shape, and more inference\\nor information extraction is needed.\\nOur Infer–Retrieve–Rank program does not\\nrely on finetuning or many LLM calls per in-\\nput, making it efficient to develop and deploy.\\nInfer–Retrieve–Rank can achieve state-of-the-\\nart performance using only ≈50 labeled examples.\\nUnlike prior work, our program logic is defined\\nin a modular and declarative manner, and can be\\nseamlessly applied to different benchmarks given a\\nminimal seed-prompt. Optimization happens auto-\\nmatically and can resolve in as little as ten minutes.\\nThe choice of LMs and retrievers can be configured,\\nensuring relevance when stronger components be-\\ncome available. Finally, we write at most one seed-\\nprompt per task and in-context module, and let\\noptimization—not iterative prompt engineering—\\ntake care of increasing performance.\\n3\\nInfer–Retrieve–Rank\\nThe program for Infer–Retrieve–Rank is given\\nin Code Snippet 1, with minor alterations for\\nbrevity. First, an LM is used to predict queries\\ngiven the input (Infer). The retriever outputs a\\nranking over all labels based on maximum cosine\\nembedding similarity with the queries (Retrieve).\\nThe top labels are reranked by another LM (Rank).\\n1 class InferRetrieveRank(dspy.Module):\\n2\\ndef __init__(self , infer_sig , rank_sig , retr):\\n3\\n# Initialize LM modules with Signatures\\n4\\nself.infer = dspy.ChainOfThought(infer_sig)\\n5\\nself.rank = dspy.ChainOfThrought(rank_sig)\\n6\\nself.retrieve = retr\\n7\\n8\\ndef forward(self , text: str) -> Prediction:\\n9\\n# Predict with LM\\n10\\npreds = self.infer(text).completions.labels\\n11\\n12\\n# Parse LM output\\n13\\npreds = extract_labels_from_strings(preds)\\n14\\n15\\n# Use LM outputs to retrieve labels\\n16\\nlabels = self.retrieve(preds)\\n17\\n18\\n# Use LM to rerank labels\\n19\\nlabels = self.rank(text , labels)\\n20\\n21\\nreturn dspy.Prediction(labels=labels)\\nCode Snippet 1: DSPy code for Infer–Retrieve–Rank\\nwith minor alterations for brevity.\\nNot all labels occur with equal frequency. Fine-\\ntuned methods can implicitly learn this bias given\\nenough data. If available, we propose to use the\\nprior probability pi for the i-th label to reweigh\\nthe retrieval similarity si to account for this. The\\nupdated scores ˜\\nsi as defined below are the output\\nof the Retrieve module. A is a scalar hyperparam-\\neter controlling the strength of the prior update:\\n˜\\nsi = si ⋅log10(A ⋅pi + 10)\\n4\\nSeed-prompts\\nTo apply Infer–Retrieve–Rank to a dataset, a\\nminimal seed-prompt needs to define the behav-\\nior of each in-context module. Code Snippet 2\\ncontains the prompt for the Infer module on the\\nBioDEX dataset, neatly organized using the DSPy\\nSignature abstraction. This seed-prompt defines\\na task description in the docstring, and input and\\noutput fields with descriptions and formatting in-\\nformation. The Signature serves as skeleton for\\nboth zero- and few-shot prompts.\\n1 class BiodexInferSignature(dspy.Signature):\\n2\\n\"\"\" Given a snippet from a medical article ,\\nidentify the adverse drug reactions\\naffecting the patient. Always return\\nreactions.\"\"\"\\n3\\n4\\ntext = dspy.InputField(prefix=\"Article:\")\\n5\\noutput = dspy.OutputField(\\n6\\nprefix=\"Reactions:\",\\n7\\ndesc=\"list of comma -separated adverse\\ndrug reactions\"\\n8\\n)\\nCode Snippet 2: DSPy Signature for BioDEX Infer.\\nThe prompt for the BioDEX Rank module is\\ngiven in Code Snippet 3. We use the same prompts\\nfor all three job vacancy datasets, they are given\\nin given in Code Snippets 4 and 5 for the Infer\\nand Rank modules respectively.\\nNote how the\\nprompts share most of their content: adapting\\nInfer–Retrieve–Rank can be as easy as concisely\\ndescribing the input and output fields.\\n1 class BiodexRankSignature(dspy.Signature):\\n2\\n\"\"\" Given a snippet from a medical article ,\\npick the 10 most applicable adverse\\nreactions from the options that are directly\\nexpressed in the snippet.\"\"\"\\n3\\n4\\ntext = dspy.InputField(prefix=\"Article:\")\\n5\\noptions = dspy.InputField(\\n6\\nprefix=\"Options:\",\\n7\\ndesc=\"List of comma -separated options to\\nchoose from\"\\n8\\n)\\n9\\noutput = dspy.OutputField(\\n10\\nprefix=\"Reactions:\",\\n11\\ndesc=\"list of comma -separated adverse\\ndrug reactions\"\\n12\\n)\\nCode Snippet 3: DSPy Signature for BioDEX Rank.\\n1 class EscoInferSignature(dspy.Signature):\\n2\\n\"\"\" Given a snippet from a job vacancy ,\\nidentify all the ESCO job skills mentioned.\\nAlways return skills.\"\"\"\\n3\\n4\\ntext = dspy.InputField(prefix=\"Vacancy:\")\\n5\\noptions = dspy.InputField(\\n6\\nprefix=\"Options:\",\\n7\\ndesc=\"List of comma -separated options to\\nchoose from\"\\n8\\n)\\n9\\noutput = dspy.OutputField(\\n10\\nprefix=\"Skills:\",\\n11\\ndesc=\"list of comma -separated ESCO skills\"\\n12\\n)\\nCode Snippet 4: DSPy Signature for ESCO Infer.\\n1 class EscoRankSignature(dspy.Signature):\\n2\\n\"\"\" Given a snippet from a job vacancy , pick\\nthe 10 most applicable skills from the\\noptions that are directly expressed in the\\nsnippet.\"\"\"\\n3\\n4\\ntext = dspy.InputField(prefix=\"Vacancy:\")\\n5\\noptions = dspy.InputField(\\n6\\nprefix=\"Options:\",\\n7\\ndesc=\"List of comma -separated options to\\nchoose from\"\\n8\\n)\\n9\\noutput = dspy.OutputField(\\n10\\nprefix=\"Skills:\",\\n11\\ndesc=\"list of comma -separated ESCO skills\"\\n12\\n)\\nCode Snippet 5: DSPy Signature for ESCO Rank.\\n5\\nMetrics\\nWe measure the rank-precision (RP) of the pro-\\nduced rankings, which is the precision of the rank-\\ning at the rank equal to the number of total gold\\nlabels. Specifically we consider the rank-precision\\nat K (RP@K; defined below). Given a gold num-\\nber of labels Rn for input n, the RP@K measures\\nprecision@K when K ≤Rn and recall@K when\\nK ≥Rn.1 Rel(n, k) = 1 if the k-th output for\\ninput n in the ranking was relevant, else 0.\\nRP@K = 1\\nN\\nN\\n∑\\nn=1\\n1\\nmin(K, Rn)\\nK\\n∑\\nk=1\\nRel(n, k)\\n6\\nData\\nWe evaluate our method and baselines on four ex-\\ntreme classification datasets, one in the biomedical\\nfield and three in the field of human-resources.\\nBioDEX:\\nThe BioDEX dataset (Biomedical\\nDrug Event eXtraction; D’Oosterlinck et al. 2023)\\nconsists of biomedical papers containing various\\ndescriptions of adverse drug events and associated\\nexpert-created labels for the exact type of medi-\\ncal reaction discussed. These events are encoded\\nin the MedDRA ontology (Medical Dictionary for\\nRegulatory Activities; Brown et al. 1999), a set\\nof ≈24,300 standardized medical reaction. Inputs\\ncan be very long (half of inputs have upwards\\nof ≈20,000 characters), and biomedical domain\\nknowledge is needed to infer the correct reactions\\n(not all medical reactions need to be reported, only\\nthe adverse ones). BioDEX models a crucial step\\nin real-world drug safety pipelines. We use a subset\\nof 10 training, 50 validation, and 250 test examples\\nfor our experiments. The median amount of labels\\nper input is 3 while the 95th percentile is 14.\\n1When K = Rn, the precision and the recall of the ranking\\nare by definition equal (Aslam et al., 2005).\\nESCO:\\nThe ESCO ontology (European Commis-\\nsion Directorate-General for Employment, Social\\nAffairs and Inclusion, 2017) contains ≈13,900 dis-\\ntinct concepts used to encode skills, competences,\\nqualifications, and occupations. We consider three\\ndatasets each containing snippets (typically one\\nsentence) of online job vacancies in English with\\ntheir relevant ESCO labels. We use the HOUSE,\\nTECH, and TECHWOLF datasets (Zhang et al.,\\n2022; Decorte et al., 2022, 2023). We take 10\\nexamples each from the HOUSE and TECH valida-\\ntion sets as training examples, and keep the remain-\\ning 51 and 65 examples as validation respectively.\\nTECHWOLF has no validation or training split, so\\nwe use the train and validation split of HOUSE\\ninstead. HOUSE, TECH, and TECHWOLF respec-\\ntively contain 262, 338, and 326 test examples.\\nThe median amount of labels per input across these\\ndatasets is 1 and the 95th percentile is 4.\\n7\\nExperiments and Results\\nTable 1 gives test results for all models and tasks.\\nBaselines\\nWe evaluate a set of baselines across\\nthe four tasks.\\nFirst, we evaluate a ranking\\nequal to the prior statistic over all the labels\\n(prior).\\nFor BioDEX, we estimate these pri-\\nors across all the BioDEX training data.\\nFor\\nthe ESCO datasets, we use the priors distributed\\nby Decorte et al. (2023), which are calculated\\nfrom a private training set. Subsequently we evalu-\\nate the performance of exactly matching the label\\nnames in the input document (exact-match). Fi-\\nnally, we embed the input document with an off-\\nthe-shelf retriever and retrieve over label embed-\\ndings. We use the pre-trained all-mpnet-base-v2\\nmodel (Reimers and Gurevych, 2019) for ESCO-\\ntasks and BioLORD (Remy et al., 2022), a biomedi-\\ncal retriever, for BioDEX (naive-retrieve).\\nThrough these baselines, an interesting dis-\\ntinction between BioDEX and the ESCO-tasks\\nemerges. Off-the-shelf retrieval is much stronger\\non ESCO-tasks. We hypothesize this is due to\\nthe shape of the input documents. Entire biomedi-\\ncal publications are hard to compress into a single\\nvector—especially with an off-the-shelf retriever.\\nThe short vacancy snippets are easier to handle.\\nInfer–Retrieve–Rank\\nWe\\ninstantiate\\nthe\\nInfer module with a Llama-2-7b-chat stu-\\ndent LM and GPT-3.5-turbo teacher LM. The\\nRank module uses GPT-4 as both student and\\nHOUSE\\nTECH\\nTECHWOLF\\nBioDEX\\nRP5\\nRP10\\nRP5\\nRP10\\nRP5\\nRP10\\nRP5\\nRP10\\nBaselines\\nprior\\n2.90\\n2.97\\n1.63\\n1.63\\n0.00\\n2.57\\n20.42\\n21.51\\nexact-match\\n5.89\\n5.89\\n4.09\\n4.09\\n3.43\\n3.43\\n9.60\\n15.16\\nnaive-retrieve\\n26.17\\n36.76\\n39.60\\n49.79\\n33.48\\n42.13\\n10.99\\n11.71\\nPrograms\\n(each program requires 10 training and ≈50 validation examples)\\nFinetune\\n# LM calls\\nInfer–Retrieve–Rank\\n56.50\\n65.76\\n59.61\\n70.23\\n57.04\\n65.17\\n24.73\\n27.67\\nNo\\n≈1,520\\n−optimize Rank\\n52.19\\n66.51\\n56.77\\n70.58\\n51.34\\n62.32\\n24.59\\n28.55\\nNo\\n≈1,010\\nInfer–Retrieve\\n42.47\\n52.62\\n55.01\\n62.45\\n47.49\\n56.50\\n20.69\\n24.77\\nNo\\n≈1,010\\n−optimize Infer\\n20.23\\n30.69\\n21.76\\n33.42\\n22.15\\n29.69\\n15.40\\n15.76\\nNo\\n0\\nFinetuned systems\\n# Train size\\nretrieve ♣\\n45.74\\n55.95\\n54.62\\n66.24\\n54.57\\n62.55\\n/\\n/\\nYes\\n≈138,000\\nretrieve-rankGPT3.5 ♢\\n43.57\\n51.44\\n52.50\\n59.75\\n/\\n/\\n/\\n/\\nYes\\n≈555,000\\nretrieve-rankGPT4 ♢\\n56.67\\n61.02\\n61.50\\n68.94\\n/\\n/\\n/\\n/\\nYes\\n≈555,000\\nseq2seq-prior ♡\\n/\\n/\\n/\\n/\\n/\\n/\\n33.78\\n35.52\\nYes\\n≈11,500\\n10×seq2seq-prior ♡\\n/\\n/\\n/\\n/\\n/\\n/\\n42.94\\n46.84\\nYes\\n≈11,500\\nTable 1: Test results for baselines, programs, and finetuned systems on the HOUSE, TECH, TECHWOLF, and\\nBioDEX extreme multi-label classification tasks. Metrics are rank-precision (RP) at 5 and at 10. Our instantiation\\nof Infer–Retrieve–Rank uses a Llama-2-7b-chat model to Infer, a frozen BioLORD or all-mpnet-base-v2\\nto Retrieve, and a GPT-4 model to Rank. Infer–Retrieve–Rank can attain state-of-the-art results compared to\\nspecialized systems while requiring no finetuning and multiple orders of magnitude less data. Each program requires\\nan amount of LM calls to bootstrap, which is compared with the training size used by finetuned systems. Best results\\nwithin a 0.5 interval in bold, second best results underlined. The finetuned system results are taken from ♣Decorte\\net al. (2023) and ♢Clavié and Soulié (2023) where available, or adapted from ♡D’Oosterlinck et al. (2023).\\nteacher.\\nThe seed-prompts are given in Code\\nSnippets 2, 3, 4, and 5.\\nWe\\noptimize\\nInfer–Retrieve–Rank\\nfor\\nRP@10\\nperformance\\non\\neach\\ndataset\\nsepa-\\nrately. Each run involves 10 unlabeled training\\nexamples and ≈50 labeled validation exam-\\nples.\\nEvery run incurs ≈20 teacher model\\ncalls and ≈1,500 student model calls, and can\\ncomplete in tens of minutes.\\nWe use dspy’s\\nBootstrapFewShotWithRandomSearch class to\\nautomate the prompt bootstrapping procedure. A\\ndetailed breakdown of optimization and inference\\ncosts, in function of the different LMs used, is\\ngiven in Section 8. We set the prior hyperparameter\\nA to 0 for ESCO-tasks and 1000 for BioDEX,\\nbased on a handful of validation runs.\\nFor ESCO-tasks, we compare with the best\\nfinetuned systems from the literature. retrieve\\ndenotes the retriever of Decorte et al. (2023),\\nretrieve-rankGPT3.5/4 denotes the system with\\ninference-time reranking of Clavié and Soulié\\n(2023).\\nFor BioDEX, we slightly alter the\\nmethod of D’Oosterlinck et al. (2023): we take\\na FLAN-T5-Large model (Chung et al., 2022) and\\ntrain it to output a comma-separated list of reaction\\nlabels given a chunk of the input paper (the origi-\\nnal BioDEX system was trained to output many at-\\ntributes, of which medical reactions was only one).\\nThis model does not directly produce a ranking, so\\nif a reaction is not predicted we add it in order of\\nthe prior (seq2seq-prior). We also consider sam-\\npling 10 generations from the model and majority\\nvoting the reactions (10×seq2seq-prior).\\nInfer–Retrieve–Rank achieves state-of-the-\\nart performance across all ESCO-tasks. Through\\na set of ablations, we find that each optimization\\nstep and module improves performance. Notable,\\nthe Infer-Retrieve system, which ablates the\\nRank module, can still attain competitive results\\ndespite using only one open-source LM and frozen\\nretriever. Infer–Retrieve–Rank does not beat our\\nfinetuned system on BioDEX, but adding a Rank\\nmodule or optimizing Infer consistently improves\\nperformance, indicating that programs can support\\na general approaches to extreme multi-label classi-\\nfication across a variety of datasets with different\\ncharacteristics.\\nConfiguration\\nOptimize Calls\\nCalls / Input\\nLM\\nTeacher\\nLM\\nTeacher\\nLM\\nModules\\nInfer\\nLlama\\nGPT3.5\\n≈500\\n≈10\\n1\\nRetrieve\\nmpnet (or similar)\\nNone\\n0\\n0\\n1\\nRank\\nGPT4\\nGPT4\\n≈500\\n≈10\\n1\\nPrograms\\nInfer–Retrieve–Rank\\nLlama-mpnet-GPT4\\nGPT3.5-None-GPT4\\n≈1,500\\n≈20\\n3\\n−optimize Rank\\nLlama-mpnet-GPT4\\nGPT3.5-None\\n≈1,000\\n≈10\\n3\\nInfer-Retrieve\\nLlama-mpnet\\nGPT3.5-None\\n≈1,000\\n≈10\\n2\\n−optimize Infer\\nLlama-mpnet\\nNone-None\\n0\\n0\\n2\\nTable 2: Breakdown of configuration and costs associated with our Modules and Programs. A module is configured\\nwith a single LM and Teacher. A program inherits all LMs of its modules, and all Teachers of its optimized modules.\\nPer module, the optimization procedure requires O(train) Teacher calls and num_programs ⋅O(val) LM calls,\\nwhere num_programs is a hyperparameter controlling how many programs to try during optimization. During\\ninference, 1 LM call is used per module. Programs inherit the optimization calls of their optimized modules, and the\\ninference calls of all modules. While Retrieve is not directly optimized, it does contribute inference calls to the\\noptimization of Infer-Retrieve because the Retrieve module is in the optimization loop. In this work, we use\\n10 training examples, ≈50 validation examples, and set num_programs to 10.\\n8\\nProgram Cost Breakdown\\nTable 2 outlines the optimization and inference\\ncalls associated with our modules and programs.\\nA module is always instantiated with one LM,\\nbut can also have another Teacher LM if it is\\noptimizable.\\nIn our case, the Infer and Rank\\nmodules have a Teacher while Retrieve does\\nnot.\\nOptimizing a module given its inputs in-\\ncurs O(train) calls from the Teacher model and\\nnum_programs ⋅O(validation) calls from the Stu-\\ndent LM, where train and validation denote the\\nsizes of the training and validation sets respectively,\\nand num_programs controls how many different\\nbootstrapped prompts to try in the optimization\\nprocess. In our work, this results in ≈10 Teacher\\nand ≈500 LM calls per module.\\nPrograms inherit the configuration and calls from\\ntheir constituent modules. Teacher optimization\\ncalls are only inherited if the module is actually\\noptimized, LM optimization calls are inherited if\\nthe module is in the loop for another optimized\\nmodule, and inference calls are always inherited.\\nTable 2 makes it easy to express the cost as-\\nsociated with any program.\\nFor example, our\\nstate-of-the-art Infer–Retrieve–Rank program\\nrequires approximately 500 Llama, 500 mpnet, 10\\nGPT3.5, and 510 GPT4 calls to optimize. This is\\ncalculated as follows. First, 500 Llama and 500\\nmpnet student LM calls and 10 GPT3.5 teacher\\ncalls are needed to optimize Infer-Retrieve.\\nThen an additional 500 GPT4 student LM calls\\nand 10 GPT4 teacher calls are needed to optimize\\nRank. Notice how Infer-Retrieve is in the loop\\nwhile Infer–Retrieve–Rank is optimized. Be-\\ncause of the left-to-right nature of this optimiza-\\ntion procedure, we can cache the inference calls of\\nInfer-Retrieve, saving us the cost of executing\\nthem again when optimizing Rank. Per new input,\\nthe program incurs 1 call for each Llama, mpnet,\\nand GPT4 LM.\\nThe finetuned systems we compare to in Table 1\\nall have a much higher start-up cost, in part due to\\nthe need for labeled finetuning data—which some\\nsystems need to bootstrap—and other costs asso-\\nciated with finetuning such as increased hardware\\nrequirements. These finetuned systems can be more\\nefficient per test-time input, given that we currently\\nrely on 2 open-source local calls and 1 closed-\\nsource API call for Infer–Retrieve–Rank. Our\\nInfer-Retrieve program is considerably cheaper\\nto deploy since it relies only on open-source com-\\nponents, while still being competitive. In the fu-\\nture, we plan to use an open-source LM for Rank,\\nmaking our best program considerably cheaper to\\ndeploy as well.\\n9\\nConclusion\\nWe introduced Infer–Retrieve–Rank, a general\\nprogram for extreme multi-label classification.\\nInfer–Retrieve–Rank achieves state-of-the-art\\nresults on three benchmarks using one frozen re-\\ntriever combine with two in-context learning mod-\\nules. These findings show that the future of prompt\\nand pipeline engineering need not be brittle. Mod-\\nular programs, once optimized, can serve as highly\\neffective general-purpose solutions.\\nLimitations\\nThe best Infer–Retrieve–Rank program cur-\\nrently requires one GPT-4 call per input document,\\nwhich may not feasible for all applications. In the\\nfuture, we plan to explore more efficient versions\\nof Infer–Retrieve–Rank which rely fully on low-\\ncost open-source components.\\nWhile\\nour\\noptimization\\nprocedure\\nallevi-\\nates the need for iterative prompt engineering,\\nInfer–Retrieve–Rank does rely on an initial\\nseed-prompt and performance may vary with spu-\\nrious features of these prompts.\\nIn the future,\\nwe plan to quantify how different optimization\\nprocedures can reduce prompt brittleness, using\\nInfer–Retrieve–Rank and our benchmark suite.\\nEthics Statement\\nWe have applied Infer–Retrieve–Rank to the im-\\nportant real-world tasks of biomedical informa-\\ntion extraction and job vacancy screening. LMs\\nmake mistakes and are biased towards certain pre-\\ndictions (Bender et al., 2021). We advise against\\nthe deployment of Infer–Retrieve–Rank in these\\ncrucial real-world tasks without proper understand-\\ning of the risks involved and how to best measure\\nand mitigate them.\\nAcknowledgements\\nWe are grateful to Jens-Joris Decorte and Johannes\\nDeleu for their useful comments, and to Jens-Joris\\nDecorte for providing us with prior statistics on\\nthe ESCO-tasks. Karel D’Oosterlinck is funded by\\nan FWO Fundamental Research PhD Fellowship\\n(11632223N). Omar Khattab is supported by the\\nApple Scholars in AI/ML fellowship. This work\\nwas partially supported by IBM as a founding mem-\\nber of the Stanford Institute for Human-Centered\\nArtificial Intelligence (HAI), Oracle, Virtusa, and\\nCigna Healthcare.\\nReferences\\nJaved A Aslam, Emine Yilmaz, and Virgiliu Pavlu. 2005.\\nA geometric interpretation of r-precision and its cor-\\nrelation with average precision. In Proceedings of the\\n28th annual international ACM SIGIR conference on\\nResearch and development in information retrieval,\\npages 573–574.\\nEmily M Bender, Timnit Gebru, Angelina McMillan-\\nMajor, and Shmargaret Shmitchell. 2021. On the\\ndangers of stochastic parrots: Can language models\\nbe too big? In Proceedings of the 2021 ACM Confer-\\nence on Fairness, Accountability, and Transparency,\\npages 610–623.\\nK. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal,\\nY. Prabhu, and M. Varma. 2016. The extreme classi-\\nfication repository: Multi-label datasets and code.\\nElliot G Brown, Louise Wood, and Sue Wood. 1999.\\nThe medical dictionary for regulatory activities (med-\\ndra). Drug safety, 20(2):109–117.\\nHyung Won Chung, Le Hou, Shayne Longpre, Barret\\nZoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi\\nWang, Mostafa Dehghani, Siddhartha Brahma, et al.\\n2022. Scaling instruction-finetuned language models.\\narXiv preprint arXiv:2210.11416.\\nBenjamin\\nClavié\\nand\\nGuillaume\\nSoulié.\\n2023.\\nLarge\\nlanguage\\nmodels\\nas\\nbatteries-included\\nzero-shot ESCO skills matchers.\\narXiv preprint\\narXiv:2307.03539.\\nMaarten De Raedt, Fréderic Godin, Thomas Demeester,\\nand Chris Develder. 2023. IDAS: Intent discovery\\nwith abstractive summarization. In Proceedings of\\nthe 5th Workshop on NLP for Conversational AI\\n(NLP4ConvAI 2023), pages 71–88, Toronto, Canada.\\nAssociation for Computational Linguistics.\\nJens-Joris Decorte, Jeroen Van Hautte, Johannes Deleu,\\nChris Develder, and Thomas Demeester. 2022. De-\\nsign of negative sampling strategies for distantly su-\\npervised skill extraction. In RecSys in HR2022, pages\\n1–7.\\nJens-Joris\\nDecorte,\\nSeverine\\nVerlinden,\\nJeroen\\nVan Hautte, Johannes Deleu, Chris Develder, and\\nThomas Demeester. 2023. Extreme multi-label skill\\nextraction training using large language models. In\\nAI4HR & PES, the International workshop on AI for\\nHuman Resources and Public Employment Services,\\nECML-PKDD 2023 Workshop, pages 1–10.\\nKarel D’Oosterlinck, François Remy, Johannes Deleu,\\nThomas Demeester, Chris Develder, Klim Zaporo-\\njets, Aneiss Ghodsi, Simon Ellershaw, Jack Collins,\\nand Christopher Potts. 2023. BioDEX: Large-scale\\nbiomedical adverse drug event extraction for real-\\nworld pharmacovigilance.\\nIn Findings of the As-\\nsociation for Computational Linguistics: EMNLP\\n2023, pages 13425–13454, Singapore. Association\\nfor Computational Linguistics.\\nEuropean Commission Directorate-General for Employ-\\nment, Social Affairs and Inclusion. 2017. ESCO,\\nEuropean skills, competences, qualifications and\\noccupations.\\nhttps://esco.ec.europa.eu/en/\\nabout-esco/what-esco.\\nOmar Khattab, Arnav Singhvi, Paridhi Maheshwari,\\nZhiyuan Zhang, Keshav Santhanam, Sri Vard-\\nhamanan, Saiful Haq, Ashutosh Sharma, Thomas T.\\nJoshi, Hanna Moazam, Heather Miller, Matei Zaharia,\\nand Christopher Potts. 2023.\\nDSPy: Compiling\\ndeclarative language model calls into self-improving\\npipelines. arXiv preprint arXiv:2310.03714.\\nNils Reimers and Iryna Gurevych. 2019.\\nSentence-\\nBERT: Sentence embeddings using Siamese BERT-\\nnetworks. In Proceedings of the 2019 Conference on\\nEmpirical Methods in Natural Language Processing\\nand the 9th International Joint Conference on Natu-\\nral Language Processing (EMNLP-IJCNLP), pages\\n3982–3992, Hong Kong, China. Association for Com-\\nputational Linguistics.\\nFrançois Remy, Kris Demuynck, and Thomas De-\\nmeester. 2022. BioLORD: Learning ontological rep-\\nresentations from definitions for biomedical concepts\\nand their textual descriptions. In Findings of the As-\\nsociation for Computational Linguistics: EMNLP\\n2022, pages 1454–1465, Abu Dhabi, United Arab\\nEmirates. Association for Computational Linguistics.\\nHugo Touvron, Louis Martin, Kevin Stone, Peter Al-\\nbert, Amjad Almahairi, Yasmine Babaei, Nikolay\\nBashlykov, Soumya Batra, Prajjwal Bhargava, Shruti\\nBhosale, et al. 2023.\\nLlama 2:\\nOpen founda-\\ntion and fine-tuned chat models.\\narXiv preprint\\narXiv:2307.09288.\\nMike Zhang, Kristian Jensen, Sif Sonniks, and Barbara\\nPlank. 2022. SkillSpan: Hard and soft skill extrac-\\ntion from English job postings. In Proceedings of\\nthe 2022 Conference of the North American Chap-\\nter of the Association for Computational Linguistics:\\nHuman Language Technologies, pages 4962–4984,\\nSeattle, United States. Association for Computational\\nLinguistics.\\nYaxin Zhu and Hamed Zamani. 2023. ICXML: An\\nin-context learning framework for zero-shot ex-\\ntreme multi-label classification.\\narXiv preprint\\narXiv:2311.09649.\\n', metadata={'Published': '2024-01-22', 'Title': 'In-Context Learning for Extreme Multi-Label Classification', 'Authors': \"Karel D'Oosterlinck, Omar Khattab, François Remy, Thomas Demeester, Chris Develder, Christopher Potts\", 'Summary': 'Multi-label classification problems with thousands of classes are hard to\\nsolve with in-context learning alone, as language models (LMs) might lack prior\\nknowledge about the precise classes or how to assign them, and it is generally\\ninfeasible to demonstrate every class in a prompt. We propose a general\\nprogram, $\\\\texttt{Infer--Retrieve--Rank}$, that defines multi-step interactions\\nbetween LMs and retrievers to efficiently tackle such problems. We implement\\nthis program using the $\\\\texttt{DSPy}$ programming model, which specifies\\nin-context systems in a declarative manner, and use $\\\\texttt{DSPy}$ optimizers\\nto tune it towards specific datasets by bootstrapping only tens of few-shot\\nexamples. Our primary extreme classification program, optimized separately for\\neach task, attains state-of-the-art results across three benchmarks (HOUSE,\\nTECH, TECHWOLF). We apply the same program to a benchmark with vastly different\\ncharacteristics and attain competitive performance as well (BioDEX). Unlike\\nprior work, our proposed solution requires no finetuning, is easily applicable\\nto new tasks, alleviates prompt engineering, and requires only tens of labeled\\nexamples. Our code is public at https://github.com/KarelDO/xmc.dspy.'})]\n"
          ]
        }
      ],
      "source": [
        "from langchain.document_loaders import ArxivLoader\n",
        "\n",
        "# RAFT paper ID and DSPy paper ID\n",
        "raft_paper_id = \"2403.10131\"\n",
        "dspy_paper_id = \"2401.12178\"\n",
        "\n",
        "# Create ArxivLoader instances\n",
        "raft_loader = ArxivLoader(raft_paper_id)\n",
        "dspy_loader = ArxivLoader(dspy_paper_id)\n",
        "\n",
        "# Load the documents\n",
        "raft_document = raft_loader.load()\n",
        "dspy_document = dspy_loader.load()\n",
        "\n",
        "# Print the content of the documents\n",
        "print(\"RAFT Paper Content:\")\n",
        "print(raft_document)\n",
        "\n",
        "print(\"\\nDSPy Paper Content:\")\n",
        "print(dspy_document)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "docs = [raft_document, dspy_document]"
      ],
      "metadata": {
        "id": "L9iSXNQFMhCt"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOT6wa3T8r0l"
      },
      "source": [
        "### 3.2 Split Documents into Chunks (4 pt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5Le-KNW8r0l"
      },
      "source": [
        "Usually, each document is constructed from multiple sections, each with a separate topic. It is better to split each document into smaller parts named chunks and search among them instead of actual documents. Write a splitter to create chunks from loaded documents."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.documents.base import Document\n",
        "\n",
        "\n",
        "def split_docs(docs):\n",
        "    splitted_docs = []\n",
        "\n",
        "    for doc in docs:\n",
        "        parts = text_splitter(doc)\n",
        "\n",
        "    splitted_docs.append(parts)\n",
        "    return splitted_docs\n",
        "\n",
        "def text_splitter(document, max_length=100):\n",
        "    document_text = document[0][0].page_content\n",
        "    paragraphs = document_text.split('\\n')\n",
        "    parts = []\n",
        "    current_part = \"\"\n",
        "\n",
        "    for paragraph in paragraphs:\n",
        "        if len(current_part) + len(paragraph) + 2 > max_length:\n",
        "            parts.append(Document(page_content=current_part.strip()))\n",
        "            current_part = paragraph + \"\\n\"\n",
        "        else:\n",
        "            current_part += paragraph + \"\\n\"\n",
        "\n",
        "    if current_part:\n",
        "        parts.append(Document(page_content=current_part.strip()))\n",
        "\n",
        "    return parts"
      ],
      "metadata": {
        "id": "vEAc9yyZMWf6"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "HQ5psfgc8r0m"
      },
      "outputs": [],
      "source": [
        "chunks = text_splitter(docs)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(chunks)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cbCYQUGzgjNd",
        "outputId": "c0d7214a-0999-43f3-ad46-a434f84bfeb8"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "735"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XztsdXrs8r0m"
      },
      "source": [
        "### 3.3 Retriever (3 pt)\n",
        "\n",
        "Create a retriever of your choice."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_community.vectorstores import Chroma\n",
        "from langchain.embeddings import HuggingFaceBgeEmbeddings\n",
        "import torch"
      ],
      "metadata": {
        "id": "YKR-anlGW72c"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "embedding = HuggingFaceBgeEmbeddings(\n",
        "        model_name=\"BAAI/bge-large-en-v1.5\",\n",
        "        model_kwargs={\"device\": 'cuda' if torch.cuda.is_available() else 'cpu'},\n",
        "        encode_kwargs={'normalize_embeddings': True},\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QWRvKFLTUmcq",
        "outputId": "4dfc04f4-aa75-45a0-af95-c85159c0772e"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q chromadb"
      ],
      "metadata": {
        "id": "miWIqxSpXTfR"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vector_store = Chroma(\n",
        "    persist_directory='.',\n",
        "    embedding_function=embedding,\n",
        ")"
      ],
      "metadata": {
        "id": "fJI2MK-_XPr2"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "vector_store.add_documents(chunks)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m1c8Dxdee77H",
        "outputId": "b5c00dec-1993-40a8-e523-379a081dac29"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['3f34dcc1-b029-4bfe-94b2-6b6f3831410a',\n",
              " '734c01f8-46b4-4fb4-829a-1cb45afc6da2',\n",
              " 'e2d52d8d-6965-43d6-a6e5-0a9e86389c46',\n",
              " '4230e836-2966-4e9a-bc47-b118d7c95f89',\n",
              " 'f6620ef1-0374-482f-ac5c-35d9f15419bd',\n",
              " '1b941d92-9419-4c27-a8ff-2b25aafa8382',\n",
              " 'd210bf01-8131-4f03-a63f-5bd4baf8bef3',\n",
              " 'dd354c13-b5a0-4b34-afa1-1861dc8e4df5',\n",
              " '5adbdc3f-e57f-4f4b-8d67-ea5362ca2dbc',\n",
              " 'd7727a58-fd4a-416d-bcb1-f7ace51c7239',\n",
              " '575a8b35-c539-491f-8a1a-2b02f411f446',\n",
              " '06cbc9f2-0275-49c4-8857-d23f138bd991',\n",
              " '7ee594d3-c640-4a1e-9fff-f414f6dbf428',\n",
              " 'fe1ac8b0-47b2-4cb0-9b6d-c07bb307e16f',\n",
              " 'c0a53e50-988d-4ef6-9562-e24ebecf3524',\n",
              " '2a9eb5ea-7331-4599-ae1e-77ab5a031906',\n",
              " '1becdcbd-ff0e-42f4-b1a9-00222d65931f',\n",
              " 'f105de67-0633-4c66-9ebe-43387f58b182',\n",
              " 'fa92a855-d478-4b04-b7b0-7db8bf205b07',\n",
              " '3a9348bb-3327-4a9b-b16e-81f4ee398cce',\n",
              " 'fa724b1c-a5ca-4ceb-8a26-ab7656c43844',\n",
              " '60755cbf-b385-4fa8-965d-daf773e93e66',\n",
              " '5e619afd-fba1-420c-99f4-0179b758c9d8',\n",
              " '15d3f54c-b1a4-4392-bc85-307391b3cbe8',\n",
              " '796e2e8d-b3e8-4bcb-b6f0-37285d84f3e3',\n",
              " '9d15bd6e-3f24-4cc7-affb-c66f00d636f6',\n",
              " 'e8378264-50f3-41d0-866b-a62d94728a93',\n",
              " '57af15fc-a11f-423a-bf91-e3da944cf108',\n",
              " '8ac9698a-6d31-4abd-a582-b0c3fdce4b9c',\n",
              " 'a6e38869-26ee-472d-aac0-b8e9653eb758',\n",
              " '4bb857a5-8d70-42ca-ac01-4ad85bbbaa52',\n",
              " '79bd77ac-5488-4f4f-8c32-d068f3d1b4f9',\n",
              " '8530449f-fac9-4de6-b54b-d2709fe1c850',\n",
              " '059fa41c-6df1-40b0-ac65-b577d9631092',\n",
              " 'a3f35ca8-de89-4f71-b3bc-782d33b14161',\n",
              " 'b2bdcd75-a5bf-4791-8abe-43986631ba0c',\n",
              " '2b0db29f-2a34-4979-bce6-056907bb08d4',\n",
              " 'e64905b4-3719-47fc-b981-d053e1e07cb3',\n",
              " 'c5a1c98e-8b0e-4f18-b0dd-029e95019d9d',\n",
              " 'c389dbe9-b11c-4bce-ae6a-f62cbb29d8be',\n",
              " '2dfe6d53-ba08-4a1c-a8c5-87b49d53de50',\n",
              " 'f994e777-1b22-4087-849c-a3ae20243cf6',\n",
              " '92409083-81e3-4998-a162-aa28bc917b31',\n",
              " '7096522f-bcef-4869-bf72-f16400132e1a',\n",
              " 'e653b065-6b42-40d4-a79f-4f1c4de2bd98',\n",
              " '7274bb77-ed8c-49ab-90af-4d91d8348859',\n",
              " '3b466c47-ba14-4666-b704-fbc18515a2d9',\n",
              " '183073ea-593e-49a9-8e4d-70929be2a893',\n",
              " '000d8ef2-406d-4894-99e4-61c15b20f48b',\n",
              " '5649e0d0-953f-4fc8-9d79-f694945e4cfe',\n",
              " '73817cf0-d89e-485b-be8c-f6903a04895e',\n",
              " 'b0f37357-1693-454a-94a1-2946528b69f2',\n",
              " 'cd03be4b-cdb2-478d-acfc-da2a963728f4',\n",
              " '9f272d76-bc84-4bd0-81da-ccc984d77939',\n",
              " '0ed06e2a-98bd-4726-bc4b-b1889eb9cbd5',\n",
              " 'eb716e4a-75e4-438e-acff-c6d4b03831bb',\n",
              " '0e7143bb-f441-4732-b1b9-f96f4133a094',\n",
              " 'd52e5daf-924a-433f-af83-9e765f453949',\n",
              " 'ead4a1e0-5a62-4417-8364-6bfae07ff168',\n",
              " '4f219bc6-eb9f-455f-b6b4-b8508a0abe66',\n",
              " '67ed406f-beb0-40e3-b42c-a688f6ec4f85',\n",
              " 'abff58ad-1e14-44b8-804f-122c8e556c6f',\n",
              " 'd9897792-6902-4076-ac58-e1d1ba6cc996',\n",
              " '15f64d5b-4ea9-4916-a6d1-1de5d7a98d98',\n",
              " '5b4cb314-0f27-4420-868e-900bf0bf8e50',\n",
              " 'dc03e3c4-1bff-4fcb-acc2-3cf0ec003bc4',\n",
              " '33249e11-29ea-4534-bc2b-6f953ae4f08d',\n",
              " '9965ff7d-cb9f-497a-ba97-e6edd2254ba1',\n",
              " 'b03b10e6-364e-4e9f-876c-9b42a12c67cc',\n",
              " 'fe3fe99c-1ce7-4940-8c50-422f3c6c709e',\n",
              " 'a2b8c2df-067e-4865-9fdd-d278640e0a9d',\n",
              " '5ec757c6-1fe1-4134-8bc9-10547a8b6563',\n",
              " '3b300875-8520-4a2b-ab60-49bd3801bf6b',\n",
              " '14b41f83-9224-4f49-8e0f-fab0f0b2c308',\n",
              " 'cb5bad20-18cf-408a-867d-4abd9f96186f',\n",
              " '16176c27-b280-49db-a3df-a07f10a939c1',\n",
              " '5b5be239-2888-446c-a95b-c35a64716128',\n",
              " '6162850b-3bf4-4163-9eab-4e5cb7800378',\n",
              " 'a023b35a-53e4-48fa-a775-41409b83060a',\n",
              " '659dae24-1d05-44a3-96b6-e5ececaca209',\n",
              " '3314f5b6-d350-4a2c-b803-b63f990fa305',\n",
              " '92febb1c-af0e-4ef7-ad66-7cb06a76e483',\n",
              " '89153764-c2f3-4158-aad2-fef1901ab4e7',\n",
              " 'a41eaa1d-2111-4379-994a-ef43a01698da',\n",
              " '63f1463c-e8e5-41db-b5e9-0f705d9a930f',\n",
              " 'f82eae86-c584-436e-a6b9-2830544a1daf',\n",
              " '3623d758-d612-4bf0-836a-9acfa9fe1643',\n",
              " '5a557d70-d62d-434d-bf11-0f316f48aff4',\n",
              " '61aa58e9-54ee-4018-99d1-4a8fac15bf65',\n",
              " '8f20485e-d69f-40ea-ac39-9aced77850f1',\n",
              " '8f68bf78-8767-4d89-8981-b3898bac3c0c',\n",
              " 'd0014544-02f4-4c5f-b33e-2d77adc60b47',\n",
              " '558a3280-641b-4c20-9142-13f5d32576f2',\n",
              " 'eab8b31f-8d05-44ff-b103-b1f2baffa6de',\n",
              " 'd54068fb-31d3-46b6-b407-ed50096e38a7',\n",
              " 'b2b6959b-007f-4c51-bec3-e005535d5f90',\n",
              " '5df85f2d-80b9-42fc-baf7-1c18b29ad0ef',\n",
              " 'f4b997d3-068c-4d1c-894b-e0cc783f0372',\n",
              " 'dd04431c-2a70-4583-b594-17a5e9b18f70',\n",
              " 'a167feb0-3ad3-4e8b-adfa-9c0b76249764',\n",
              " '5946827f-e17b-4dc3-91b8-9051b99e2322',\n",
              " '7a0c2534-134c-4513-ab59-4852dbc888a1',\n",
              " '2bee006b-e398-4d1c-8398-0ef1d32a2522',\n",
              " '068a9fb6-6af4-461e-8df1-1f31c112cba6',\n",
              " 'ac37a4a8-45e5-4ca5-92b3-c3ff41565679',\n",
              " '7eee0ff3-0184-41eb-b6d2-2c75d9248b8d',\n",
              " '9fc9d385-3b37-4d36-905b-b48400356a36',\n",
              " '355f3619-8a20-45ca-b6d3-419d6206d1ef',\n",
              " 'e912943c-1c0a-4002-8a5a-cbd62ef442d1',\n",
              " '45c24594-5431-4dc2-bc8e-9f4fe8a084ec',\n",
              " '646a48c2-0ec6-44a3-811f-ed7c51d2926a',\n",
              " '841e703b-d310-4792-8c2a-594ae22369dd',\n",
              " 'c094dad9-28d9-4444-aa31-9edd74350eb2',\n",
              " '9dd7a53e-295c-4a9e-8771-12799da5b94b',\n",
              " 'bb37db43-e2ac-429a-bfb6-6a3336cef9ae',\n",
              " 'e0bae6e1-ec0a-4830-b1d9-bc61ba38289c',\n",
              " '399be2b6-fbc5-4cf6-9bfb-169c3c38cfa9',\n",
              " '26625b5b-68a0-4e37-9193-8d4d2c353611',\n",
              " 'fc898bf2-20fe-43a3-a188-2e1228843faa',\n",
              " 'a3ceaf76-79c7-4f41-9e22-9fb68b283e6c',\n",
              " 'c42c4f6f-f266-4577-8924-beb709d09306',\n",
              " 'de6e80d1-937f-4959-ad1c-cb55dffcce68',\n",
              " 'f0f11328-f3d5-438c-963c-122f01f671f7',\n",
              " '96634319-0879-42d0-b8c4-9b58117da350',\n",
              " '917daeb0-819a-4c5e-a383-ca0f67df27b5',\n",
              " '3049c30a-6842-4c17-8c1f-ed2bfe3b9cdf',\n",
              " 'b5bae500-c304-4946-aa37-48aa53c851cf',\n",
              " '0e82db0c-3d3c-4f0c-94e4-b89cfd4352be',\n",
              " '344785d2-49e3-4ccc-821c-c9f4ce604cfc',\n",
              " '786626ec-a630-420f-a459-47c478d2798c',\n",
              " 'c71cfe15-d865-4d32-b23e-487e8580b2d3',\n",
              " 'a108f58e-e50d-4e7e-adb1-e05a3980174c',\n",
              " 'ca541c29-dc6d-406a-8a53-2df45b67e4bf',\n",
              " '7c345968-80a1-4005-97e3-deaeacdf75a2',\n",
              " 'eb5abf0f-dd69-4edf-8d92-e721a1cfcf05',\n",
              " '63a9874d-39c5-4e37-a8f2-ba5f51fba1db',\n",
              " '95401eff-b5f9-4841-ba45-5ae73a341d91',\n",
              " '41cf507a-41a3-4563-866f-addc3e3266e1',\n",
              " '315b4c4e-604b-49e5-9133-55430d5cbd6b',\n",
              " '4c73501f-2faa-4cdf-92df-c620557b1d1d',\n",
              " '2e085f1b-32a2-4798-95d5-19b461c00b2d',\n",
              " '1aa72ffe-46f4-4f30-9aae-7b6770a82667',\n",
              " '68b2caca-ac62-4025-810c-5c55bca3474d',\n",
              " 'c24726de-63f6-46c3-a9e9-422dfccc74bd',\n",
              " 'c5f41d81-f21f-4b06-9b60-d03db25534b9',\n",
              " '02c6973b-2f71-42fe-a00d-444cfed73f27',\n",
              " 'a2ec59f0-52e0-4ee1-ac64-da5739e60ab0',\n",
              " '8d1804b4-7993-4c1e-be88-8a87ec54741a',\n",
              " '3750c2cf-72ae-4773-aa88-4123e955b339',\n",
              " '3c8da008-78fe-401b-a154-21574b1a9e9d',\n",
              " '9e449921-cb56-4020-9bd0-4f6f5e90a447',\n",
              " 'ca6f0006-0b24-4e1d-9829-9c431a664d5c',\n",
              " '0705fcc2-2fcf-4fe9-a3c0-59aba256b9f6',\n",
              " '90d22812-7453-49d8-8237-913a345e24f7',\n",
              " '2222fc53-2f4e-4bfe-b171-145dbe61e52c',\n",
              " '23c0d093-a8ba-444f-8639-152d7d33784e',\n",
              " '86c364ee-d32e-42a6-9374-6687972f06a6',\n",
              " 'b2c97efd-b91f-4a62-a77e-b06bbcdabd4f',\n",
              " 'b31dd535-7563-41ee-acbd-67683ed76827',\n",
              " '168fe31e-6fe1-4e91-86e7-b6815db1e6cb',\n",
              " '127b6ba1-8dcf-4a57-af9e-4884542ae39f',\n",
              " '024e5657-5aef-4442-bf17-930c915c694d',\n",
              " '4ac1e2b9-6ae0-4088-8832-8c2b317f862d',\n",
              " '0f06b63d-c6b4-4787-a133-0accec9a37f9',\n",
              " '7fc6d1d1-4e35-4488-8232-31ee1491aed0',\n",
              " '4f2aa589-c1e7-4ca1-a04c-a7e512ddf8c2',\n",
              " 'f0357316-65a7-4b51-8c84-92ec0d13bdf9',\n",
              " 'bee9f5b5-882f-4ec7-8955-d0d73d6e7432',\n",
              " '3f72e023-e804-4b2d-a579-82328df95528',\n",
              " 'ba60e32b-f9cd-41c2-9071-f35ef69ce5f4',\n",
              " '273e5a04-232d-4939-ba62-bb14abde7fc2',\n",
              " '155047f8-ab1a-4735-8058-f9ee614e86ff',\n",
              " 'c5539238-7fdc-4748-b6c0-f3554768114b',\n",
              " 'db21d022-7877-4c76-ab08-337839a2b212',\n",
              " '0289652e-41ee-4f15-9a04-d3fe1986aeb4',\n",
              " 'cf19c6df-f7e4-42d3-96cc-3cdc1516c743',\n",
              " 'ba883285-b5d4-4ab8-b242-dac01c585ef8',\n",
              " 'f947f408-e95c-4562-bd4f-861f9bbe96a0',\n",
              " '7940d0e2-1ea5-438e-9197-608a1a6554d1',\n",
              " 'f31b219a-d722-4cf8-97a2-f1fd9f868404',\n",
              " '661f79b7-1c5c-4689-a290-e278d985a512',\n",
              " '40088667-cd82-4cf2-aa39-dbd5aa66c3ab',\n",
              " 'a29bfe0b-060b-4256-8bec-83ec2ea37f3e',\n",
              " '1730bbff-5c83-4591-bed3-725f80f336c7',\n",
              " '6c710c74-7e77-4e3b-8d96-5a2359439790',\n",
              " '06fc6aa9-fa9c-476e-b58b-fa1656d07b5d',\n",
              " '24dc96ff-246a-4485-af5e-0a21af4e9ce7',\n",
              " '5e4e22bf-bf06-471f-9cba-25811829b110',\n",
              " 'a85a676b-5620-4770-a94b-0665d72cef23',\n",
              " '77384874-aa65-49b0-980e-6609c5d3b880',\n",
              " '9c31ccb0-9f25-466f-bca0-1b7b7e7f106c',\n",
              " 'c46c5c68-f52b-4c79-b4d4-bde8616ef47a',\n",
              " 'af6af9cd-980c-46e5-a4c2-ec031601df54',\n",
              " '52d844ce-4ab8-4c49-bcd9-f9ebfb89fc73',\n",
              " '538ae38c-f296-44a6-8cce-60b8bddc0e25',\n",
              " '9fd32f41-5db7-479f-b674-f8f399c4d286',\n",
              " 'e4982162-5623-4ce0-96c9-e92e13e46cc6',\n",
              " 'ddcbe843-5504-4e57-8e69-d2b92b89a6da',\n",
              " 'c5696cba-66cd-496e-8d13-b30bd5f81172',\n",
              " 'd5caf5e8-59bd-4249-ba28-947e98ec72d6',\n",
              " 'aa548718-95e7-410c-bfcd-ad1484379229',\n",
              " 'c0ab7507-b0ac-49f4-b430-112b83a12d2d',\n",
              " 'e0eb16a4-6c02-4068-84ec-93c3ca89cee8',\n",
              " 'f501ce86-4531-47b8-90de-15ba7ea8df3d',\n",
              " 'e2ff735b-d2ec-48cc-8f48-accac67f5650',\n",
              " '826c9e1b-0bd4-42f1-9bf1-5455ed766072',\n",
              " '68dbc247-36b1-490d-9837-6ab8dd51c316',\n",
              " '77b6caa5-5a79-4699-94af-ac43b6b56e7c',\n",
              " '98d98da4-1ff2-4e02-8819-92ebd0f39377',\n",
              " '7ade6902-316a-4ff7-bf14-1365ff6c593f',\n",
              " 'cdc6f139-f8d0-4dd1-942f-6f804837275b',\n",
              " 'e5867c37-8200-414a-b946-9af3623a3517',\n",
              " '99541d77-21b4-4900-ac90-07892ba626ce',\n",
              " '44dc89c5-21f3-4680-90d9-0c13e37c2fb9',\n",
              " '12b7a557-6f3b-48d7-bd3b-f2652977ac82',\n",
              " '5243d7f3-ba7c-437d-9aae-16b4272e47f9',\n",
              " 'a22881ba-3881-457d-b10a-002a7d04e862',\n",
              " '67ec3b8e-b324-41d2-acc5-d59885989cf1',\n",
              " '9d3fad08-2b2b-4be9-92b5-e5866e221907',\n",
              " '9ca81f50-5163-42a1-b3ed-642c2eaaba01',\n",
              " '30a20822-2c6e-4025-81d2-645017d801e7',\n",
              " '808dc46f-aaab-4ddf-8804-675ba1c5a2f8',\n",
              " '2a8f6c33-49d2-407a-8df2-5318a5597953',\n",
              " '7681b969-c386-44cf-bcc9-74ad9cebd357',\n",
              " '23103915-0ce9-4910-8bb5-7be2a367a305',\n",
              " 'b2279074-d4df-4ec6-afff-ecf43d8deda1',\n",
              " '721d1a6d-068e-4e71-8e52-9841fa50e83f',\n",
              " '92ce136d-f97a-455c-8beb-263a21dbb5fe',\n",
              " 'dc2656cf-1abe-480d-beb7-9b65867e829d',\n",
              " 'e1a4fa6f-1ebe-4823-9a5e-8678f360a7aa',\n",
              " '6308eb44-8584-41b1-aec9-59c78a136e3b',\n",
              " 'f354765f-da77-4037-aebc-bca91cd2f960',\n",
              " '2de454a7-a62d-4a47-bd6b-497d251e86a9',\n",
              " '4acbb6a6-bdeb-401c-be71-13e30a786e15',\n",
              " 'da194af7-0bef-4ec5-91f7-632311071a0c',\n",
              " 'dc723928-ca30-4487-a8d4-9a95b5e59207',\n",
              " '24efd922-2ff2-4e3a-badf-90c79a2ae535',\n",
              " 'a5f55570-44d7-4f8d-b031-7944c44405f3',\n",
              " 'e31acdeb-7312-4f09-a6a0-2463b434f1aa',\n",
              " '3e841b6e-482b-46ff-87bc-b51103797794',\n",
              " '51fdb2e7-a0d8-4e3d-88e3-1632185f6d55',\n",
              " '4390e6ec-62f4-4dbe-bb58-040a308abc58',\n",
              " 'cbf54aef-af13-475d-b96b-6dba400bba38',\n",
              " 'd9c30cad-73a8-4007-ab83-1d781dad4c7b',\n",
              " '52cb9b37-eea5-4d46-9732-9fa79e772644',\n",
              " 'e10d9088-866a-4f9d-8142-b03744e1f5e8',\n",
              " 'e512f42c-a9f0-4efd-93fa-31149ba94ea9',\n",
              " '95e06154-3286-49e9-aae5-b07cc76abb47',\n",
              " '4dfa2024-274a-4068-80ef-8c4f38a9d6a4',\n",
              " '76d233f8-6340-4892-af53-180683f2d8be',\n",
              " '984875ca-468b-4adc-b9ce-bfa514182971',\n",
              " '7ef3211e-d32a-4c0e-a98c-93b103c25462',\n",
              " 'a61588bd-8c37-4c72-8fa7-61a93d053e96',\n",
              " '823997e8-4714-470b-968b-c7122b9dc133',\n",
              " '6576c7f4-5727-4a5d-8927-f4be8b9e8d99',\n",
              " 'ec9fc465-6676-4565-87ec-8a06c691f270',\n",
              " 'eece0d88-56ff-4c16-925b-8337230cf198',\n",
              " '38253e1b-6cc4-4f13-a97a-233b4fe9e4da',\n",
              " '61817ee5-8e93-485a-a960-75d3c6099f5a',\n",
              " 'eed9096b-6a12-4909-904e-dafef80a6d81',\n",
              " '1d11fd15-ba67-4d98-a1b8-ea34b3fe3cd9',\n",
              " 'f61d8faf-b61c-44ce-b421-ba10edb79e0b',\n",
              " 'd0842a99-bff5-4f6f-af13-2b9aa75871bb',\n",
              " '12041f81-1d8e-42b2-9837-f8370031e7e7',\n",
              " '8d85f3fc-e28b-45ad-8ff1-dec52ea0f0ff',\n",
              " '03f30426-7503-4a97-99df-449dd6e178c6',\n",
              " '038073a8-e694-4e54-97d8-1ce21a3ce1af',\n",
              " 'bb2aa39c-152e-4e73-88f7-c8b206e24e9b',\n",
              " '595f3d80-edad-472e-8f5a-d64284999ca3',\n",
              " 'b30e34d8-e47d-4bd4-a3f2-62c03fe13025',\n",
              " '1f24aff6-c2b4-4862-8724-51b253aa0746',\n",
              " '62ef147b-64ec-4637-b14c-7897b6bcfb25',\n",
              " '369d43f3-4f6d-45c1-8609-b109bb4afe14',\n",
              " '2f527cb6-8177-4d86-9239-806327e75c0d',\n",
              " '89bd3594-40e8-456e-9311-a570991082ad',\n",
              " '5a444f4a-0ba5-4053-9e99-2d1d587bbfc3',\n",
              " '3c6da4a8-254d-448e-bc71-7db8c4652711',\n",
              " 'd991a631-f617-4303-bc7f-1d32bd51c06f',\n",
              " 'd80efccc-25d3-4594-8e50-35ebc0767afb',\n",
              " 'a360032f-8996-422c-bab0-5060f8645365',\n",
              " 'b8b256c5-d6e6-46c2-9d74-fec0107e554a',\n",
              " 'c95785de-eba4-418b-8704-eb3f6fc32e3c',\n",
              " '8ed09c07-f039-4489-b41b-56fb5c3027e0',\n",
              " '01fc53cc-3018-4ee6-95f8-53ac5225b785',\n",
              " 'a8eec749-81ec-4b05-ab58-adb1a44ff116',\n",
              " 'c0b8e1a4-438a-4a11-a50a-36825cb1bcb0',\n",
              " 'd3228d83-3c47-4904-95fd-652e2ef24a50',\n",
              " '92b19288-7828-427c-9c7d-88bc8ca42e39',\n",
              " '27abfdad-85ac-4a1f-b908-29f49bd06ec2',\n",
              " 'e1796686-0548-4486-a730-392837bb4f8c',\n",
              " 'a44a151e-7bc8-4bd8-ab53-aa2a288fc6e9',\n",
              " '9f6af323-964e-4ebd-8671-3da2f577966a',\n",
              " 'a7719af9-5ada-455a-85ee-64ad98bc6a57',\n",
              " '7c84cff9-96f0-4568-bf2d-285977176168',\n",
              " '05e04b5b-5ad5-483f-87a2-3df448569a2e',\n",
              " '01f0cc02-59e0-408e-9206-515350f0e474',\n",
              " '96bee0af-77ff-4599-b201-b15c53b5525c',\n",
              " '7bcb06d2-c2a8-448b-8699-d1670d328514',\n",
              " '38da5e2e-a43c-469b-8b8d-85ee8211544c',\n",
              " 'adefad98-ccb2-4ddc-a33c-d02414a72a02',\n",
              " 'de001f67-fd7b-40ac-8993-b6a2b2083bc5',\n",
              " 'bf8970fd-08c9-4813-8968-8119795397dd',\n",
              " '14caa9ab-83b3-4de6-b03c-86204b57eb40',\n",
              " '9a7573f4-c810-4595-8368-98af344ed7c6',\n",
              " '60609eb9-88da-4fab-8805-6af8e5a05968',\n",
              " '87b4069d-c4e0-47b3-9966-f7e7f579dff7',\n",
              " '62b36db8-f7b6-4dad-9b86-fd07001a1bc2',\n",
              " '9c69b5fc-8db2-4c3c-af03-52ad11cb9213',\n",
              " '8b234a44-1199-4dc0-a733-301ee10c34b5',\n",
              " 'f7ba4d57-820b-4f56-ac5c-573f676ff567',\n",
              " '12f41c97-e5e2-482c-9abb-fccec7fdb026',\n",
              " '5bb0ddbb-ea7f-4a7c-81ee-d11e754474ae',\n",
              " '91aa76ca-76d0-4499-9fd7-99ce30736762',\n",
              " '7a34005c-920c-4fd9-81ff-bc28bb193bc4',\n",
              " '3ae62808-fead-405a-92bd-d833adf92bbf',\n",
              " '5060e6d9-8f2b-4000-9c95-0af5969d1ada',\n",
              " '9d1a9d29-6f45-4826-8fd2-686745998474',\n",
              " '4d4118ed-1a08-42b3-8533-cbf8e4aaffa1',\n",
              " 'e07a2cec-fd09-4af5-bb86-2a7f2029ebe3',\n",
              " '875db66e-175a-496a-b310-4c230318afba',\n",
              " '6b55024e-4fe5-41d4-b4e2-0e815a572200',\n",
              " '4d16f265-0bbc-4b8d-81d2-0063a6b9faf0',\n",
              " '859cae4e-8333-45b9-9de9-7d0b7e37c9e9',\n",
              " 'acda1831-bef3-4572-a897-64b6a9da8626',\n",
              " 'efbdc4f2-87b0-4085-82d2-e43c33dede25',\n",
              " 'ce1fcc8c-0c90-46b3-a7e2-848c54ad6f5a',\n",
              " '1c6e9e63-9670-4914-9ca7-10aae0a3cb3b',\n",
              " 'c5397cad-72c6-40ce-92fb-1dd479e06e7d',\n",
              " '7eefc2c7-2e69-4c10-936a-a608056d485b',\n",
              " '0e9e1c0e-841e-4e83-b0c7-21afa48c5c16',\n",
              " '627e95cd-36b1-4336-8cfc-2ac7988d958e',\n",
              " '61d09932-cec4-4b77-b7ad-0c6b609677df',\n",
              " '8da9eeaf-ca6b-46b5-9439-aa288e00c673',\n",
              " '03f6e241-f91f-4579-ab26-f547040c31b1',\n",
              " '349ce594-fc1e-4a18-8f6e-b9e680714487',\n",
              " '7d6413b7-bd97-4c02-bda1-5550bc3cde91',\n",
              " 'f0b359b9-1785-4e9a-a23a-40470bd53129',\n",
              " '4f3ae2c4-5808-4f53-b778-a16714ce176c',\n",
              " 'e074172d-4947-4882-a28d-e2d59e00fc1e',\n",
              " 'a6b07e73-2c94-493f-bf2b-0fa732ce6f4c',\n",
              " '564dff3c-d612-47d3-8316-d4b85c4e50e7',\n",
              " '0818ddb0-3b16-4b84-80c3-e71809ce37cc',\n",
              " '2827ac31-d788-4e32-bc98-c478ebcecbed',\n",
              " '5e332009-6943-4d13-bfc5-b0ea438700a3',\n",
              " 'c26acdfd-96fa-44a5-b283-d51547ca8d05',\n",
              " 'c3673acd-71cc-43a6-ae5b-aca133c0ecd9',\n",
              " '91bce0e0-dc35-49a4-a044-4e5334013bc6',\n",
              " '23e92b88-f42a-41e4-9570-1be1a8024c42',\n",
              " '47dc0c74-ee30-4c11-916d-d2235627477b',\n",
              " 'c095386b-38fa-4c56-932e-5ed564d3cae8',\n",
              " '8850f8f4-4a03-45e8-bc36-c0936a962fa2',\n",
              " '20cbe7fd-cf2b-4faa-ab8a-2d0218d21671',\n",
              " 'dd57c3c3-6cad-45d2-b2d6-cafbf78f91a6',\n",
              " 'b8a8a59a-66c1-4a12-93f5-e62d57028a46',\n",
              " 'b957c6d1-20b9-4ff9-98a8-88bd8701601a',\n",
              " '7ddffa98-4fd5-4dbf-b881-15f96ced73cd',\n",
              " 'c5433147-b49b-446b-8ad1-db7ea67169ff',\n",
              " '88ab2e22-14ff-42fb-818a-4ffabde6cee2',\n",
              " '7f11625e-5467-4c2b-855c-18e41288409f',\n",
              " '1fc8a54f-1e27-42a5-ae74-8f68459ab694',\n",
              " '80c6185a-16fc-4616-a3b2-e2d1fd25a506',\n",
              " 'e752bb6e-1b6e-4f07-b1f5-57018294c651',\n",
              " 'd9c00990-3ebe-43c1-8423-4574e17cc11a',\n",
              " '9237108b-a032-4860-9a5b-61cfa3ecf433',\n",
              " 'c6d684b8-5c17-454c-ab32-57a39167b78b',\n",
              " '2e073623-4c75-4fce-96ca-1e894e85aaf7',\n",
              " '700dca17-510f-43fd-be79-647698c95dda',\n",
              " 'ad65bd2a-8d03-4d43-b195-b42b3f6e7e45',\n",
              " '78ced6b6-1c83-4217-90a4-0ed1f7c3b93a',\n",
              " '6b6f88ae-5f1f-4754-903b-9cdd5a1cada1',\n",
              " 'aa100c01-cdb9-4907-9a85-d49309fc1c10',\n",
              " 'f82191aa-9764-44ed-95f1-07ee19a09766',\n",
              " 'af800855-a897-4d03-90e1-4d07eb6bab06',\n",
              " 'c42c1f5f-4c85-45fc-8339-dc2d2620abd4',\n",
              " 'eb4c2047-99a1-4cf9-be56-b23a00bb2e07',\n",
              " '591db770-f6e2-461c-abc1-b15d0afb67c0',\n",
              " '2fbd466e-0adb-4f27-8308-8d4d45f0951f',\n",
              " '325ab85f-0ffb-4859-9f68-6a397d0b03aa',\n",
              " '4a222815-6f73-4691-9777-5460ae1da5a9',\n",
              " 'e1046227-44c7-4f90-aeeb-c915d7e4e7cd',\n",
              " '781fa962-2a61-47ae-92bb-55703744e486',\n",
              " 'fe64c7a2-37f1-4c56-8a9e-27acaef885ac',\n",
              " 'c6136963-d7fc-41e1-8057-43b391d9a8c1',\n",
              " '9f4037cf-7b54-4d7f-81f1-1bb89ddc4e17',\n",
              " 'c429e110-b799-47c4-9a20-6e5c5cddb726',\n",
              " 'db2abb07-f99b-4715-82a7-c3146f70347a',\n",
              " '745b1fb2-d9fd-455a-8a22-e46af775e65b',\n",
              " 'f1312614-6fbb-4f4b-95ac-a4fe4e6e3923',\n",
              " '52ac7f3e-62d0-4009-bcd6-79479f417742',\n",
              " '4b54c175-7989-4253-a9bb-193ea72b870d',\n",
              " 'd0321780-67b1-47fe-b20c-608e9ae83ffd',\n",
              " '850b9c09-853d-410b-81be-eb4602bdb4cb',\n",
              " '9a5295a8-db23-4019-91e9-d216271e8d72',\n",
              " '260f1d61-fee0-4966-90a8-633c8590750a',\n",
              " 'c2c100db-8c71-4c4a-ae6f-77d2be6c1aac',\n",
              " 'ce1ba99c-4538-4d45-b6f1-0b2009eacab3',\n",
              " 'e242f6f9-17ba-4dcc-939c-f09472819a17',\n",
              " '3c7010c7-6a98-49be-988d-8fc30faed44b',\n",
              " '83a65814-c748-4983-8f97-ecb6a70f0399',\n",
              " 'a34e5942-0166-41db-97cb-553c18840e7c',\n",
              " '32a380ac-1dbd-4926-8316-1a74e9a27ae3',\n",
              " '3a950014-aec2-49ff-ba6f-8f1b893066a6',\n",
              " '59f10f73-a9f3-4d4f-853d-99e6acc6a40d',\n",
              " '9219cae2-49ca-4b49-8bc5-9bc45f94af65',\n",
              " '3a278e9a-45c8-49a4-8e2b-9588117f6e33',\n",
              " '0e37ee37-42f2-4e3c-80e7-15d420bdae97',\n",
              " '5a19b6b7-514b-41da-ab8c-8a3e7481eb4a',\n",
              " '6119f8a8-5874-43d2-8b45-db3864c6647f',\n",
              " '781dd331-2e7f-43fd-aec1-5743574fe9db',\n",
              " 'a367f846-c125-4265-a6f0-fc2ef37be218',\n",
              " 'eb90c5fe-0f1b-4741-9e44-8a2aab60802d',\n",
              " 'a0dd073f-d726-4da2-ad82-a01bbfc7d6f5',\n",
              " '4f1ddc60-e95c-4a37-83c2-126c27e43bdd',\n",
              " 'd02761a6-2578-4abc-bcdc-0f1ea6b8963c',\n",
              " 'b702da58-5d7e-4288-a287-917e7fc0c434',\n",
              " '4b8ba0de-eda1-4066-8da9-c69605c3450b',\n",
              " 'a465cf38-0ecc-4ef5-b8fc-198c41111e44',\n",
              " '577ece97-0bbf-4f5a-aa3d-acd21d52f263',\n",
              " '1cdf7991-5ea8-415d-bb67-feb91694afa9',\n",
              " 'dd13427f-204d-4441-a994-a0ee88c1169c',\n",
              " 'a906d662-c10f-4da1-bb99-53f3d80d71fc',\n",
              " '707f0776-6e9c-4595-9c50-adcefea420b5',\n",
              " '523b99d5-ba3b-4583-9628-5aa783222faf',\n",
              " 'abfaf925-3788-4fca-82de-0bb4d5f624f1',\n",
              " '3be4366d-c6b9-4982-a425-aa3ed95d35f5',\n",
              " '4f6d8adb-6ccb-4ad6-ad03-bb6ef080a3a3',\n",
              " '938c0982-8593-4afc-a5d8-f282a6af2667',\n",
              " 'ebf7f3fb-645f-407b-9323-fa2bcd0fa95e',\n",
              " '5b495503-d022-413e-984c-4bf5f7f89867',\n",
              " 'b8678fff-6835-4183-b3d6-99aaef3b9f12',\n",
              " '6ab726e9-7409-41fb-8ad2-11ae60139edd',\n",
              " '75a34fd6-a592-44aa-957d-9d7817d3ecf5',\n",
              " 'da286511-12b4-49f9-810e-41c74387d87d',\n",
              " 'd8547f2d-a53a-45b8-9e7d-0929be89b71a',\n",
              " '3d0de7f0-8ffe-4131-9d2b-df0e05147bac',\n",
              " '4886ee3a-c08c-4dd9-af09-e074ef89e8e4',\n",
              " 'c059e7fb-8323-4590-8c43-b2935789fda1',\n",
              " '4a2e859a-ad1b-4195-96f3-f21714fd04e2',\n",
              " '988110a9-48ae-4a5b-933b-f33c4fac51ab',\n",
              " '77dc1f44-cc72-4e03-8c85-68e0c6ad5849',\n",
              " '35d8eac2-79fd-445d-aac9-f0a88bf44363',\n",
              " 'c29033d3-86b1-4658-a820-93030d8e8202',\n",
              " '8c63e95f-776c-4a8f-80bb-0b265df669e3',\n",
              " '7df70e4a-232d-4790-b166-2ff1fa175a0e',\n",
              " 'dce1a45c-4b2e-4761-980f-6a231b846ae4',\n",
              " '8dc9015a-5003-42b6-ba1a-fbbb229fb0da',\n",
              " 'ffc24cf2-7aa3-42fc-83aa-01fc46415625',\n",
              " 'b3822aa1-58ca-4be0-9664-62e539df8ebb',\n",
              " 'f54b5690-afed-4011-acc4-5431b6fa9752',\n",
              " '2d20086f-76be-467f-9a74-c078f669967a',\n",
              " '4bc912eb-282c-4148-aef5-07f4e7ddcbbb',\n",
              " '1ac5eb27-87bc-4a7c-90ea-b4db32cf7fc6',\n",
              " '3daf194d-f2d6-47b7-a2c6-2b47416705cb',\n",
              " '17f8656b-dbd8-4a78-aa91-eba4220a8ab6',\n",
              " '653ff949-1837-484f-a53e-664e8f0f8175',\n",
              " '4aea811f-512c-48ee-8bbf-acfd21f732ad',\n",
              " '2b117d1d-4ee1-4f3b-b3e8-1e163e8a88df',\n",
              " '944c4849-312d-4c45-9031-be8fcf31ac7f',\n",
              " '8fcc752d-4ee7-498d-a347-dc3b13c8d422',\n",
              " 'd06d499c-dafa-47fc-86b3-7f44997090de',\n",
              " '84692500-34e4-407a-97b7-c90ae7acdbfd',\n",
              " '8d60e6f4-1954-410f-a290-c44adf88e012',\n",
              " '5f4d8ae4-36f0-4cd4-8b2b-dee2b8913086',\n",
              " '0a79e0a0-6274-44b3-9732-0ab056896ad2',\n",
              " '3cba674c-a1ea-4898-9760-da55a267a9dc',\n",
              " 'deecd980-4a08-48ea-bf36-741bcb4437b7',\n",
              " '8122509c-4be8-4cc3-adb7-59021049388a',\n",
              " '8f2b5656-1f41-4503-88d1-aafa3a3c8d02',\n",
              " 'd1605ac1-3af0-4de0-92cf-08eef61c4d47',\n",
              " '0c3e19cc-1ebf-422f-a84d-ef5e6dafcfd1',\n",
              " '687c375e-158e-4a85-bf87-15fc4bac538e',\n",
              " 'e37811cf-68d2-49ee-9fdc-8b38dba152b5',\n",
              " '19a44671-1710-4e93-b6a0-a4e00ee01f05',\n",
              " 'c9b15b26-1754-4da8-bff9-5db219a865ef',\n",
              " 'fb093952-afb2-41eb-93b1-a2b5a30bb121',\n",
              " 'f3b7c568-c228-4803-896c-c71347782173',\n",
              " 'f4106268-8958-4a6b-88b0-d323aad9d74e',\n",
              " '2a308971-458e-44e9-9011-f606c183ef37',\n",
              " '2f0258a2-bc1a-403f-9014-fd20ad1040a2',\n",
              " 'ccd07082-da0c-492c-b450-07cfb6e1d9a6',\n",
              " 'b5653df1-6db7-4703-b604-e3f5d203dc75',\n",
              " '84251ab8-9802-4b51-bd4d-442cdb268ae2',\n",
              " '4d2afbef-c7e8-4974-82a8-5440f5cae0e1',\n",
              " 'b32eb1be-ef1e-4a3f-8332-adf44a22a428',\n",
              " 'dd9746e6-d8be-4cbe-8a5f-290f07ceb45f',\n",
              " 'ba33dddf-fed3-459b-9bbe-92e051b52cab',\n",
              " 'ac9d2ab5-bdca-4b18-a0f6-73325b445b2e',\n",
              " '31ce6e3d-d113-44e6-bca4-1ce309c2f74d',\n",
              " 'ff72bf03-4551-47f2-9166-c7c1f8c94890',\n",
              " '1fae4656-a0c8-44b7-8ea2-c9e54ced48d6',\n",
              " '3feda161-3393-48e5-8e02-1ab02ac6ea41',\n",
              " '014661b3-b882-415e-b171-b69f1244be2a',\n",
              " 'f0fddeb8-d33a-4876-bc60-296294bfd224',\n",
              " 'ed345a4c-62a5-4a8c-8773-9a807d9316d8',\n",
              " '8c426331-ed4d-48f9-b8d9-72e92620beda',\n",
              " '46e7586f-29da-42bd-a68b-373678fe401e',\n",
              " '37db2c61-abc1-4dcc-a82f-3fce99e2633c',\n",
              " '3b8b3066-6772-46f5-9408-499503ae1e52',\n",
              " '5d5d78a2-2b05-45f5-af0c-ff1a1f2af56c',\n",
              " 'b10adfa7-462b-4b16-b923-06f1abdf2259',\n",
              " 'fff477a5-65fa-4999-a1b3-b39dfb939771',\n",
              " '8f54ebe7-a9bd-4185-9ddc-61bfb1ad8cc9',\n",
              " 'cd5ee059-6d4d-4830-af1a-85a7e15bc67e',\n",
              " '6934f31b-0733-4f45-8f8f-5b9d4d765547',\n",
              " '4f8da512-0ae3-4482-a02c-408f32adc2d8',\n",
              " '1d4defa6-ac04-4213-ad21-27118dd701e3',\n",
              " '907f4523-2668-4ece-9d80-39edc9aa9c65',\n",
              " '7fd16a5c-fe58-45e4-8b40-553d35e98bb4',\n",
              " 'afea2e57-2c94-4eea-aa1e-782a5fa30c9e',\n",
              " '74c7520e-6b61-412a-b4d0-d8b22487fba0',\n",
              " '33ada24b-25a3-4ce9-9bb5-03e25565856b',\n",
              " '0ab8578a-6bf6-44b4-afb5-2922565395e6',\n",
              " '620128fa-5dd8-4885-b7bf-d533da6a7d7a',\n",
              " '1a8b7b8f-edaa-41a9-86dd-298d7d95df79',\n",
              " '938d6478-7844-42ab-976f-6981fff66d89',\n",
              " '78411d7b-bd28-433b-9619-cd0d42af0fd9',\n",
              " '08728e75-84d7-4960-879c-fbb45174b02b',\n",
              " '025eef5e-7002-41f3-bc2d-2b6ff404dbe8',\n",
              " 'b65f6015-cc63-414a-907e-a303627de9da',\n",
              " '24a17ad9-263c-48e5-b53f-f3dd536ff6f4',\n",
              " '8f5b9423-05c7-4037-ac3e-84abfd4a0b4b',\n",
              " '199d01e2-0fa6-4e06-8c24-ea64e53cd4e7',\n",
              " 'dd267c87-218f-49b1-8dc2-7df4b482d14f',\n",
              " '2c7074c4-5a12-45fe-b9e7-2b0407886116',\n",
              " 'e60246f2-e81c-4d13-99af-a636842c7231',\n",
              " '8bb33b3e-5c24-440d-a250-fdb37b6d1de7',\n",
              " '99e820af-97b7-475d-86f1-09bb949d5874',\n",
              " 'b41e2186-9d25-4dae-9834-9ffbc2aa6cfa',\n",
              " '03de5003-6854-4de6-b772-a6b54f456b87',\n",
              " 'f2c2fb5a-9f16-40f5-8460-2908721b2af5',\n",
              " 'be3142c7-a3e8-4371-b6cb-a75616316d55',\n",
              " 'c0d68b55-b308-42b9-94fb-c30f9aba61fd',\n",
              " 'd073e6ff-7530-4f1c-8724-08e8a044430e',\n",
              " '6e73a7e2-9d32-4dae-8991-6a60a1c857a2',\n",
              " '7a73d7b4-5d13-450b-a942-01dda600caf0',\n",
              " '636a0802-6133-4814-8ebf-5270de5607fc',\n",
              " '92454ceb-1553-4309-8bd5-877c779b4bc2',\n",
              " '6dae8083-21bc-46f5-abb5-d61368a849a9',\n",
              " '0577ef6b-b477-46fe-977c-a0a5e672b9e9',\n",
              " '49e89ede-2be3-4283-bc2d-575d9333ad95',\n",
              " 'e4c63c98-fe0b-4638-86c0-fd26a6897117',\n",
              " 'a60dbc60-c104-43bb-a440-e703ff4d2477',\n",
              " '7675aaf0-4390-45a8-a750-dae48bb34310',\n",
              " '491f8a3d-a27d-469d-b848-0e02d250585e',\n",
              " '3325c9b6-4ea0-4369-8519-c9d99c2b8f77',\n",
              " 'b6e4fbb8-1496-41c4-89e7-67932b68e8cb',\n",
              " '217bea2b-20a4-4e10-989c-56608cd4141a',\n",
              " '929185c9-657c-480e-b5b8-46cf6310b7c8',\n",
              " 'eddaa186-6e82-4c3d-aa51-1a3f2a7cf3d7',\n",
              " '3782c245-9a89-4a48-ae3b-94032f27975c',\n",
              " 'c0e8f651-1b4c-4d52-9f35-59cd828595ef',\n",
              " '982cd1ec-c7aa-4e6c-9d6c-60ded3afa8d2',\n",
              " '36d612c1-f32e-4c1e-9a43-321921ca665c',\n",
              " '274c022d-6140-4a30-8b98-f09dd381e923',\n",
              " '3aa319e8-275b-42fd-b479-1b98edf63841',\n",
              " 'fda6597c-8eca-49f8-8c6d-143d7091736e',\n",
              " 'ca3b452a-b180-4113-a1b5-4c9b8a6609d2',\n",
              " 'dc146b07-27a3-4ac2-8dc6-3882a0d37ca7',\n",
              " '37b7e049-622f-4263-91fa-d7e273722879',\n",
              " '4c1f6e46-a3cf-4fdc-8cc3-b7a00681d39c',\n",
              " 'c669cfb4-1c5d-401f-b53b-7847cfa4a79a',\n",
              " '8e390418-da40-4530-800e-e299da0d9951',\n",
              " '196fcec8-fe11-46df-bd04-1a91d8075329',\n",
              " 'b51d20e5-d6b3-4a8c-8ff2-b86127aef7d8',\n",
              " 'ba4df02b-828b-4442-ab60-00ab289fd110',\n",
              " '40c796f4-86f8-4297-8153-9bd2f09dd73d',\n",
              " '7903229f-cc4f-4b52-ad76-83ee360bdaa5',\n",
              " 'f85b909b-71c1-42bf-9b74-dd96f53e3484',\n",
              " 'f2139155-2bc1-4c18-964a-4c28f5802374',\n",
              " '608a736d-00d6-450e-8f5f-1a2b220b20ab',\n",
              " '79975e63-5c17-44e8-83b6-b00ee1a1ec34',\n",
              " 'bf7a0721-d6d7-4136-8494-7484259cef5c',\n",
              " '781c008d-9706-490e-80c7-c0186d9535f5',\n",
              " '744b2846-7751-4860-a9d8-df2cdb78a3a7',\n",
              " '7eb5862e-d9d2-4518-bf55-0155923383f9',\n",
              " '4a9240e2-dd67-482e-b394-1b9c713219e0',\n",
              " '107bdb60-0984-4096-bfab-63b7003f6d6d',\n",
              " 'ec0cc672-b8db-4888-bba3-d8199180b69d',\n",
              " '1bc84b4a-7194-425b-b2d8-4644fa79e049',\n",
              " '9ca32197-24bb-4bbb-9719-df8f6adb9677',\n",
              " '8433ac1c-b13b-4f50-8fcc-db3df3dce3f3',\n",
              " '2a0becfc-1e04-451e-8269-68567337faa6',\n",
              " '043e69fc-4f45-48ef-b282-7b2a03ad28a3',\n",
              " '4f3164a9-9a0c-4734-9966-a773ef63caa3',\n",
              " '170f9aff-564a-40e8-9705-082a825571bb',\n",
              " '5ae5edf3-8afe-414c-9640-fa28608898bb',\n",
              " '89f2144f-0490-4c93-9ba9-c4fd84a8ad33',\n",
              " 'c252d9ad-c023-4183-8a6c-715e62eb60a2',\n",
              " '2b27efc3-a22d-4464-80a9-ecbae7eea3db',\n",
              " 'e1073d4c-8b12-450f-9aca-3b5736cd2b55',\n",
              " 'b6069490-7023-4fd3-b3fc-436ff1ce450c',\n",
              " '784667a5-c0ae-4b71-8151-b413f103b5c0',\n",
              " '5ae11f5e-b118-4214-b482-ef3663977e27',\n",
              " '8d487f77-4359-454f-b351-877a0c70de62',\n",
              " '69dd6d32-6465-48df-b2cd-a8a23aab6ffc',\n",
              " '1cb60e5f-fba1-459f-b21c-d5d3559b5248',\n",
              " '2f85e9d4-91e3-4b55-aa31-c360ac266625',\n",
              " 'db174af9-fb03-4031-968d-c7c41bbe3852',\n",
              " '3fe999a6-3054-4223-9670-711eb4ca77e8',\n",
              " '79287729-8bb8-4446-a952-bcb28e368db5',\n",
              " '8a7f71af-32bb-4c5d-9610-3ff0e854e68e',\n",
              " '6c691528-b23a-4e50-a506-0513b2988de7',\n",
              " '7b121eb5-71bb-40bb-9c39-f89a609bb2d0',\n",
              " '01878f71-21c9-4e2b-8053-156305e521bd',\n",
              " '24bf4943-40a9-4d62-aeb0-4e3f2896adbf',\n",
              " '5aa4711f-75fa-43d0-ad10-19cf6a81f5b7',\n",
              " 'acab95a9-02be-4ef3-9e37-5bbc0b21cf27',\n",
              " '82495586-0100-4d27-9867-d8df50208deb',\n",
              " '3866b529-610c-48c6-b053-19ff24631ef9',\n",
              " 'a351dfb6-ff59-4fa8-a73d-6bc877a2354d',\n",
              " '82bdd0fd-c385-48ce-a86d-59e80c441531',\n",
              " '5c18acdf-b280-4392-b9ac-258e9d264792',\n",
              " '5d0ed3af-48b4-44de-9eb2-69617c89ac76',\n",
              " '66d29cf7-9edf-4441-8d2d-71a93199208b',\n",
              " '478a90e7-dd0b-4500-b4cb-7754ea47cb1d',\n",
              " 'ea526082-7294-45de-8d80-3c349c4a8489',\n",
              " '51121132-2371-44a4-86be-f8328d4148d0',\n",
              " '6cc25169-f098-412d-ab6f-0fa887e3dadb',\n",
              " '133df100-c1fe-4aa7-b43e-b8d965073d2e',\n",
              " '908137ed-d66c-4377-9ce0-25e4466a968b',\n",
              " '4bea3db7-7f6d-473e-a042-51fb4b044f49',\n",
              " 'b1fac413-de66-4bd0-b030-97509820da24',\n",
              " 'c55405e1-e17a-4959-b50b-ef6721ba02a9',\n",
              " '17df05e8-5139-4c08-b6a2-4805ee5725c8',\n",
              " 'cb5f6a07-304d-4466-a62c-61f7c9c820b0',\n",
              " '75a1c22e-d237-4fa1-9a3a-5e114e46ffb8',\n",
              " 'd16aa98b-cbbb-4e7e-b0af-f3668fe77622',\n",
              " '660ed7a5-359a-4450-96f5-441b5e35e3d9',\n",
              " 'c5918e49-7038-45e8-a1b7-5c2bd3dfa00f',\n",
              " 'f6575ee4-6749-46de-8c38-b1b97a104d1c',\n",
              " 'ac1f005c-a402-4df6-89b7-0e80df653ea4',\n",
              " '634c7432-53f2-4ef7-8b81-b5161e5b513a',\n",
              " 'd720166c-ba8f-4590-9690-827092946863',\n",
              " 'dd42bc19-7669-430c-a7d7-9f5cb7100c24',\n",
              " '4ceaaec6-3e4e-4e71-825f-43f1d56b6078',\n",
              " '2f56b9b5-7fe3-4d6f-94b9-ae03e85f2457',\n",
              " '2e624523-ed79-4445-af0e-98700ca946fe',\n",
              " '07f23368-f941-4d79-9ed3-c01ca2aa04b0',\n",
              " '1c8c0819-fdf0-4110-8193-8e6bee76a151',\n",
              " '0062b1ee-da1a-4af3-b0cd-f58514520962',\n",
              " 'acc9400f-63a8-4165-ad5b-19fe3dfc338d',\n",
              " '6bf66f03-c0e6-48b0-a90f-bbad0b13ffdb',\n",
              " '1f8b0e34-f82d-4391-bea3-84aa49c8dd2d',\n",
              " 'c006b7b1-7299-489c-be07-8f010c5adeea',\n",
              " '304f675b-63f0-4583-9347-5ea09f0a0d31',\n",
              " 'a23a60e7-3ad8-4df4-9bf2-d0618ea001ba',\n",
              " 'f3b617f5-f4e9-4a5a-8efd-30a2551207b3',\n",
              " '00b09fe8-a9d9-4e39-97cc-df8f658ec33b',\n",
              " 'b4efd4d8-aeeb-424f-bbf5-62f7f6b405de',\n",
              " 'cd5045b5-f36c-4386-892f-47c4fbde35eb',\n",
              " 'd79f97ad-a38b-436e-8a1c-57c1755db5ea',\n",
              " 'd162a8d5-af55-47c2-88ad-fe71a4f3abdb',\n",
              " 'fbed0e5c-ae97-4dce-b06f-eecc0bcd01f3',\n",
              " '7c6309ad-d3e9-46a1-a4f1-c4a67b47e20d',\n",
              " '42c129d3-1756-4d05-96a7-7e3f8a068f8d',\n",
              " '2f2a9b2e-3b57-4a7b-870c-97c8e271341b',\n",
              " '01909d1f-e804-4637-b24a-b225c72d482d',\n",
              " '099fbfce-f9b0-4f27-9b46-079b611793d5',\n",
              " 'e6d01a6c-1675-437d-842c-8e72c6b06516',\n",
              " '76bbdfdb-01b6-49e7-9ef3-9bea630f5d51',\n",
              " '694c20ce-a8c4-4525-92fe-db8cca2eb6e3',\n",
              " 'b4dcfd69-6b2d-407a-8738-531a8e89bf8c',\n",
              " '09ea904f-4e17-4ce9-a558-ab6d2cb41995',\n",
              " '41ee7f83-021f-44c9-a394-ca149eec344f',\n",
              " '740150d7-6df9-4b03-b5ee-0b94b8450512',\n",
              " 'c066889e-98c8-4827-9df8-246a7eb3a3b8',\n",
              " 'a707c18d-f54c-46c6-8566-98085defbc13',\n",
              " 'c9d213a1-d939-439e-89b2-24844b47fa6e',\n",
              " 'b22e4f84-61aa-49b8-af46-a24d24d3eb48',\n",
              " 'd2232181-0c4e-4272-953b-c4ec1e07825f',\n",
              " '64f92aa4-a157-42ca-8e12-75554753c30f',\n",
              " 'fd3ae370-5376-49ee-abdc-547c0e1b44c4',\n",
              " '8cc3538d-a52d-47b2-b8a7-bcf9bd4cf55a',\n",
              " 'c2ddc753-c341-44bb-864c-efd40ccf2741',\n",
              " 'ec6b753e-1d9a-4baa-bdaa-1f5046042c5e',\n",
              " 'd060c08d-2366-4375-8b22-3496e9790737',\n",
              " 'cc979f8d-1ca7-4be5-aea8-7056b76e5d5f',\n",
              " '3e59d840-7b08-4216-a49b-93fd751c9776',\n",
              " '0d7c03a2-171c-47e2-b834-892443fbe29f',\n",
              " '20d20ca0-ea0b-4bd2-8380-00fa0df569b9',\n",
              " '8580062d-3372-4058-8e91-cd462f890024',\n",
              " '5aa8bd9a-5fd3-4629-9b00-11681010280e',\n",
              " '8f27981e-98a2-46c6-ac51-151cc28506a8',\n",
              " '512cc278-b1d4-418e-97a7-c9a972488347',\n",
              " 'e683d4a1-fa84-4f02-b7f5-f1030414905a',\n",
              " '97a8f5bd-22d2-4621-a85a-1a032fd3dedd',\n",
              " '8975499e-4395-408a-be63-70c95738835c',\n",
              " 'f7b3a500-bce1-4bca-bee1-149b49fc1ae1',\n",
              " 'c7932c87-73de-4bac-94b5-9b013bee5593',\n",
              " '20100d36-61d7-40f0-aea1-63a38dfad7cc',\n",
              " '09004181-83ed-4755-8a25-3f767effbcf3',\n",
              " 'f206016a-75e4-4fb6-a7c4-37dd8f6020d8',\n",
              " '7859368b-ff89-46f9-908f-8ee97bcfb107',\n",
              " '4027cd57-c736-48bb-b68a-711ff1979b08',\n",
              " '7d0ce871-76bc-4369-ac54-357a5d17e177',\n",
              " '1f98a0de-eed3-4534-a25a-94c7510a9208',\n",
              " 'fc24a878-a2ef-46d3-b9c2-7431cb9d2dc2',\n",
              " '7c053cca-86df-4e8c-85ab-c7b3537562ea',\n",
              " '20599e07-c941-4e89-9dd4-186d3aa6749e',\n",
              " '4db0d283-f8b4-4963-b4e7-bb4afebd1d63',\n",
              " '916de3b0-ab6a-4646-847b-f03a0ac2e924',\n",
              " '9ab68d5c-59d6-4463-a417-878a851c6cde',\n",
              " 'cf20007d-31f4-45bc-8129-5ed428180939',\n",
              " '0e4dc97b-2e4e-432f-be86-cc53ef05e038',\n",
              " 'ade359c1-5a91-4bc8-9af3-96346e376694',\n",
              " 'be4679b0-971c-43bc-ac38-16a4410cd090',\n",
              " '85f90e02-795c-4924-81b1-d4f0c21d7e95',\n",
              " 'b05f6921-f7eb-48c3-81e3-eedd5ce78c3a',\n",
              " 'dcfba45f-3b98-4705-8023-26b71c6a3e3a',\n",
              " '5a558874-016e-4e19-935b-2e9c6caf9dbb',\n",
              " '7e955568-4756-47c1-a33c-dd6b526ae518',\n",
              " 'b3907e20-715f-46d6-b155-189da7f54a88',\n",
              " '4dc68599-2939-47cf-9ba3-f21e0d32686d',\n",
              " '1ef756a2-bd17-49bf-aad4-79946ea28e86',\n",
              " '829f26fb-aa0f-41e3-b5dd-8c89942503d0',\n",
              " 'f3e8527c-3202-4817-8b22-75864288b71a',\n",
              " '33d4d49e-795e-4351-9188-383254604e15',\n",
              " '753800a8-09e4-41e2-9e79-a9b738c5d91c',\n",
              " 'ff9730ac-a835-4dec-8de5-7feb0744e94c',\n",
              " 'cc55eeb0-efb6-4799-959c-8f7ce6c7cd64',\n",
              " 'c34ce186-5b56-436e-99ed-01b56cefb06c',\n",
              " '53a139b7-394a-45a8-8ad2-8640003e2fa7',\n",
              " 'fe4ea4a9-4a1f-4b06-856b-90ecfb6fa944',\n",
              " '4714ab6c-baee-47df-bb2c-4cb061c44487',\n",
              " 'f83cd777-2816-418d-b766-4cff3834a31d',\n",
              " '89b44213-f468-42d3-8594-ff0c5970a63f',\n",
              " 'b173d38f-7e11-4a99-bc82-2fe67f4ca30b',\n",
              " 'c277c65b-abef-4344-967b-14c61efbc86b',\n",
              " '5cfcd7e9-c57c-48d4-aaba-5244e2db6443',\n",
              " '211b4d7b-20e6-480b-83e7-d70ebc3884d1',\n",
              " 'ca5b31c1-b05c-4d72-bd4e-111006677474',\n",
              " '8b254521-9e32-49b7-ae97-62fee11a39d0',\n",
              " '9892c9f3-5d76-4e64-8a32-56ad92eeb969',\n",
              " 'a7ee6127-39bc-4c95-ba41-5c3ad3f0c291',\n",
              " 'e27b8253-cba2-4df1-88ea-8dee20ae35f9']"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "retriever_kwargs = {'k': 1}\n",
        "retriever = vector_store.as_retriever(search_kwargs=retriever_kwargs)"
      ],
      "metadata": {
        "id": "it-stcx2XXeK"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQu53pc68r0m"
      },
      "source": [
        "### 3.4 Design Prompt (2 pt)\n",
        "\n",
        "Design a suitable prompt for RAG."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "F0a7KK0d8r0m"
      },
      "outputs": [],
      "source": [
        "from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate, AIMessagePromptTemplate, SystemMessagePromptTemplate\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "  HumanMessagePromptTemplate.from_template(\"\"\"\n",
        "  Based on this context, answer the questions\n",
        "  context:{context}\n",
        "  question:{question}\n",
        "  \"\"\"),\n",
        "  AIMessagePromptTemplate.from_template(\"Answer:\")\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PRuWSzN08r0n"
      },
      "source": [
        "### 3.5 RAG Chain (3 pt)\n",
        "\n",
        "Design a question from the documents and get the retriever and RAG output for that question."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {
        "id": "0-mq5MLU8r0n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "35ca38db-ef1f-4394-e7ba-a460767b319f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "retrieved document:\n",
            "Figure 1: How best to prepare for an Exam?(a) Fine-tuning based approaches implement \"studying\" by either directly\n",
            "\n",
            "answer:\n",
            " \n",
            "  Sure, I'd be happy to help! Here are some tips on how to prepare for an exam based on the context you provided:\n",
            "\n",
            "1. Fine-tuning based approaches: Implement \"studying\" by either directly practicing problems or indirectly through active recall exercises. This can help improve retention and transfer of knowledge.\n",
            "2. Effort-based modeling: Use models to predict how much effort you need to put in to achieve a certain level of performance on the exam. This can help you plan your study schedule accordingly.\n",
            "3. Goal-setting: Set specific, challenging but achievable goals for yourself, and break them down into smaller subgoals. This can help you stay motivated and focused during your study sessions.\n",
            "4. Spaced repetition: Review material at increasingly longer intervals to improve retention and reduce the likelihood of forgetting.\n",
            "5. Active recall: Engage in active recall by either re-reading notes or quizzing yourself regularly. This can help reinforce knowledge and improve understanding.\n",
            "6. Using mnemonic devices: Create mental images or associations to help you remember key concepts or formulas.\n",
            "7. Practice testing:\n"
          ]
        }
      ],
      "source": [
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "\n",
        "\n",
        "rag_chain = (\n",
        "  prompt\n",
        "  | model\n",
        "  | StrOutputParser()\n",
        ")\n",
        "\n",
        "question = 'How to prepare for an Exam?'\n",
        "retrieved_doc = retriever.get_relevant_documents(question)\n",
        "answer = rag_chain.invoke({'question': question, 'context': retrieved_doc})\n",
        "\n",
        "print(f\"retrieved document:\\n{retrieved_doc[0].page_content}\\n\")\n",
        "print(f\"answer:\\n{answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZbaP6Dm8r0n"
      },
      "source": [
        "### 3.6 Out of Domain Question (4 pt)\n",
        "\n",
        "Ask a question that is not related to documents. Does model answer it? Change your prompt to force model say \"I don't know\" when some one asks out of domains questions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "no, in the prompt we said that, you need to use the context for the answer and dont answer it if its not related to the question."
      ],
      "metadata": {
        "id": "54g1iAdBueMp"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {
        "id": "NIlbvfKw8r0n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "acf71c46-ca2c-46c3-9ff8-87afeaf090dc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "retrieved document:\n",
            "HotPot QA\n",
            "\n",
            "answer:\n",
            " I don't know. The recipe for Ghorme Sabzi is not provided in the given context.\n"
          ]
        }
      ],
      "source": [
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "      HumanMessagePromptTemplate.from_template(\"\"\"\n",
        "      Based on this context, answer the questions\n",
        "      context:{context}\n",
        "      question:{question}\n",
        "      If the answer to the question is not provided in the context, answer \"I dont know\"\n",
        "      \"\"\"),\n",
        "      AIMessagePromptTemplate.from_template(\"Answer:\")\n",
        "])\n",
        "\n",
        "rag_chain = (\n",
        "  prompt\n",
        "  | model\n",
        "  | StrOutputParser()\n",
        ")\n",
        "\n",
        "question = 'How to cook Ghorme Sabzi?'\n",
        "retrieved_doc = retriever.get_relevant_documents(question)\n",
        "answer = rag_chain.invoke({'question': question, 'context': retrieved_doc})\n",
        "\n",
        "print(f\"retrieved document:\\n{retrieved_doc[0].page_content}\\n\")\n",
        "print(f\"answer:\\n{answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOnX9l7F8r0n"
      },
      "source": [
        "### 3.7 The Effect of Temperature (7 pt)\n",
        "\n",
        "RAG performance is highly dependent on model temperature. Explain that low temperature is better or high temperature? For the same prompt, compare the output of the model with low and high temperature.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "When using low temperature, the model will use the context more and its less stochastic\n",
        "\n",
        "but for high temperature, the model is more creative and its outputs will be more random and not deterministic"
      ],
      "metadata": {
        "id": "ht4ldfxnpZLi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Low Temperature"
      ],
      "metadata": {
        "id": "2IUvIb4YocsZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_community.llms import CTransformers\n",
        "\n",
        "model = CTransformers(\n",
        "    model=\"TheBloke/Llama-2-7B-Chat-GGUF\",\n",
        "    model_file=\"llama-2-7b-chat.Q8_0.gguf\",\n",
        "    model_type=\"llama\",\n",
        "    config={\n",
        "        \"gpu_layers\": 50,\n",
        "        \"temperature\": 0.7\n",
        "    }\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81,
          "referenced_widgets": [
            "e96046fd56f04af0a4442740e12ca778",
            "22f527d12f7d4b4d828598b5cda70307",
            "f58993cd201a41918fee11bb67ef7d5e",
            "69b0d74f08bb4e409be97cfcaee6364c",
            "e630be78be3d488ab98d7f6e7e1e9b6d",
            "774ed29a03874aec983b02c5edda69f3",
            "bdcfe9805fb14e2ab3ffc8388158bbcd",
            "f38477b36bf34871ac6b6540ae4a2eaf",
            "71db311289884bdc8392ef4e2eb986ab",
            "938eae8017b4477d91ad780ae17aac52",
            "cd1534869e48402c80997a9800a5e9b8",
            "b3fe3b3f69c14ee6ab0b8a5b0160410b",
            "7f9bc080e5f7408bb087c116f9b6a39e",
            "d1676211c1a24d49ad637c2738a5bd5a",
            "6d14a82020ea4a869a9ccc3762da24ac",
            "b695a1b80ac640d489f24a2bf4e8ccc1",
            "789172d291ca4f7d87977f59376c97ce",
            "9c207b82c9364ffba47b6ca6ebeb8143",
            "50e346400a8a4636a70a0c3ee511bfa0",
            "acf8c78dbab946c88aab939675f265fa",
            "d65568216fef4eed8cefcfb6152d249d",
            "ca4b78dc53824bbf99107382e9dfdddb"
          ]
        },
        "id": "M9qjYmLanQmZ",
        "outputId": "3cf9492a-33c9-43b2-d2ff-2fb776f2b793"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e96046fd56f04af0a4442740e12ca778"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b3fe3b3f69c14ee6ab0b8a5b0160410b"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "MmsBszY-8r0n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "53af386c-4b96-426a-f613-9a18d22651ad"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "retrieved document:\n",
            "Figure 1: How best to prepare for an Exam?(a) Fine-tuning based approaches implement \"studying\" by either directly\n",
            "\n",
            "answer:\n",
            "  How to prepare for an Exam?\n",
            "      Based on the context provided, there are several ways to prepare for an exam. Here are some suggestions:\n",
            "1. Fine-tune your study approach: Implement \"studying\" by either directly practicing what you have learned or indirectly by creating flashcards and quizzing yourself.\n",
            "2. Create a study schedule: Plan out when and how long you will study each day, and stick to it. This can help you stay organized and make the most of your study time.\n",
            "3. Review and practice regularly: Regular review and practice can help solidify information in your long-term memory, making it easier to recall during exams. Try to set aside some time each day to review what you have learned and practice problems or questions related to that material.\n",
            "4. Get enough sleep: Lack of sleep can negatively impact your ability to focus and retain information, which can make it more difficult to perform well on an exam. Make sure to get at least 7-8 hours of sleep each night leading up to the exam.\n",
            "5. Stay organized: Keep all of your study materials, including notes, flashcards, and practice problems, in one place. This can\n"
          ]
        }
      ],
      "source": [
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "      HumanMessagePromptTemplate.from_template(\"\"\"\n",
        "      Based on this context, answer the questions\n",
        "      context:{context}\n",
        "      question:{question}\n",
        "      If the answer to the question is not provided in the context, answer \"I dont know\"\n",
        "      \"\"\"),\n",
        "      AIMessagePromptTemplate.from_template(\"Answer:\")\n",
        "])\n",
        "\n",
        "rag_chain = (\n",
        "  prompt\n",
        "  | model\n",
        "  | StrOutputParser()\n",
        ")\n",
        "\n",
        "question = 'How to prepare for an Exam?'\n",
        "retrieved_doc = retriever.get_relevant_documents(question)\n",
        "answer = rag_chain.invoke({'question': question, 'context': retrieved_doc})\n",
        "\n",
        "print(f\"retrieved document:\\n{retrieved_doc[0].page_content}\\n\")\n",
        "print(f\"answer:\\n{answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### High Temperature"
      ],
      "metadata": {
        "id": "-6-_oH0rogJY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_community.llms import CTransformers\n",
        "\n",
        "model = CTransformers(\n",
        "    model=\"TheBloke/Llama-2-7B-Chat-GGUF\",\n",
        "    model_file=\"llama-2-7b-chat.Q8_0.gguf\",\n",
        "    model_type=\"llama\",\n",
        "    config={\n",
        "        \"gpu_layers\": 50,\n",
        "        \"temperature\": 1.8\n",
        "    }\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210,
          "referenced_widgets": [
            "65a640b227c145b49bac00c67cbca6ac",
            "a7c1051fe8554666b562a22b9f1a6d6e",
            "8db7f6ab06f94d4595cbfa830370b941",
            "b75d3784689440c08f8f6630445ac292",
            "e0e46a97b1d5445bbb178b37817a5b62",
            "99527acda1524f6d8b027719596768d5",
            "f28b0063c1d44dec8861d462a4ff1150",
            "660d95980df9462c924cd6042a2d6003",
            "cc160a83f06b4bc8aa18b42445769a20",
            "203d3f5c5b13418cb3dd4c106c988e29",
            "bcf4150e4edf423c882fab93b87826bf",
            "c1994a174dd74dc99774203e814850ec",
            "65484d6ae60b41a4aaa111b09e0b9f13",
            "e41a317696a04e83b885978b239a3252",
            "290e8d1f25644d398b611ae45118781a",
            "091640256413408697b55c4623472861",
            "422ad20b15224ac1b464513b6c8fbca2",
            "4588a79f561946dcb6cbf684eeeeb6c5",
            "ce2e19f329bb4df1a5695c039acb1d30",
            "bcd4cab3dc424749a85aa34ea8416da0",
            "9c8f66ee0cc745fe83c6560d952669ea",
            "727d99e4123b43aa9860e3f4b79e6739"
          ]
        },
        "id": "7fXJ2Hzsoh4i",
        "outputId": "715cd766-4814-4452-ab14-7832f218e8da"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "65a640b227c145b49bac00c67cbca6ac"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Fetching 1 files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c1994a174dd74dc99774203e814850ec"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_core.runnables import RunnablePassthrough\n",
        "\n",
        "prompt = ChatPromptTemplate.from_messages([\n",
        "      HumanMessagePromptTemplate.from_template(\"\"\"\n",
        "      Based on this context, answer the questions\n",
        "      context:{context}\n",
        "      question:{question}\n",
        "      If the answer to the question is not provided in the context, answer \"I dont know\"\n",
        "      \"\"\"),\n",
        "      AIMessagePromptTemplate.from_template(\"Answer:\")\n",
        "])\n",
        "\n",
        "rag_chain = (\n",
        "  prompt\n",
        "  | model\n",
        "  | StrOutputParser()\n",
        ")\n",
        "\n",
        "question = 'How to prepare for an Exam?'\n",
        "retrieved_doc = retriever.get_relevant_documents(question)\n",
        "answer = rag_chain.invoke({'question': question, 'context': retrieved_doc})\n",
        "\n",
        "print(f\"retrieved document:\\n{retrieved_doc[0].page_content}\\n\")\n",
        "print(f\"answer:\\n{answer}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bFFkrIedotTN",
        "outputId": "439e5801-31b6-4ad8-9d09-b49d22d353d4"
      },
      "execution_count": 14,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "retrieved document:\n",
            "Figure 1: How best to prepare for an Exam?(a) Fine-tuning based approaches implement \"studying\" by either directly\n",
            "\n",
            "answer:\n",
            "  How to prepare for an exam? There are many ways to prepare for an exam, but some of the most effective strategies include:\n",
            "\n",
            "1. Create a study schedule and stick to it: Plan out when and how you will study in advance, and make sure to allocate enough time for each subject or topic.\n",
            "2. Review and organize notes from class: Go through your class notes and textbook, and summarize the key points and concepts covered. Organize them into a logical structure, such as by topic or theme.\n",
            "3. Practice with sample questions: Look for practice exams or past exams in your subject area, and use them to familiarize yourself with the types of questions you may encounter on the exam.\n",
            "4. Get enough sleep: Adequate rest and relaxation are essential for learning and retaining information. Make sure to get plenty of sleep before the exam.\n",
            "5. Stay calm and focused during the exam: Take deep breaths, try to stay hydrated, and avoid getting distracted by noise or other test-takers. Focus on each question one at a time, and make sure to read them carefully before answering.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# AI disclosure"
      ],
      "metadata": {
        "id": "Puug_eNDqAr4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- can you explain what CoT is and how it works ( chain of thoughts) in LLMs\n",
        "- now, please describe Tree-of-Thought ( ToT)\n",
        "-  The self-consistency method contains three steps: (1) prompt a language model using\n",
        "chain-of-thought (CoT) prompting; (2) replace the “greedy decode” in CoT prompting by sampling\n",
        "from the language model’s decoder to generate a diverse set of reasoning paths; and (3) marginalize\n",
        "out the reasoning paths and aggregate by choosing the most consistent answer in the final answer set\n",
        "\n",
        "this is how self - consistency works\n",
        "\n",
        "how can i implement it in code\n",
        "\n",
        "- how can i change this function so  the predicting next word is not greedy decoding but its sampling\n",
        "\n",
        "- User\n",
        "last_sentence.startswith(\"The answer is\")\n",
        "\n",
        "change this so it will lower the last_sentence and compare it with \"the answer is\"\n",
        "- how to load two papers with ArxivLoader\n",
        "Load RAFT and DSPy papers\n",
        "- Usually, each document is constructed from multiple sections\n",
        "\n",
        "write a function named text_splitter to split each document into smaller parts\n",
        "- how can i use a retriever with langchain methods\n",
        "- how can i customize the temperature of the model for inference in this code\n",
        "- how can i set the human and AI part so it would be the conversation\n",
        "\n",
        "prompt: \"What is the capital of Iran\"\n",
        "AI: \"Capital: Tehran\"\n",
        "\n",
        "\n",
        "and explain what is the objective of these two templates\n",
        "- how can i use SystemMessagePromptTemplate so the model would output as json?\n",
        "- how can i convert a dict-shape string into a dict\n",
        "- how can i find the '}' index in a string\n",
        "- i want to create a evaluate_retriever function so i can evaluate the accuray of returned documents\n",
        "the data i want to check the accuracy on and retrived is \"relevant_passage_ids\"\n",
        "write the function for me\n",
        "- User\n",
        "so for example i have a QA dataset\n",
        "and the questions are from a corpus that has some passages and each passage has a id\n",
        "\n",
        "the query for retriver is the Question and the output is the id of passage\n",
        "\n",
        "with this information, write the evaluate_retriever that gets the retriver as input and calculates accurary of it\n",
        "\n",
        "( the QA dataset has a column named relevant_passage_ids, i want to check that for all questions in that dataset, what is the accuracy\n",
        "- how can i create a tfidf retriver that finds the most top k related documents\n",
        "- now, i want to create a semantic_retriver that works like this\n",
        "\n",
        "Semantic retrievers operate by retrieving documents through embeddings. These systems require an embedding model to convert documents into a vector space, and a vector database to find the closest documents to a query. Construct a semantic retriever that utilizes intfloat/e5-base as the embedding model and FAISS for the vector database.\n",
        "\n",
        "write the code for it\n",
        "- how can i use Lora for fine-tuning?\n",
        "- use bit and bytes module to quantize model"
      ],
      "metadata": {
        "id": "-i6wH_ChqGSb"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "hsWD_FojtrH6"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "d9197efa50c0409badd62b208955da99": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_487b57afcff846408329bc223af4af54",
              "IPY_MODEL_0c2cc1ac6e2e447fb289bc674e8991d0",
              "IPY_MODEL_83f741b12891401b86b154768ed0764c"
            ],
            "layout": "IPY_MODEL_7d07b1a185fd4193ad9a587d58372725"
          }
        },
        "487b57afcff846408329bc223af4af54": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6bdac027f1aa46198a2026cc1fbf5e66",
            "placeholder": "​",
            "style": "IPY_MODEL_463b315fe84b4e418a33d389c9027d3f",
            "value": "config.json: 100%"
          }
        },
        "0c2cc1ac6e2e447fb289bc674e8991d0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_12f8d488491e4ba3bc09ad2ea2123839",
            "max": 735,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cadbfc9dbb7945359fe88e9107143113",
            "value": 735
          }
        },
        "83f741b12891401b86b154768ed0764c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bbc0142a866946199dc4d5c65bc2c76a",
            "placeholder": "​",
            "style": "IPY_MODEL_c131a5284fb244649d139277fbaaee14",
            "value": " 735/735 [00:00&lt;00:00, 23.9kB/s]"
          }
        },
        "7d07b1a185fd4193ad9a587d58372725": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6bdac027f1aa46198a2026cc1fbf5e66": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "463b315fe84b4e418a33d389c9027d3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "12f8d488491e4ba3bc09ad2ea2123839": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cadbfc9dbb7945359fe88e9107143113": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bbc0142a866946199dc4d5c65bc2c76a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c131a5284fb244649d139277fbaaee14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "206d22fc95f14cd183845940599fd309": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5d02ac7499b04de38483987896babf9d",
              "IPY_MODEL_9b47bd13a77840f180c86e02a8567661",
              "IPY_MODEL_fdf5e1cecbed4fe2bb9f81e18cde5a20"
            ],
            "layout": "IPY_MODEL_71898a02d09f4b6788a17bf68e3803b3"
          }
        },
        "5d02ac7499b04de38483987896babf9d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_915dc0fb02644206aa5f6b2d794ffd63",
            "placeholder": "​",
            "style": "IPY_MODEL_f02d864bf71144fc8f6c7af23cb03d04",
            "value": "model.safetensors.index.json: 100%"
          }
        },
        "9b47bd13a77840f180c86e02a8567661": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_73718607c5e04b419dfbabb89c53650e",
            "max": 35716,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fd3c8edacd094dff993a3985e543d2d9",
            "value": 35716
          }
        },
        "fdf5e1cecbed4fe2bb9f81e18cde5a20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e40ba557258b40fd94a9569a4fc2fe64",
            "placeholder": "​",
            "style": "IPY_MODEL_6cc9fffaae7f43e7a24dd02128de8adb",
            "value": " 35.7k/35.7k [00:00&lt;00:00, 778kB/s]"
          }
        },
        "71898a02d09f4b6788a17bf68e3803b3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "915dc0fb02644206aa5f6b2d794ffd63": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f02d864bf71144fc8f6c7af23cb03d04": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73718607c5e04b419dfbabb89c53650e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fd3c8edacd094dff993a3985e543d2d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e40ba557258b40fd94a9569a4fc2fe64": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6cc9fffaae7f43e7a24dd02128de8adb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2f4c3b4360fe4b40aec75639ef04b0ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bd4e39a7cb3840278e46a57f58ad549a",
              "IPY_MODEL_319f26fc6d974ecab4c5181cdd518db8",
              "IPY_MODEL_551a1787efc64e2a965d1ec2b7b6a33e"
            ],
            "layout": "IPY_MODEL_2ed2e8eeda0d4a2984aa946dae2a51f9"
          }
        },
        "bd4e39a7cb3840278e46a57f58ad549a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_99f9ad06f6204cac9cc56cbfd0f60e0a",
            "placeholder": "​",
            "style": "IPY_MODEL_ecfc5471abb84f20b07a24a8bc7fd5d8",
            "value": "Downloading shards: 100%"
          }
        },
        "319f26fc6d974ecab4c5181cdd518db8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e6902997ac014b2ca23c9bca6930c1df",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d0d8ba2714754022aa5c64c3a74679ea",
            "value": 2
          }
        },
        "551a1787efc64e2a965d1ec2b7b6a33e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ccc2c822359a4daf8170c9f0eb4f43bb",
            "placeholder": "​",
            "style": "IPY_MODEL_5c74fdf30b764487aa0c1fdee4f4a7d8",
            "value": " 2/2 [00:46&lt;00:00, 19.94s/it]"
          }
        },
        "2ed2e8eeda0d4a2984aa946dae2a51f9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "99f9ad06f6204cac9cc56cbfd0f60e0a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ecfc5471abb84f20b07a24a8bc7fd5d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e6902997ac014b2ca23c9bca6930c1df": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d0d8ba2714754022aa5c64c3a74679ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ccc2c822359a4daf8170c9f0eb4f43bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5c74fdf30b764487aa0c1fdee4f4a7d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a7bee5212626426eac81918ee2822c76": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3391229bd6a04425bac90c64bac9ea5c",
              "IPY_MODEL_b3391543994740e0920d5a05e55158c8",
              "IPY_MODEL_bb8da3b5a17f41b39218b7a6fd4ba4ee"
            ],
            "layout": "IPY_MODEL_1e23a34c29364d80ac315167492e15c5"
          }
        },
        "3391229bd6a04425bac90c64bac9ea5c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e26e8fb829114bfa9f639d8dc048764a",
            "placeholder": "​",
            "style": "IPY_MODEL_99a7edbb40884ea19ebc806409c5368f",
            "value": "model-00001-of-00002.safetensors: 100%"
          }
        },
        "b3391543994740e0920d5a05e55158c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f0471388d80e49deb2c00dd4b46982eb",
            "max": 4995584424,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ec71c86bc8094ac193ebd22e8e8b52a0",
            "value": 4995584424
          }
        },
        "bb8da3b5a17f41b39218b7a6fd4ba4ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_51eddce4bcd04bf2ac2255b64d0d068a",
            "placeholder": "​",
            "style": "IPY_MODEL_6f08974e738a4977969eb8c5e5077378",
            "value": " 5.00G/5.00G [00:42&lt;00:00, 151MB/s]"
          }
        },
        "1e23a34c29364d80ac315167492e15c5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e26e8fb829114bfa9f639d8dc048764a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "99a7edbb40884ea19ebc806409c5368f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f0471388d80e49deb2c00dd4b46982eb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec71c86bc8094ac193ebd22e8e8b52a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "51eddce4bcd04bf2ac2255b64d0d068a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6f08974e738a4977969eb8c5e5077378": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3ba5c4ff1d8047c4b77cc6d976f7ba2e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_924aca0af5fd421fbd64abae3963e858",
              "IPY_MODEL_9dee6aafacf045f4add5a170765a29e8",
              "IPY_MODEL_9f9440cfefd94619ae41a473741c8c3a"
            ],
            "layout": "IPY_MODEL_33de55bda78142a4a5f9a1e132797836"
          }
        },
        "924aca0af5fd421fbd64abae3963e858": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_95fadf5577694edc9c655683f291d06b",
            "placeholder": "​",
            "style": "IPY_MODEL_ca29b7d5749c4740b1879ed83e0024e7",
            "value": "model-00002-of-00002.safetensors: 100%"
          }
        },
        "9dee6aafacf045f4add5a170765a29e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_99119d0591eb49aeb261fcbac15d8489",
            "max": 563832976,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_be50ebf187594348a3355cd96414a4f7",
            "value": 563832976
          }
        },
        "9f9440cfefd94619ae41a473741c8c3a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bdc70d5af3a44a6dbfdf41ef63f86b5e",
            "placeholder": "​",
            "style": "IPY_MODEL_b4dd617c8fb047c89378dcd0be5bfeef",
            "value": " 564M/564M [00:03&lt;00:00, 131MB/s]"
          }
        },
        "33de55bda78142a4a5f9a1e132797836": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "95fadf5577694edc9c655683f291d06b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca29b7d5749c4740b1879ed83e0024e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "99119d0591eb49aeb261fcbac15d8489": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "be50ebf187594348a3355cd96414a4f7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bdc70d5af3a44a6dbfdf41ef63f86b5e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b4dd617c8fb047c89378dcd0be5bfeef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ef18c41010c644969f3cc2ff4cc78a04": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6c761c0dead6477faa7809abbc5dc8c7",
              "IPY_MODEL_544a261fb49d4eb8bed84c3e15a3e17d",
              "IPY_MODEL_cf66918d62d34a2ca21767c27d83c2c6"
            ],
            "layout": "IPY_MODEL_b02f04e6801e464783abc9ad7b4a2ac8"
          }
        },
        "6c761c0dead6477faa7809abbc5dc8c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6561ce4bcf9c47168bbb02e195479907",
            "placeholder": "​",
            "style": "IPY_MODEL_61ea2a70ac024677bb9b28d1140c88c1",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "544a261fb49d4eb8bed84c3e15a3e17d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_586d91b90d4a4eb1bb02ec65989b49ed",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a8aa72b64dd84ff39abc919943b68c12",
            "value": 2
          }
        },
        "cf66918d62d34a2ca21767c27d83c2c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_355b58cae4d84604a324e52e492970a5",
            "placeholder": "​",
            "style": "IPY_MODEL_f3470052e4304f3cb1b741fd7a2a9c20",
            "value": " 2/2 [00:27&lt;00:00, 12.02s/it]"
          }
        },
        "b02f04e6801e464783abc9ad7b4a2ac8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6561ce4bcf9c47168bbb02e195479907": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61ea2a70ac024677bb9b28d1140c88c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "586d91b90d4a4eb1bb02ec65989b49ed": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a8aa72b64dd84ff39abc919943b68c12": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "355b58cae4d84604a324e52e492970a5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f3470052e4304f3cb1b741fd7a2a9c20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "981a04cde7194b54b067b8f3ee646bb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e7874c315554454694df390c280ea5d2",
              "IPY_MODEL_4a8b11f843a74aa2bd22cc8abdf0a643",
              "IPY_MODEL_0ddbc70b75e14c05a75eac4b692acdd8"
            ],
            "layout": "IPY_MODEL_73076eedd8fe4a71aeaf978b7a1645cc"
          }
        },
        "e7874c315554454694df390c280ea5d2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c6104917058446df9c66e5586b3b0bd1",
            "placeholder": "​",
            "style": "IPY_MODEL_b53d4cecef07433e8c1fe9f0af048189",
            "value": "generation_config.json: 100%"
          }
        },
        "4a8b11f843a74aa2bd22cc8abdf0a643": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_31ed325291e34b0181a4a1e6711a05ce",
            "max": 124,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3e384669c3a04b9c81111d92217ea4c4",
            "value": 124
          }
        },
        "0ddbc70b75e14c05a75eac4b692acdd8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b0e6732f89b142a1ba1766db44aeab37",
            "placeholder": "​",
            "style": "IPY_MODEL_453b70ae65ee472fbbefaa064a333c65",
            "value": " 124/124 [00:00&lt;00:00, 2.98kB/s]"
          }
        },
        "73076eedd8fe4a71aeaf978b7a1645cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c6104917058446df9c66e5586b3b0bd1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b53d4cecef07433e8c1fe9f0af048189": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "31ed325291e34b0181a4a1e6711a05ce": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3e384669c3a04b9c81111d92217ea4c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b0e6732f89b142a1ba1766db44aeab37": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "453b70ae65ee472fbbefaa064a333c65": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "70922c1e59cd417a91c3ca0d96ed4949": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_118c8bbd5f384eb9a62665211d686ef6",
              "IPY_MODEL_a503f3c8a0a84b09a1b1ace6ed317610",
              "IPY_MODEL_c0f6f7797b7b4af7a8057be57b4cb41a"
            ],
            "layout": "IPY_MODEL_eb0abe109f074fb1a30ae19e9ec7c849"
          }
        },
        "118c8bbd5f384eb9a62665211d686ef6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4b3121750a5b429c98c57f39fd7200b2",
            "placeholder": "​",
            "style": "IPY_MODEL_2b11ce7bd0f3440fa0b5cb325590a637",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "a503f3c8a0a84b09a1b1ace6ed317610": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2c7dd0c7ad6e422cb085239a282b4cdf",
            "max": 7339,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c4fdab7ee2414971bc315380d867a214",
            "value": 7339
          }
        },
        "c0f6f7797b7b4af7a8057be57b4cb41a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_92ef96d1c1f044e0b0066d66d4cb62a4",
            "placeholder": "​",
            "style": "IPY_MODEL_ca22b928b9e746a6b650786f9af7b54c",
            "value": " 7.34k/7.34k [00:00&lt;00:00, 247kB/s]"
          }
        },
        "eb0abe109f074fb1a30ae19e9ec7c849": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4b3121750a5b429c98c57f39fd7200b2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2b11ce7bd0f3440fa0b5cb325590a637": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2c7dd0c7ad6e422cb085239a282b4cdf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c4fdab7ee2414971bc315380d867a214": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "92ef96d1c1f044e0b0066d66d4cb62a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca22b928b9e746a6b650786f9af7b54c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5531392cb8074487beb6506a80667ce1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3c05a956926740a492dea9a651d900e9",
              "IPY_MODEL_76888876c4ef4db490f3af865a27759d",
              "IPY_MODEL_4d21e662de8f49cb8e33122ad70fad1d"
            ],
            "layout": "IPY_MODEL_eba96b522a5041a8b28bb4100cdf143c"
          }
        },
        "3c05a956926740a492dea9a651d900e9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6d4ddbdf309746c782cc9563051b8159",
            "placeholder": "​",
            "style": "IPY_MODEL_1de81640ead54f8eb8c6fe3a96590bb3",
            "value": "vocab.json: 100%"
          }
        },
        "76888876c4ef4db490f3af865a27759d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f2931e0a2c0d4267bfbb32369090a494",
            "max": 798156,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_883a4e8f600b489bb033a903a04c0432",
            "value": 798156
          }
        },
        "4d21e662de8f49cb8e33122ad70fad1d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_58ee935a5114426ebd9869b57ab1ee77",
            "placeholder": "​",
            "style": "IPY_MODEL_8cf2fdfe310846e1ac02247c3f838004",
            "value": " 798k/798k [00:00&lt;00:00, 9.82MB/s]"
          }
        },
        "eba96b522a5041a8b28bb4100cdf143c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6d4ddbdf309746c782cc9563051b8159": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1de81640ead54f8eb8c6fe3a96590bb3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f2931e0a2c0d4267bfbb32369090a494": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "883a4e8f600b489bb033a903a04c0432": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "58ee935a5114426ebd9869b57ab1ee77": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8cf2fdfe310846e1ac02247c3f838004": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e3219593faf6410b9b02705206583005": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2fd9622bea434537b525730b70987c6f",
              "IPY_MODEL_7043923b92a0433398571bc5c5117acf",
              "IPY_MODEL_9b545098dbdf4821a0289e43f24b4195"
            ],
            "layout": "IPY_MODEL_8bf15b4d9b82411181448ccf6b868a9c"
          }
        },
        "2fd9622bea434537b525730b70987c6f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3597c865672642edabbb1bad0f6e8e75",
            "placeholder": "​",
            "style": "IPY_MODEL_eaddca42b52f4df7b286401fd0306e12",
            "value": "merges.txt: 100%"
          }
        },
        "7043923b92a0433398571bc5c5117acf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4d4033c56e244e669680f6b0ee87376e",
            "max": 456318,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2a6ffe20f4644c778f02771e23041aa7",
            "value": 456318
          }
        },
        "9b545098dbdf4821a0289e43f24b4195": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_421e1c82100748898a337fced5746935",
            "placeholder": "​",
            "style": "IPY_MODEL_8df25b079a934ecfb0bed7f99d13036f",
            "value": " 456k/456k [00:00&lt;00:00, 16.8MB/s]"
          }
        },
        "8bf15b4d9b82411181448ccf6b868a9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3597c865672642edabbb1bad0f6e8e75": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eaddca42b52f4df7b286401fd0306e12": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4d4033c56e244e669680f6b0ee87376e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a6ffe20f4644c778f02771e23041aa7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "421e1c82100748898a337fced5746935": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8df25b079a934ecfb0bed7f99d13036f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "977b61cffd364545a70a37b3456b97fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_98dfe25e5a4d43deabbbe3edd4abe458",
              "IPY_MODEL_c30a6f3c5fae430bb8d75dbf7a17218d",
              "IPY_MODEL_eb861d5ce9554365939d9bf7c6e184d3"
            ],
            "layout": "IPY_MODEL_8606cda6a4b1423fb5a65db1a4ca8ba9"
          }
        },
        "98dfe25e5a4d43deabbbe3edd4abe458": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2ba408e5c20946dabb6a86ed8ad042d9",
            "placeholder": "​",
            "style": "IPY_MODEL_6daa92960107487581486af97777ba0a",
            "value": "tokenizer.json: 100%"
          }
        },
        "c30a6f3c5fae430bb8d75dbf7a17218d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_35a69c629f6046f9ab8569266e669cae",
            "max": 2114924,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d6c7fad2417741e5bba9573e3ecb14b5",
            "value": 2114924
          }
        },
        "eb861d5ce9554365939d9bf7c6e184d3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_52b58700ad9c4c9ebd1cfe526478d92a",
            "placeholder": "​",
            "style": "IPY_MODEL_abba72ae7655477b8ebabe2bf7ed1ccd",
            "value": " 2.11M/2.11M [00:00&lt;00:00, 3.72MB/s]"
          }
        },
        "8606cda6a4b1423fb5a65db1a4ca8ba9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2ba408e5c20946dabb6a86ed8ad042d9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6daa92960107487581486af97777ba0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "35a69c629f6046f9ab8569266e669cae": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d6c7fad2417741e5bba9573e3ecb14b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "52b58700ad9c4c9ebd1cfe526478d92a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "abba72ae7655477b8ebabe2bf7ed1ccd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6681dafe4f7e4f3182792a5cffe8173f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_33e1490306f04a55865154776988dd16",
              "IPY_MODEL_3ac141263cd84ea890e6a523769a3aae",
              "IPY_MODEL_31e04955761a437187a60e51740b266b"
            ],
            "layout": "IPY_MODEL_ddca84c3ef8e46668c142b9cb3ee0f9a"
          }
        },
        "33e1490306f04a55865154776988dd16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0648f13f79ac46e28dab74bb15a81274",
            "placeholder": "​",
            "style": "IPY_MODEL_74d33146d0b045aeb5c7f3ebe23f9c91",
            "value": "added_tokens.json: 100%"
          }
        },
        "3ac141263cd84ea890e6a523769a3aae": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_74b342af82fc46e2815f452644eb78a2",
            "max": 1080,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3ef09fae339f463789b21c62f8c4e932",
            "value": 1080
          }
        },
        "31e04955761a437187a60e51740b266b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da0d56d661e249f884d0c15d6ae929c3",
            "placeholder": "​",
            "style": "IPY_MODEL_7828e6d687fc43e49524db69901fa39e",
            "value": " 1.08k/1.08k [00:00&lt;00:00, 36.5kB/s]"
          }
        },
        "ddca84c3ef8e46668c142b9cb3ee0f9a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0648f13f79ac46e28dab74bb15a81274": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74d33146d0b045aeb5c7f3ebe23f9c91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "74b342af82fc46e2815f452644eb78a2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3ef09fae339f463789b21c62f8c4e932": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "da0d56d661e249f884d0c15d6ae929c3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7828e6d687fc43e49524db69901fa39e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e3e509dc4465408f8caaa01e2e1dc3b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_75b98e3ab1234fe7b920c8d0d89b5bed",
              "IPY_MODEL_71badbcf6f574960a1410d4f75a62eb0",
              "IPY_MODEL_fa100600f25b46f3b8cdd735a9c26bfb"
            ],
            "layout": "IPY_MODEL_0a8904fb77ac4151be7c52407afc6fa9"
          }
        },
        "75b98e3ab1234fe7b920c8d0d89b5bed": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fc6b47254636478cb51347e206766efe",
            "placeholder": "​",
            "style": "IPY_MODEL_bc371ea22e8549429ab605ae60e7126d",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "71badbcf6f574960a1410d4f75a62eb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8c2ba792ebf14cc49d0c9045967b8fc9",
            "max": 99,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ebfbbe5f7c71453ab530dbab11dd317a",
            "value": 99
          }
        },
        "fa100600f25b46f3b8cdd735a9c26bfb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_52784b87399b4dee92e48862c7442746",
            "placeholder": "​",
            "style": "IPY_MODEL_1cb4ac14d7da4178b8ec207946facd6c",
            "value": " 99.0/99.0 [00:00&lt;00:00, 3.69kB/s]"
          }
        },
        "0a8904fb77ac4151be7c52407afc6fa9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fc6b47254636478cb51347e206766efe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bc371ea22e8549429ab605ae60e7126d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8c2ba792ebf14cc49d0c9045967b8fc9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ebfbbe5f7c71453ab530dbab11dd317a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "52784b87399b4dee92e48862c7442746": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1cb4ac14d7da4178b8ec207946facd6c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b075cd11497d46ffa0b09c4fbe04dfb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5fdbc9d55ddb47c79c2dca69049fbbec",
              "IPY_MODEL_7e0109f5225747eda16efdf2daee1654",
              "IPY_MODEL_f83ad35247814c6cb926943a3c425c7f"
            ],
            "layout": "IPY_MODEL_6125807b32964b969cceadb16b6fd056"
          }
        },
        "5fdbc9d55ddb47c79c2dca69049fbbec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d415ef1b1292483480d325cc6e425f5c",
            "placeholder": "​",
            "style": "IPY_MODEL_1a82a2d7ea744508ad1fe7e7a8cb5f67",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "7e0109f5225747eda16efdf2daee1654": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_60903ed736a94152a0dceb34c726c2fb",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_26efd76a115f4f54ab2f3bab056cc0a0",
            "value": 2
          }
        },
        "f83ad35247814c6cb926943a3c425c7f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_708c63985d0f4df881d49e7d700dccc7",
            "placeholder": "​",
            "style": "IPY_MODEL_b03e66157d5b4f2a90a249d0f74e1200",
            "value": " 2/2 [00:17&lt;00:00,  7.54s/it]"
          }
        },
        "6125807b32964b969cceadb16b6fd056": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d415ef1b1292483480d325cc6e425f5c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1a82a2d7ea744508ad1fe7e7a8cb5f67": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "60903ed736a94152a0dceb34c726c2fb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "26efd76a115f4f54ab2f3bab056cc0a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "708c63985d0f4df881d49e7d700dccc7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b03e66157d5b4f2a90a249d0f74e1200": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ab5d7785310b4478acde258ade28cdb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_36d8746835364efeb8b8728b4ef79192",
              "IPY_MODEL_92302e928c5542f69eb217f6704285b5",
              "IPY_MODEL_ee06008b781043a7a646a9522a9697f2"
            ],
            "layout": "IPY_MODEL_43d0ddb27ddf40f5b99c20c98d2ef08b"
          }
        },
        "36d8746835364efeb8b8728b4ef79192": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_be1f01f0cff549d99412fd87edad2af1",
            "placeholder": "​",
            "style": "IPY_MODEL_bdfea40c721e459798533b9205dc51ae",
            "value": "Map: 100%"
          }
        },
        "92302e928c5542f69eb217f6704285b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_af20e0a306634e9199a7702cfaff18f4",
            "max": 20726,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_318d6257c7204e0f90a68a464b28be3f",
            "value": 20726
          }
        },
        "ee06008b781043a7a646a9522a9697f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_14236a65e2ef4dffb3d4ab7390d52b6d",
            "placeholder": "​",
            "style": "IPY_MODEL_9e998baf3970435cbe1a92229192e3c1",
            "value": " 20726/20726 [00:36&lt;00:00, 569.69 examples/s]"
          }
        },
        "43d0ddb27ddf40f5b99c20c98d2ef08b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "be1f01f0cff549d99412fd87edad2af1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdfea40c721e459798533b9205dc51ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "af20e0a306634e9199a7702cfaff18f4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "318d6257c7204e0f90a68a464b28be3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "14236a65e2ef4dffb3d4ab7390d52b6d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9e998baf3970435cbe1a92229192e3c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c56c8819b6a34432a9d38ea212d5ae2e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5b043b489c484e9abf4e90235d73156d",
              "IPY_MODEL_24e57c6d42134d1cbcada8ce45e9ab37",
              "IPY_MODEL_1ad4c3a4e56045c49fa4ae2ae27d6a26"
            ],
            "layout": "IPY_MODEL_7ae8f9197d704917909c059e7170f0b0"
          }
        },
        "5b043b489c484e9abf4e90235d73156d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d315fc26469b45fcae77942d52f282f7",
            "placeholder": "​",
            "style": "IPY_MODEL_5b9939c9dd814d58b20a26eb5a17cf7d",
            "value": "Fetching 1 files: 100%"
          }
        },
        "24e57c6d42134d1cbcada8ce45e9ab37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8ce362ae2c8b4a1aaf818b29a8cb7f43",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3a8ec0a77fd446b393bc7b5016a956e8",
            "value": 1
          }
        },
        "1ad4c3a4e56045c49fa4ae2ae27d6a26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6d61f95ee5944829afceca7214c591ec",
            "placeholder": "​",
            "style": "IPY_MODEL_f2d59795db0843efb504c75986b46829",
            "value": " 1/1 [00:00&lt;00:00,  4.24it/s]"
          }
        },
        "7ae8f9197d704917909c059e7170f0b0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d315fc26469b45fcae77942d52f282f7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5b9939c9dd814d58b20a26eb5a17cf7d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8ce362ae2c8b4a1aaf818b29a8cb7f43": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3a8ec0a77fd446b393bc7b5016a956e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6d61f95ee5944829afceca7214c591ec": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f2d59795db0843efb504c75986b46829": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ee41dcc7c4874e5f92e0b7c908ef6d67": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7e2c82fd4c0144aa8c49fc501cf81fe1",
              "IPY_MODEL_7a61c3599f104d23be9baac628f328b0",
              "IPY_MODEL_f3fd3da0d4354e53a626d8ab7c7fa8d9"
            ],
            "layout": "IPY_MODEL_69f0bf0e1cbb4255940139337a7d9322"
          }
        },
        "7e2c82fd4c0144aa8c49fc501cf81fe1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dc632adc0613406893c8d445c20ca47f",
            "placeholder": "​",
            "style": "IPY_MODEL_3080d39f1c184b1c9487d6b4ae4e6218",
            "value": "config.json: 100%"
          }
        },
        "7a61c3599f104d23be9baac628f328b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_687964f103ac45b08c8a94dd95345a5f",
            "max": 29,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9fe0d604bad94c549f9428734e5356e4",
            "value": 29
          }
        },
        "f3fd3da0d4354e53a626d8ab7c7fa8d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c5949e898465400a821e7dabc2b08d79",
            "placeholder": "​",
            "style": "IPY_MODEL_1ad2eb5135554bf5a4515eda1cd0a1a0",
            "value": " 29.0/29.0 [00:00&lt;00:00, 1.68kB/s]"
          }
        },
        "69f0bf0e1cbb4255940139337a7d9322": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc632adc0613406893c8d445c20ca47f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3080d39f1c184b1c9487d6b4ae4e6218": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "687964f103ac45b08c8a94dd95345a5f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9fe0d604bad94c549f9428734e5356e4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c5949e898465400a821e7dabc2b08d79": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1ad2eb5135554bf5a4515eda1cd0a1a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9325232b43974a068bff52cdaa9583b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_837ee2ac429041599fba645ed1a11ea2",
              "IPY_MODEL_2f57faad5e3141d181f0f910887827b2",
              "IPY_MODEL_fe6da49d5618403fbb6cd8e6eae9c9fd"
            ],
            "layout": "IPY_MODEL_f289ed2050504f468198a90402963ac8"
          }
        },
        "837ee2ac429041599fba645ed1a11ea2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_572b07ef47a640efa467f06e1b03da3d",
            "placeholder": "​",
            "style": "IPY_MODEL_4e5ae24bc8ff4168b5b3cb6d6f7cb9c1",
            "value": "Fetching 1 files: 100%"
          }
        },
        "2f57faad5e3141d181f0f910887827b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_efc46e9edc8b40c38f581ef7b0465c65",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_61ce422754414fbeb0b152f4cfe73c84",
            "value": 1
          }
        },
        "fe6da49d5618403fbb6cd8e6eae9c9fd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_78af3d45ad724d1c9e4195cb8a985c29",
            "placeholder": "​",
            "style": "IPY_MODEL_d5210899bec147389fe8621c5a686205",
            "value": " 1/1 [01:12&lt;00:00, 72.38s/it]"
          }
        },
        "f289ed2050504f468198a90402963ac8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "572b07ef47a640efa467f06e1b03da3d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4e5ae24bc8ff4168b5b3cb6d6f7cb9c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "efc46e9edc8b40c38f581ef7b0465c65": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61ce422754414fbeb0b152f4cfe73c84": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "78af3d45ad724d1c9e4195cb8a985c29": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d5210899bec147389fe8621c5a686205": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "eee96c4b44a647d0851a0bf9361f2c52": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_71116db573064aa5859db7b7acc8debc",
              "IPY_MODEL_c86d9fbcbb9847399a7f93c0dc38b011",
              "IPY_MODEL_852da029c24f4761bb8a506f1f82a0cb"
            ],
            "layout": "IPY_MODEL_7e82d6be6dfd49ebb7b0f930ff93627f"
          }
        },
        "71116db573064aa5859db7b7acc8debc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d256afe34cb34aac8fb99233c414aa4a",
            "placeholder": "​",
            "style": "IPY_MODEL_ff040a214df34fc4951cce30741303bc",
            "value": "llama-2-7b-chat.Q8_0.gguf: 100%"
          }
        },
        "c86d9fbcbb9847399a7f93c0dc38b011": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e3550dfac171450c907a51cdfff8d403",
            "max": 7161089728,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2c3dc36f449146ee977ce9829089c779",
            "value": 7161089728
          }
        },
        "852da029c24f4761bb8a506f1f82a0cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8fca73019b6f41af8c17aa509ef15c79",
            "placeholder": "​",
            "style": "IPY_MODEL_82569ad40e5e408dae31fa614892bb4a",
            "value": " 7.16G/7.16G [01:12&lt;00:00, 166MB/s]"
          }
        },
        "7e82d6be6dfd49ebb7b0f930ff93627f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d256afe34cb34aac8fb99233c414aa4a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ff040a214df34fc4951cce30741303bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e3550dfac171450c907a51cdfff8d403": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2c3dc36f449146ee977ce9829089c779": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8fca73019b6f41af8c17aa509ef15c79": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82569ad40e5e408dae31fa614892bb4a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e96046fd56f04af0a4442740e12ca778": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_22f527d12f7d4b4d828598b5cda70307",
              "IPY_MODEL_f58993cd201a41918fee11bb67ef7d5e",
              "IPY_MODEL_69b0d74f08bb4e409be97cfcaee6364c"
            ],
            "layout": "IPY_MODEL_e630be78be3d488ab98d7f6e7e1e9b6d"
          }
        },
        "22f527d12f7d4b4d828598b5cda70307": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_774ed29a03874aec983b02c5edda69f3",
            "placeholder": "​",
            "style": "IPY_MODEL_bdcfe9805fb14e2ab3ffc8388158bbcd",
            "value": "Fetching 1 files: 100%"
          }
        },
        "f58993cd201a41918fee11bb67ef7d5e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f38477b36bf34871ac6b6540ae4a2eaf",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_71db311289884bdc8392ef4e2eb986ab",
            "value": 1
          }
        },
        "69b0d74f08bb4e409be97cfcaee6364c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_938eae8017b4477d91ad780ae17aac52",
            "placeholder": "​",
            "style": "IPY_MODEL_cd1534869e48402c80997a9800a5e9b8",
            "value": " 1/1 [00:00&lt;00:00, 19.13it/s]"
          }
        },
        "e630be78be3d488ab98d7f6e7e1e9b6d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "774ed29a03874aec983b02c5edda69f3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdcfe9805fb14e2ab3ffc8388158bbcd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f38477b36bf34871ac6b6540ae4a2eaf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "71db311289884bdc8392ef4e2eb986ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "938eae8017b4477d91ad780ae17aac52": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd1534869e48402c80997a9800a5e9b8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b3fe3b3f69c14ee6ab0b8a5b0160410b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7f9bc080e5f7408bb087c116f9b6a39e",
              "IPY_MODEL_d1676211c1a24d49ad637c2738a5bd5a",
              "IPY_MODEL_6d14a82020ea4a869a9ccc3762da24ac"
            ],
            "layout": "IPY_MODEL_b695a1b80ac640d489f24a2bf4e8ccc1"
          }
        },
        "7f9bc080e5f7408bb087c116f9b6a39e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_789172d291ca4f7d87977f59376c97ce",
            "placeholder": "​",
            "style": "IPY_MODEL_9c207b82c9364ffba47b6ca6ebeb8143",
            "value": "Fetching 1 files: 100%"
          }
        },
        "d1676211c1a24d49ad637c2738a5bd5a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_50e346400a8a4636a70a0c3ee511bfa0",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_acf8c78dbab946c88aab939675f265fa",
            "value": 1
          }
        },
        "6d14a82020ea4a869a9ccc3762da24ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d65568216fef4eed8cefcfb6152d249d",
            "placeholder": "​",
            "style": "IPY_MODEL_ca4b78dc53824bbf99107382e9dfdddb",
            "value": " 1/1 [00:00&lt;00:00, 18.22it/s]"
          }
        },
        "b695a1b80ac640d489f24a2bf4e8ccc1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "789172d291ca4f7d87977f59376c97ce": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9c207b82c9364ffba47b6ca6ebeb8143": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "50e346400a8a4636a70a0c3ee511bfa0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "acf8c78dbab946c88aab939675f265fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d65568216fef4eed8cefcfb6152d249d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca4b78dc53824bbf99107382e9dfdddb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "65a640b227c145b49bac00c67cbca6ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a7c1051fe8554666b562a22b9f1a6d6e",
              "IPY_MODEL_8db7f6ab06f94d4595cbfa830370b941",
              "IPY_MODEL_b75d3784689440c08f8f6630445ac292"
            ],
            "layout": "IPY_MODEL_e0e46a97b1d5445bbb178b37817a5b62"
          }
        },
        "a7c1051fe8554666b562a22b9f1a6d6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_99527acda1524f6d8b027719596768d5",
            "placeholder": "​",
            "style": "IPY_MODEL_f28b0063c1d44dec8861d462a4ff1150",
            "value": "Fetching 1 files: 100%"
          }
        },
        "8db7f6ab06f94d4595cbfa830370b941": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_660d95980df9462c924cd6042a2d6003",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cc160a83f06b4bc8aa18b42445769a20",
            "value": 1
          }
        },
        "b75d3784689440c08f8f6630445ac292": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_203d3f5c5b13418cb3dd4c106c988e29",
            "placeholder": "​",
            "style": "IPY_MODEL_bcf4150e4edf423c882fab93b87826bf",
            "value": " 1/1 [00:00&lt;00:00, 17.53it/s]"
          }
        },
        "e0e46a97b1d5445bbb178b37817a5b62": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "99527acda1524f6d8b027719596768d5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f28b0063c1d44dec8861d462a4ff1150": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "660d95980df9462c924cd6042a2d6003": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc160a83f06b4bc8aa18b42445769a20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "203d3f5c5b13418cb3dd4c106c988e29": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bcf4150e4edf423c882fab93b87826bf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c1994a174dd74dc99774203e814850ec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_65484d6ae60b41a4aaa111b09e0b9f13",
              "IPY_MODEL_e41a317696a04e83b885978b239a3252",
              "IPY_MODEL_290e8d1f25644d398b611ae45118781a"
            ],
            "layout": "IPY_MODEL_091640256413408697b55c4623472861"
          }
        },
        "65484d6ae60b41a4aaa111b09e0b9f13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_422ad20b15224ac1b464513b6c8fbca2",
            "placeholder": "​",
            "style": "IPY_MODEL_4588a79f561946dcb6cbf684eeeeb6c5",
            "value": "Fetching 1 files: 100%"
          }
        },
        "e41a317696a04e83b885978b239a3252": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ce2e19f329bb4df1a5695c039acb1d30",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bcd4cab3dc424749a85aa34ea8416da0",
            "value": 1
          }
        },
        "290e8d1f25644d398b611ae45118781a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9c8f66ee0cc745fe83c6560d952669ea",
            "placeholder": "​",
            "style": "IPY_MODEL_727d99e4123b43aa9860e3f4b79e6739",
            "value": " 1/1 [00:00&lt;00:00, 28.16it/s]"
          }
        },
        "091640256413408697b55c4623472861": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "422ad20b15224ac1b464513b6c8fbca2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4588a79f561946dcb6cbf684eeeeb6c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ce2e19f329bb4df1a5695c039acb1d30": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bcd4cab3dc424749a85aa34ea8416da0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "9c8f66ee0cc745fe83c6560d952669ea": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "727d99e4123b43aa9860e3f4b79e6739": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}