cookbooks/1-rhubarb-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document classification\n",
    "---\n",
    "\n",
    "You can perform a more accurate, and cost effective document classification with Rhubarb's vector sampling functionality. Internally, Rhubarb uses Amazon Titan Multi-modal embedding model for this purpose. Here are the high level steps you will follow to setup a classifier, and then run document classifications with it.\n",
    "\n",
    "### How are the documents classified?\n",
    "---\n",
    "The premise of classification based on vector sampling relies on the fact that you have a small set of labeled documents and their corresponding vectors (embeddings). Given this sample set, you can generate vector embeddings of a new document and perform similarity check with your sample set to determine which type of document the new document closely resembles to. \n",
    "\n",
    "For that, Rhubarb offers two methods of choice- \n",
    "1. **Cosine Similarity** (`cosine`) to measure the similarity of a new document's vector embeddings with the sample set of labeled vectors. Cosine Similarity measures the cosine of the angle between two non-zero vectors in a multi-dimensional space. It's derived from the dot product of the vectors divided by the product of their magnitudes (norms). For each class, it finds the maximum similarity score among all its vectors compared to the page's embedding. This score represents how closely the page's content resembles the content typical of that class. If the highest similarity score across all classes is below a certain threshold (unknown_threshold), it classifies the page as \"UNKNOWN\". This means that the page does not closely resemble any of the known classes above the specified confidence level. The UNKNOWN class is assigned the highest similarity score found, even though it's below the threshold (more on this later).\n",
    "2. **Euclidean distance**, also known as L2 distance (`l2`), measures the straight line distance between two points in a multi-dimensional space. It is derived from the square root of the sum of the squared differences between corresponding elements of the two vectors. For each class, the Euclidean distance between the page's embedding and each of the class's sample vectors is calculated. Unlike cosine similarity, which measures the angle between vectors, Euclidean distance measures the magnitude of difference between vectors. A smaller Euclidean distance indicates a closer or more similar match between the vectors. Therefore, for each class, you would typically look for the minimum Euclidean distance between the page's embedding and the class's vectors to determine the closest match.\n",
    "\n",
    "----\n",
    "\n",
    "Following are the steps required\n",
    "\n",
    "1. You will define a set of classes you want to use. \n",
    "2. Collect a few sample documents for each class (minimum 1 sample max 10 samples each class)\n",
    "3. Create a CSV manifest file\n",
    "4. Run sampling using the manifest file, capture the `sample ID` (you will need this to run classification)\n",
    "5. Run classifications with new document using the `sample ID` from step 4.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a manifest file\n",
    "---\n",
    "\n",
    "Your manifest file should be of CSV format. For example\n",
    "\n",
    "manifest.csv\n",
    "\n",
    "```bash\n",
    "BANK_STATEMENT,s3://your-bucket/samples/bank_stmt_0.pdf,1\n",
    "BANK_STATEMENT,s3://your-bucket/samples/bank_stmt_1.pdf,1\n",
    "BANK_STATEMENT,s3://your-bucket/samples/bank_stmt_2.pdf,1\n",
    "INVOICE,s3://your-bucket/samples/invoice_0.pdf,1\n",
    "INVOICE,s3://your-bucket/samples/invoice_1.pdf,1\n",
    "RECEIPT,s3://your-bucket/samples/receipt_0.pdf,1\n",
    "RECEIPT,s3://your-bucket/samples/receipt_1.pdf,1\n",
    "RECEIPT,s3://your-bucket/samples/receipt_2.pdf,1\n",
    "```\n",
    "\n",
    "The CSV file should contain three fields-\n",
    "\n",
    "1. The class label should be the first field\n",
    "2. The sample document belonging to that class, either local path or `s3://` location of the sample document. S3 locations are recommended but local path's can be used during development.\n",
    "3. Third column is the page number of the document (in case the document is multi-page pdf)\n",
    "\n",
    "> If you would like to include all pages of a multi-page document in the sample dataset then you must make separate lines in the csv with the same document and the subsequent page number.\n",
    "\n",
    "You can store this manifest file into an S3 location (recommended) or use it locally. Once you have your manifest file ready we can create a classifier.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup a classifier\n",
    "---\n",
    "\n",
    "You initialize an instance of `DocClassification` and call the `run_sampling` function with the manifest file path to start the sampling process. This should be a fairly quick process and shouldn't take more than a few minutes. The function will return a `sample_id` which you can subsequently use to run document classification tasks. You will also need to provide an S3 bucket where the resulting vector samples will be stored for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sample_id': 'rb_classifier_1711608335'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "classifier = dc.run_sampling(manifest_path=\"s3://your-bucket/manifest.csv\")\n",
    "classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a classifier (sample) is a one-time process, i.e. once you setup your classifier you can use the `sample_id` to perform document classification tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a classifier\n",
    "---\n",
    "You use the `run_classify` function with path to the new document you would like to classify. The file can be a multi-page document and Rhubarb will classify each page individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'page': 1, 'classification': [{'class': 'BANK_STATEMENT', 'score': 0.92}]}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "results = dc.run_classify(sample_id=\"rb_classifier_1711608335\", \n",
    "                          file_path=\"./test_docs/bank_stmt.pdf\")\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `top_n` classes\n",
    "---\n",
    "By default Rhubarb will return the top document class for a given page, from the list of classes. You can change this behavior by passing a value of `top_n` to the function call. You can only pass up-to 3 top results to be returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'page': 1,\n",
       "  'classification': [{'class': 'BANK_STATEMENT', 'score': 0.92},\n",
       "   {'class': 'INVOICE', 'score': 0.76}]}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "results = dc.run_classify(sample_id=\"rb_classifier_1711608335\", \n",
    "                          file_path=\"./test_docs/bank_stmt.pdf\", \n",
    "                          top_n=2) # top 2 classes\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `unknown_threshold` \n",
    "---\n",
    "In some cases, your document processing pipeline may encounter documents that do not belong to any of the pre-configured classes. In such cases, it can be useful to mark these documents as `UNKNOWN` and isolate them for further analysis. This can be achieved using the `unknown_threshold` parameter. By default, the value is set to `0,8` i.e. any document below this threshold is automatically marked as `UNKNOWN`. However, you can override this value to further tune your classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'page': 1,\n",
       "  'classification': [{'class': 'BANK_STATEMENT', 'score': 0.93},\n",
       "   {'class': 'DISCHARGE_SUMMARY', 'score': 0.76}]},\n",
       " {'page': 2,\n",
       "  'classification': [{'class': 'RECEIPT', 'score': 1.0},\n",
       "   {'class': 'INVOICE', 'score': 0.78}]},\n",
       " {'page': 3,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.6150004682895025}]},\n",
       " {'page': 4,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.7916701829486292}]},\n",
       " {'page': 5,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.8255265125891919}]},\n",
       " {'page': 6,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.768307452125929}]}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "results = dc.run_classify(sample_id=\"rb_classifier_1711608335\", \n",
    "                          file_path=\"./test_docs/Sample1.pdf\", \n",
    "                          top_n=2,\n",
    "                          unknown_threshold=0.85)\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score for `UNKNOWN` classification is the highest similarity score (even if it's below the threshold). This gives a sense of how close the document was to being classified into one of the known classes before being deemed unknown. To view what the unknown pages is actually getting classified into, you can reduce the `unknown_threshold` and re-run the classification. Ideally, you would experiment and tune this score based on your use cases, the number of samples you used while creating the classifier, the number of classes, and the different type of documents you receive for your use-case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `similarity_metric`\n",
    "---\n",
    "By default Rhubarb uses _Cosine similarity_ to determine the class of a given page. If default Cosine similarity is used then scores can be between 0 and 1, and the higher the score the better. Higher number means the given document is most similar to the samples in the given class. \n",
    "\n",
    "However, you can also choose _Euclidian distance_ similarity metric (a.k.a L2 distance). In this case, scores will range beween 0 and 1, and the lower the number, the better, which means the straight line distance between the vector of the give document and the vector's of the class it is categorized in is the least. You can set the `similarity_metric` to `l2` to force using Euclidian distance for classification.\n",
    "\n",
    "Note, that in case of Euclidian distance, your `unknown_threshold` is the inverse of `cosine` similarity. Which means if `unknown_threshold` is set to `0.5` then any class with score more than `0.5` will be marked as `UNKNOWN` since it is the most dissimilar to all the known classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'page': 1, 'classification': [{'class': 'BANK_STATEMENT', 'score': 0.38}]},\n",
       " {'page': 2, 'classification': [{'class': 'RECEIPT', 'score': 0.0}]},\n",
       " {'page': 3,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.87749589194715}]},\n",
       " {'page': 4,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.6454917724396393}]},\n",
       " {'page': 5,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.5907173241851739}]},\n",
       " {'page': 6,\n",
       "  'classification': [{'class': 'UNKNOWN', 'score': 0.6807239544254996}]}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "results = dc.run_classify(sample_id=\"rb_classifier_1711608335\", \n",
    "                          file_path=\"./test_docs/Sample1.pdf\", \n",
    "                          similarity_metric=\"l2\",\n",
    "                          unknown_threshold=0.5) # threshold is low because smaller is better\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing a classifier\n",
    "---\n",
    "You can view the details of an existing classifier using the `view_sample` function. This will show you the classes you have configured for your classifiers and the samples you used while creating the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'class': 'BANK_STATEMENT', 'num_samples': 6},\n",
       " {'class': 'INVOICE', 'num_samples': 6},\n",
       " {'class': 'RECEIPT', 'num_samples': 6}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "dc.view_sample(sample_id=\"rb_classifier_1711608335\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Updating a classifier\n",
    "---\n",
    "You may have a need to subsequently update your classifier to add new classes, or samples. You can easily do that just like creating a new classifier with the help of a new manifest CSV file. But this time, you will need to supply an existing `sample_id` to the `run_sampling` function. This will process the manifest file but instead of creating a new classifier, will update the existing classifier (sample). For example with our new `manifest2.csv` we would like to update our existing sample.\n",
    "\n",
    "manifest2.csv\n",
    "\n",
    "```bash\n",
    "DISCHARGE_SUMMARY,s3://your-bucket/samples/discharge_summary_0.pdf,1\n",
    "DISCHARGE_SUMMARY,s3://your-bucket/samples/discharge_summary_1.pdf,1\n",
    "DISCHARGE_SUMMARY,s3://your-bucket/samples/discharge_summary_2.pdf,1\n",
    "DISCHARGE_SUMMARY,s3://your-bucket/samples/discharge_summary_3.pdf,1\n",
    "DISCHARGE_SUMMARY,s3://your-bucket/samples/discharge_summary_4.pdf,1\n",
    "```\n",
    "\n",
    "Note that it is recommended that you use samples that were not used before while creating the classifier (sample), since Rhubarb doesn't do any de-duplication on your behalf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sample_id': 'rb_classifier_1711608335'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "classifier = dc.run_sampling(manifest_path=\"s3://your-bucket/manifest2.csv\", \n",
    "                             update_sample_id=\"rb_classifier_1711608335\")\n",
    "classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view the updated classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'class': 'BANK_STATEMENT', 'num_samples': 6},\n",
       " {'class': 'INVOICE', 'num_samples': 6},\n",
       " {'class': 'RECEIPT', 'num_samples': 6},\n",
       " {'class': 'DISCHARGE_SUMMARY', 'num_samples': 5}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rhubarb import DocClassification\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "dc.view_sample(sample_id=\"rb_classifier_1711608335\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How is the classifier stored?\n",
    "---\n",
    "Your classifier will contain three distinct parts\n",
    "\n",
    "1. A unique identifier (`sample_id`)\n",
    "2. The class labels (i.e. `BANK_STATEMENT`, `RECEIPTS` and so on)\n",
    "3. The multi-modal vector embeddings for each of the samples for each class.\n",
    "\n",
    "All of this data is arranged and stored in a compressed Parquet file in the S3 bucket you provide (via the `bucket_name` parameter). Within the bucket, Rhubarb will store all your classifiers under the prefix `rb_classification`. However, you can override this prefix via Rhubarb's `GlobalConfig` and overriding `classification_prefix` with your desired prefix. The prefix must not contain any leading or trailing `/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rhubarb import DocClassification, GlobalConfig\n",
    "import boto3\n",
    "\n",
    "session = boto3.Session()\n",
    "\n",
    "GlobalConfig.update_config(classification_prefix=\"my_classifiers\")\n",
    "dc = DocClassification(bucket_name=\"your-classifier-bucket\", boto3_session=session)\n",
    "classifier = dc.run_sampling(manifest_path=\"s3://your-bucket/manifest.csv\")\n",
    "classifier"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}