CLIP_Dataset_Filtering.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CLIP-Dataset-Filtering.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dvschultz/ml-art-colabs/blob/master/CLIP_Dataset_Filtering.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C76CDbIJi2DY"
      },
      "source": [
        "# How to use CLIP Zero-Shot on your own classification dataset\n",
        "\n",
        "This notebook provides an example of how to benchmark CLIP's zero shot classification performance on your own classification dataset.\n",
        "\n",
        "[CLIP](https://openai.com/blog/clip/) is a new zero shot image classifier relased by OpenAI that has been trained on 400 million text/image pairs across the web. CLIP uses these learnings to make predictions based on a flexible span of possible classification categories.\n",
        "\n",
        "CLIP is zero shot, that means **no training is required**. \n",
        "\n",
        "This notebook is modified from a notebook by [Roboflow](https://colab.research.google.com/drive/1LXla2q9MCRRI_kTjpvag2Vz-7EGLnki5)\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "If you find this notebook useful, please consider signing up for my [Patreon](https://www.patreon.com/bustbright) or [YouTube channel](https://www.youtube.com/channel/UCaZuPdmZ380SFUMKHVsv_AA/join). You can also send me a one-time payment on [Venmo](https://venmo.com/Derrick-Schultz).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOF3Feb7jrnu"
      },
      "source": [
        "# Download and Install CLIP Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "idR_p-RZdNK7"
      },
      "source": [
        "!nvidia-smi -L"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FyUIWjzOi23X"
      },
      "source": [
        "#installing some dependencies, CLIP was release in PyTorch\n",
        "import subprocess\n",
        "\n",
        "CUDA_version = [s for s in subprocess.check_output([\"nvcc\", \"--version\"]).decode(\"UTF-8\").split(\", \") if s.startswith(\"release\")][0].split(\" \")[-1]\n",
        "print(\"CUDA version:\", CUDA_version)\n",
        "\n",
        "if CUDA_version == \"10.0\":\n",
        "    torch_version_suffix = \"+cu100\"\n",
        "elif CUDA_version == \"10.1\":\n",
        "    torch_version_suffix = \"+cu101\"\n",
        "elif CUDA_version == \"10.2\":\n",
        "    torch_version_suffix = \"\"\n",
        "else:\n",
        "    torch_version_suffix = \"+cu110\"\n",
        "\n",
        "!pip install torch==1.7.1{torch_version_suffix} torchvision==0.8.2{torch_version_suffix} -f https://download.pytorch.org/whl/torch_stable.html ftfy regex\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import os\n",
        "\n",
        "print(\"Torch version:\", torch.__version__)\n",
        "os.kill(os.getpid(), 9)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Q52wLJo_Mjk"
      },
      "source": [
        "## Clone the CLIP repo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cqN0UVpssA7J"
      },
      "source": [
        "#clone the CLIP repository\n",
        "!git clone https://github.com/openai/CLIP.git\n",
        "%cd CLIP"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sIiia2Q29wjm"
      },
      "source": [
        "import torch \n",
        "import clip\n",
        "\n",
        "print(clip.available_models())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwCkgL73rHE0"
      },
      "source": [
        "## Import Dataset\n",
        "\n",
        "Next we need to import a dataset. You can upload a zip directly to Colab (drag and drop it into the Files tab on your left), sync your Drive, or import using gdown."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_-9luoZ_mmG"
      },
      "source": [
        "#sync Drive account\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mgs2NwkM_vDl"
      },
      "source": [
        "!gdown --id 1BWGeWn0LMLa0ZgfoXTqO2hhGu6BPVh2O -O /content/mineral-samples.zip\n",
        "%cd /content/\n",
        "!unzip mineral-samples.zip\n",
        "%cd CLIP"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zrxAdgFACH_"
      },
      "source": [
        "## Create Tokens\n",
        "CLIP will use pieces of text, called tokens, to compare your image against. Below we will create a single token to test with.\n",
        "\n",
        "I rcommend experimenting with phrases you use. CLIP can respond to  particular sentence structures for better or worse responses."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gpXaqPH3oSyO"
      },
      "source": [
        "captions = ['A photo containing text','A photograph without text' ]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMZJy1SduxiE"
      },
      "source": [
        "# Single Image Scoring\n",
        "\n",
        "Let’s start by looking at a single image and a single caption. CLIP can take the image and provide a probablity for how likely the model thinks the caption and image match."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAi7cvucnFPr"
      },
      "source": [
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "captions = ['A photograph of a gemstone','A photograph of a gemstone held by a hand' ]\n",
        "img = '/content/content/mineral-samples/121477819_181900606849103_3709047619295376368_n.jpg'\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "\n",
        "#define our target classifications, you can should experiment with these strings of text as you see fit\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "with torch.no_grad():\n",
        "    image_features = model.encode_image(image)\n",
        "    text_features = model.encode_text(text)\n",
        "    \n",
        "    logits_per_image, logits_per_text = model(image, text)\n",
        "    probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "    print(probs)\n",
        "    pred = captions[argmax(list(probs)[0])]\n",
        "    display(Img(filename=img, width=400))\n",
        "    print(pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GhuvF1KoUXYs"
      },
      "source": [
        "!unzip /content/mineral-samples.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QSVwioNaMB1"
      },
      "source": [
        "## Sorting into two classes\n",
        "Below we’ll extend the above example to look at every image in a folder and sort the images into two folders. We’ll use the probability score and take the class that gets the higher probability. I recommend using tokens that express som binary operation.\n",
        "\n",
        "As this proceess runs the image, probability score, and predictions will be displayed. Pay close attention to false positives and consider editing your tokens if you see too many."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KHVwgwXq3Agi"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "imgs = glob.glob('/content/tests/A_photograph_of_a_gemstone/*.*')\n",
        "captions = ['A photograph of a gemstone','A photograph of a gemstone containing text' ]\n",
        "\n",
        "fpaths = []\n",
        "for f in captions:\n",
        "    fpath = '/content/test2/'+f.replace(' ','_')\n",
        "    fpaths.append(fpath)\n",
        "    if not os.path.exists(fpath):\n",
        "        os.makedirs(fpath)\n",
        "print(fpaths)\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "\n",
        "#define our target classifications, you can should experiment with these strings of text as you see fit\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "for img in imgs:\n",
        "    image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "    with torch.no_grad():\n",
        "        image_features = model.encode_image(image)\n",
        "        text_features = model.encode_text(text)\n",
        "        \n",
        "        logits_per_image, logits_per_text = model(image, text)\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])]\n",
        "        display(Img(filename=img, width=400))\n",
        "        print(probs)\n",
        "        print(pred)\n",
        "\n",
        "        img_name = img.split('/')[-1]\n",
        "        path = fpaths[argmax(list(probs)[0])] + '/' + img_name\n",
        "        !cp {img} {path}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "24A8w4mXBClw"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-mw6m2mZbWnJ"
      },
      "source": [
        "## Sorting into multiple classes (Max probabilty)\n",
        "\n",
        "So far we’ve only looked at two classes, but you can technically use any number of categories. This example will only sort by maximum probability, so each image will only end up in one class at a time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4yrqnKyUofrb"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "captions = ['A photograph of a gemstone on a black background','A photograph of a gemstone on a white background','A photograph of a gemstone on a gradient background' ]\n",
        "imgs = glob.glob('/content/minerals-min1024/*.jpg')\n",
        "\n",
        "fpaths = []\n",
        "for f in captions:\n",
        "    fpath = '/content/'+f.replace(' ','_')\n",
        "    fpaths.append(fpath)\n",
        "    if not os.path.exists(fpath):\n",
        "        os.makedirs(fpath)\n",
        "print(fpaths)\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "\n",
        "#define our target classifications, you can should experiment with these strings of text as you see fit\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "for img in imgs:\n",
        "    image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "    with torch.no_grad():\n",
        "        image_features = model.encode_image(image)\n",
        "        text_features = model.encode_text(text)\n",
        "        \n",
        "        logits_per_image, logits_per_text = model(image, text)\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])]\n",
        "        display(Img(filename=img, width=400))\n",
        "        print(probs)\n",
        "        print(pred)\n",
        "\n",
        "        #saves images\n",
        "        img_name = img.split('/')[-1]\n",
        "        path = fpaths[argmax(list(probs)[0])] + '/' + img_name\n",
        "        !cp {img} {path}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kcT-iBvjceE1"
      },
      "source": [
        "## Sorting into multiple classes (Greedy)\n",
        "\n",
        "You might want to sort an image into multiple folders. To do this you’ll ned to set a minimum probability score.\n",
        "\n",
        "This can be pretty tricky. Because all probability scores add up to one, you’ll need to find a good value that will define \"confidence, but not under- or over-confidence.\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3y456v8_DnpV"
      },
      "source": [
        "%cd CLIP/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ycQPl11Rco-g"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "import numpy as np\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "imgs = glob.glob('/content/minerals/*.*')\n",
        "# imgs = ['/content/26871485_568905930137798_4275492080328900608_n.jpg',\n",
        "#       '/content/27579032_703652923356546_8926669318020661248_n.jpg',\n",
        "#       '/content/28156709_2038566666361358_5503147325851172864_n.jpg',\n",
        "#       '/content/34982795_174824020040310_6797853509149523968_n.jpg',\n",
        "#       '/content/35459382_252143185338131_7462939386892517376_n.jpg',\n",
        "#       '/content/35518589_666909173653781_4904083633842683904_n.jpg']\n",
        "captions = ['A photo of a gemstone and no visible hands','A photo of gemstone and a visible hand' ]\n",
        "min_prob = .2\n",
        "\n",
        "fpaths = []\n",
        "for f in captions:\n",
        "    fpath = '/content/drive/MyDrive/CLIP-data/'+f.replace(' ','_')\n",
        "    fpaths.append(fpath)\n",
        "    if not os.path.exists(fpath):\n",
        "        os.makedirs(fpath)\n",
        "print(fpaths)\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "\n",
        "#define our target classifications, you can should experiment with these strings of text as you see fit\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "for img in imgs:\n",
        "    image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "    with torch.no_grad():\n",
        "        image_features = model.encode_image(image)\n",
        "        text_features = model.encode_text(text)\n",
        "        \n",
        "        logits_per_image, logits_per_text = model(image, text)\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])]\n",
        "        display(Img(filename=img, width=400))\n",
        "        print(probs)\n",
        "\n",
        "        #saves images\n",
        "        img_name = img.split('/')[-1]\n",
        "        for i in range(len(probs[0])):\n",
        "            if(probs[0][i] >= min_prob):\n",
        "                print(captions[i])\n",
        "                path = fpaths[i] + '/' + img_name\n",
        "                !cp {img} {path}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkkcya8crMUt"
      },
      "source": [
        "## Trying Something else"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xifeJRgmqVkv"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "imgs = glob.glob('/content/mineral-samples/*.*')\n",
        "# imgs = ['/content/75580670_179414003112257_70106094399349512_n.jpg', '/content/45345032_1060841144103369_8637295453703979280_n.jpg', '/content/121477819_181900606849103_3709047619295376368_n.jpg', '/content/121531725_973722799796652_2455931968095755240_n.jpg','/content/93375640_256338438825322_6148423300973295637_n.jpg','/content/64895378_2350623818592868_2370413862839102887_n.jpg']\n",
        "#define our target classifications, you should experiment with these strings of text as you see fit\n",
        "captions = ['An uncropped photo of a gemstone', 'An uncropped photo of a gemstone, contains a hand', 'A cropped photo of a gemstone', 'A photo of gemstone, contains text' ,'A photo of gemstone, contains a hand' ]\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "# model, transform = clip.load(\"RN101\", device=device)\n",
        "\n",
        "\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "# print(text)\n",
        "\n",
        "for img in imgs:\n",
        "    image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "    with torch.no_grad():\n",
        "        image_features = model.encode_image(image)\n",
        "        text_features = model.encode_text(text)\n",
        "        \n",
        "        logits_per_image, logits_per_text = model(image, text)\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])]\n",
        "        display(Img(filename=img, width=400))\n",
        "        print(probs)\n",
        "        print(pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBTvkuEW7Egw"
      },
      "source": [
        "import os\n",
        "import torch\n",
        "import clip\n",
        "from PIL import Image\n",
        "import glob\n",
        "from IPython.display import Image as Img, display\n",
        "\n",
        "def argmax(iterable):\n",
        "    return max(enumerate(iterable), key=lambda x: x[1])[0]\n",
        "\n",
        "imgs = glob.glob('/content/mineral-samples/*.*')\n",
        "# imgs = ['/content/75580670_179414003112257_70106094399349512_n.jpg', '/content/45345032_1060841144103369_8637295453703979280_n.jpg', '/content/121477819_181900606849103_3709047619295376368_n.jpg', '/content/121531725_973722799796652_2455931968095755240_n.jpg','/content/93375640_256338438825322_6148423300973295637_n.jpg','/content/64895378_2350623818592868_2370413862839102887_n.jpg']\n",
        "captions = ['An uncropped photo of a gemstone','A cropped photo of a gemstone','A photo containing text', 'A photo that does not contain text', 'A photo with hands', 'A photo without hands']\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, transform = clip.load(\"ViT-B/32\", device=device)\n",
        "# model, transform = clip.load(\"RN101\", device=device)\n",
        "\n",
        "#define our target classifications, you can should experiment with these strings of text as you see fit\n",
        "text = clip.tokenize(captions).to(device)\n",
        "\n",
        "for img in imgs:\n",
        "    image = transform(Image.open(img)).unsqueeze(0).to(device)\n",
        "    with torch.no_grad():\n",
        "        # t = text - t_not\n",
        "        image_features = model.encode_image(image)\n",
        "        text_features = model.encode_text(text)\n",
        "        \n",
        "        display(Img(filename=img, width=400))\n",
        "\n",
        "        logits_per_image, logits_per_text = model(image, text[:2])\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])]\n",
        "        \n",
        "        print(probs)\n",
        "        print(pred)\n",
        "\n",
        "        logits_per_image, logits_per_text = model(image, text[2:4])\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])+2]\n",
        "        \n",
        "        print(probs)\n",
        "        print(pred)\n",
        "\n",
        "        logits_per_image, logits_per_text = model(image, text[4:6])\n",
        "        probs = logits_per_image.softmax(dim=-1).cpu().numpy()\n",
        "        pred = captions[argmax(list(probs)[0])+4]\n",
        "        \n",
        "        print(probs)\n",
        "        print(pred)\n",
        "\n",
        "        # logits_per_image, logits_per_text = model(image, t_not)\n",
        "        # print(logits_per_image)\n"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}