clip

_layout/sidebar.html

@@ -158,6 +158,10 @@ <h1 style="font-size:1em; opacity: 0.95;"><a href="/">Deep Learning DIY</a></h1>
           href="/modules/18a-diffusion">
       <b>Module 18a</b> - <em>Denoising Diffusion Probabilistic Models</em>
       </a>
+      <a class="sidebar-nav-item {{ispage /modules/19-clip}}active{{end}}"
+          href="/modules/19-clip">
+      <b>Module 19</b> - <em>Zero-shot classification with CLIP</em>
+      </a>
 
       <!-- <div class="week">Unit 7</div>-->
 

modules/19-clip.md

@@ -0,0 +1,19 @@
+@def sequence = ["clip"]
+
+# Module 19 - Zero-shot classification with CLIP
+
+~~~<img src="../extras/clip/diagram.png"
+           style="width: 700px; height: auto; display: inline">
+~~~
+
+
+## Notebook
+
+- in [Zeroshot\_with\_CLIP.ipynb](https://github.com/dataflowr/notebooks/blob/master/Module19/Zeroshot_with_CLIP.ipynb) we build a zero-shop classifier using the pretrained CLIP network and improve its performance with descriptors generated with GPT.
+
+## References
+
+> [CLIP](https://github.com/openai/CLIP) Learning Transferable Visual Models From Natural Language Supervision (ICML 2021) Alec Radford et al.
+
+
+> [Visual Classification via Description from Large Language Models](https://github.com/sachit-menon/classify_by_description_release) (ICLR 2023) Menon, Sachit and Vondrick, Carl