siamese-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Siamese Networks\n",
    "\n",
    "<img src=\"static/siamese.png\" width=\"500\">\n",
    "\n",
    "## Contrastive Loss\n",
    "\n",
    "Consider a set $\\mathcal{X}$ composed of high dimensional vectors $\\vec{X_i}$. For each $\\vec{X_i}$, there exists is a set of vectors $\\mathcal{S}_{\\vec{X_i}}$ that are considered to be similar. This distance between vectors can be computed using some prior knowledge. The result of this is a set of parameters(W) that help map the vectors to a manifold.\n",
    "\n",
    "This mapping from high to low dimensional space maps similar vectors to nearby points and dissimilar points to distant points in the output manifold. We define the contrastive function whose minimization will result in the desired optimal mapping:\n",
    "\n",
    "Let $\\vec{X_1}$ and $\\vec{X_2} \\in \\mathcal{X}$ be a pair of vectors that need to be compared. We also have $Y$, a binary label associated with this pair. If the pair is similar, $Y = 0$, else $Y = 1$. Suppose we have learnt a mapping $G_W$ from the high dimensional space of vectors from $\\mathcal{X}$ to some lower dimensional space. This space is such that, the euclidean distance $D_W$ between points gives a measure of similarity of the higher dimensional vectors.\n",
    "\n",
    "Therefore, we learn the following distance function:\n",
    "$$\n",
    "D_W (\\vec{X_1}, \\vec{X_2}) = \\vert\\vert G_W(\\vec{X_1}) - G_W(\\vec{X_2}) \\vert\\vert_2\n",
    "$$\n",
    "\n",
    "\n",
    "To optimize the distance function, we minimize the following loss function:\n",
    "$$\n",
    "\\mathcal{L}(W) = \\sum_{i=1}^{P} L(W, (Y, \\vec{X_1}, \\vec{X_2})^i)\n",
    "\\\\\n",
    "L(W, (Y, \\vec{X_1}, \\vec{X_2})^i) = (1 - Y) L_S(D_W^i) + Y L_D(D_W^i)\n",
    "$$\n",
    "\n",
    "\n",
    "Here, $W$ by itself and as a subscript in $G_W$ and $D_W$ refers to learned parameters for the mapping of vectors in $\\mathcal{X}$. $P$ is the number of vector pairs available and $(Y, \\vec{X_1}, \\vec{X_2})^i$ is one of these pairs with the label. $L$ is broken into partial functions: $L_S$ for similar points and $L_D$ for dissimilar points.\n",
    "\n",
    "$L_S$ and $L_D$ are to be designed such that minimizing $L$ would result in lower values of $D_W$ for similar pairs and higher values of $D_W$ for dissimilar pairs. $L_S$ and $L_D$ are defined as follows:\n",
    "$$\n",
    "L_S = \\frac{1}{2}(D_W)^2\n",
    "\\\\\n",
    "L_D = \\frac{1}{2}(max\\{0, m - D_W\\})^2\n",
    "$$\n",
    "\n",
    "\n",
    "Therefore, L can be rewritten as:\n",
    "$$\n",
    "L(W, (Y, \\vec{X_1}, \\vec{X_2})) = (1 - Y)\\frac{1}{2}(D_W)^2 + Y\\frac{1}{2}max\\{0, m - D_W\\}^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "from tensorflow.examples.tutorials.mnist import input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Siamese:\n",
    "    def __init__(self, n_input, margin, learning_rate, model_dir, model_name):\n",
    "        self.X1 = tf.placeholder(tf.float32, [None, n_input], name=\"X1\")\n",
    "        self.X2 = tf.placeholder(tf.float32, [None, n_input], name=\"X2\")\n",
    "        self.y_ = tf.placeholder(tf.float32, [None,], name=\"y_\")\n",
    "        \n",
    "        self.margin = tf.constant(margin, name=\"margin\")\n",
    "        self.learning_rate = learning_rate\n",
    "        \n",
    "        self.out1, self.out2 = self.create_network()\n",
    "        self.loss = self.contrastive_loss()\n",
    "        self.optimizer = self.optimizer_init()\n",
    "        \n",
    "        self.saver = tf.train.Saver()\n",
    "        self.sess = tf.Session()\n",
    "        self.sess.run(tf.global_variables_initializer())\n",
    "        \n",
    "        \n",
    "        self.model_dir = model_dir\n",
    "        self.model_name = model_name\n",
    "    \n",
    "    def dense_layer(self, input, dim, name):\n",
    "        n_features = input.get_shape()[1]\n",
    "        \n",
    "        W = tf.get_variable(\n",
    "            name='{}_W'.format(name),\n",
    "            dtype=tf.float32,\n",
    "            shape=[n_features, dim])\n",
    "        \n",
    "        b = tf.get_variable(\n",
    "            name='{}_b'.format(name),\n",
    "            dtype=tf.float32,\n",
    "            shape=[dim],\n",
    "            initializer=tf.zeros_initializer())\n",
    "        \n",
    "        dense = tf.nn.bias_add(tf.matmul(input, W), b)\n",
    "        return dense\n",
    "    \n",
    "    def network(self, x):\n",
    "        dense1 = self.dense_layer(x, 1024, \"Dense_1\")\n",
    "        a1 = tf.nn.relu(dense1)\n",
    "        \n",
    "        dense2 = self.dense_layer(a1, 1024, \"Dense_2\")\n",
    "        a2 = tf.nn.relu(dense2)\n",
    "        \n",
    "        dense3 = self.dense_layer(a2, 2, \"Dense_3\")\n",
    "        return dense3\n",
    "    \n",
    "    def create_network(self):\n",
    "        with tf.variable_scope(\"siamese\") as scope:\n",
    "            out1 = self.network(self.X1)\n",
    "            scope.reuse_variables()\n",
    "            out2 = self.network(self.X2)\n",
    "        return out1, out2\n",
    "    \n",
    "    def euclidian_dist(self, p1, p2):\n",
    "        sum_squared_diff = tf.reduce_sum(tf.pow(tf.subtract(p1, p2), 2), axis=1, name=\"euclid_dist_sq\")\n",
    "        distance = tf.sqrt(sum_squared_diff + 1e-6, name=\"euclid_dist\")\n",
    "        \n",
    "        return distance, sum_squared_diff\n",
    "    \n",
    "    def contrastive_loss(self):\n",
    "        # yi*distance + (1-yi)*max(0, m-distance)\n",
    "        with tf.variable_scope(\"loss\") as scope:\n",
    "            dist, dist_sq = self.euclidian_dist(self.out1, self.out2)\n",
    "            \n",
    "            loss_similar = 0.5 * dist_sq\n",
    "            pos = tf.multiply(self.y_, loss_similar, name=\"loss_pos\")\n",
    "        \n",
    "            y_dissimilar = tf.subtract(1.0, self.y_)\n",
    "            loss_dissimilar = 0.5 * tf.pow(tf.maximum(tf.subtract(self.margin, dist), 0), 2)\n",
    "            neg = tf.multiply(y_dissimilar, loss_dissimilar, name=\"loss_neg\")\n",
    "        \n",
    "            loss = tf.reduce_mean(tf.add(pos, neg), name=\"loss\")\n",
    "        return loss\n",
    "    \n",
    "    def optimizer_init(self):\n",
    "        optimizer = tf.train.GradientDescentOptimizer(self.learning_rate).minimize(self.loss)\n",
    "        return optimizer\n",
    "    \n",
    "    def train(self, input1, input2, label):\n",
    "        _, loss = self.sess.run([self.optimizer, self.loss], feed_dict={self.X1: input1,\n",
    "                                                                        self.X2: input2,\n",
    "                                                                        self.y_: label})\n",
    "        return loss\n",
    "    \n",
    "    def test_model(self, input):\n",
    "        output = self.sess.run(self.out1, feed_dict = {self.X1: input})\n",
    "        return output\n",
    "\n",
    "    def load_model(self, model_name=None):\n",
    "        if model_name is None:\n",
    "            model_name = self.model_name\n",
    "        self.saver.restore(self.sess, os.path.join(self.model_dir, model_name))\n",
    "\n",
    "    def save_model(self, model_name=None):\n",
    "        if not os.path.exists(self.model_dir):\n",
    "            os.makedirs(self.model_dir)\n",
    "        if model_name is None:\n",
    "            model_name = self.model_name\n",
    "        self.saver.save(self.sess, os.path.join(self.model_dir, model_name))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_input = 28 * 28\n",
    "margin = 5.0\n",
    "learning_rate = 0.01\n",
    "batch_size = 128\n",
    "n_epochs = 10\n",
    "model_dir = \"tf_logs/siamese-0.1\"\n",
    "model_name = \"siamese-0.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "X_train = X_train.astype(np.float32).reshape((-1, n_input)) / 255.\n",
    "X_test = X_test.astype(np.float32).reshape((-1, n_input)) / 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "siamese = Siamese(n_input=n_input, margin=margin, learning_rate=learning_rate,\n",
    "                  model_dir=model_dir, model_name=model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_batch(X, y, batch_size):\n",
    "    rnd_idx = np.random.permutation(len(X))\n",
    "    n_batches = len(X) // batch_size\n",
    "\n",
    "    idxs = [batch_idx for batch_idx in np.array_split(rnd_idx, n_batches)]\n",
    "    len(idxs)\n",
    "    for i,j in zip(idxs[0::2], idxs[1::2]):\n",
    "        X1_batch = X[i]\n",
    "        X2_batch = X[j]\n",
    "        y_batch = (y[i] == y[j]).astype(np.float32)\n",
    "\n",
    "        yield X1_batch, X2_batch, y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 10, loss: 6.698212623596191\n",
      "step 20, loss: 3.2929441928863525\n",
      "step 30, loss: 3.1118156909942627\n",
      "step 40, loss: 2.5552587509155273\n",
      "step 50, loss: 2.4288651943206787\n",
      "step 60, loss: 1.8632421493530273\n",
      "step 70, loss: 1.7675292491912842\n",
      "step 80, loss: 2.0606393814086914\n",
      "step 90, loss: 2.7103660106658936\n",
      "step 100, loss: 1.5873215198516846\n",
      "step 110, loss: 1.111082673072815\n",
      "step 120, loss: 1.3335705995559692\n",
      "step 130, loss: 2.2416059970855713\n",
      "step 140, loss: 1.32935631275177\n",
      "step 150, loss: 1.529503583908081\n",
      "step 160, loss: 2.1370935440063477\n",
      "step 170, loss: 1.1548789739608765\n",
      "step 180, loss: 1.7969799041748047\n",
      "step 190, loss: 1.6378898620605469\n",
      "step 200, loss: 1.8471750020980835\n",
      "step 210, loss: 1.428123950958252\n",
      "step 220, loss: 1.8339557647705078\n",
      "step 230, loss: 1.6519286632537842\n",
      "end of epoch 0, loss: 1.7872884273529053\n",
      "step 240, loss: 1.497567057609558\n",
      "step 250, loss: 1.4996349811553955\n",
      "step 260, loss: 1.3743512630462646\n",
      "step 270, loss: 0.7755866050720215\n",
      "step 280, loss: 1.741788387298584\n",
      "step 290, loss: 1.5345065593719482\n",
      "step 300, loss: 0.9428651928901672\n",
      "step 310, loss: 1.3866527080535889\n",
      "step 320, loss: 2.3728528022766113\n",
      "step 330, loss: 1.5040614604949951\n",
      "step 340, loss: 1.4979608058929443\n",
      "step 350, loss: 2.1577298641204834\n",
      "step 360, loss: 1.0948479175567627\n",
      "step 370, loss: 1.5550541877746582\n",
      "step 380, loss: 1.7761040925979614\n",
      "step 390, loss: 1.1946444511413574\n",
      "step 400, loss: 1.3588244915008545\n",
      "step 410, loss: 1.7009998559951782\n",
      "step 420, loss: 0.9497608542442322\n",
      "step 430, loss: 1.3771017789840698\n",
      "step 440, loss: 1.5726964473724365\n",
      "step 450, loss: 1.8548997640609741\n",
      "step 460, loss: 0.8524545431137085\n",
      "end of epoch 1, loss: 1.3618378639221191\n",
      "step 470, loss: 0.9795188903808594\n",
      "step 480, loss: 1.220240592956543\n",
      "step 490, loss: 0.9127670526504517\n",
      "step 500, loss: 1.2281829118728638\n",
      "step 510, loss: 1.4411673545837402\n",
      "step 520, loss: 2.2194223403930664\n",
      "step 530, loss: 1.5047252178192139\n",
      "step 540, loss: 1.5077366828918457\n",
      "step 550, loss: 1.458924651145935\n",
      "step 560, loss: 1.757338047027588\n",
      "step 570, loss: 0.8440114259719849\n",
      "step 580, loss: 1.264305830001831\n",
      "step 590, loss: 1.380305290222168\n",
      "step 600, loss: 1.0943362712860107\n",
      "step 610, loss: 1.1426680088043213\n",
      "step 620, loss: 0.8767198324203491\n",
      "step 630, loss: 1.2379279136657715\n",
      "step 640, loss: 1.890114426612854\n",
      "step 650, loss: 1.1664992570877075\n",
      "step 660, loss: 1.046138048171997\n",
      "step 670, loss: 1.7624447345733643\n",
      "step 680, loss: 1.4792544841766357\n",
      "step 690, loss: 1.616786003112793\n",
      "step 700, loss: 1.3772844076156616\n",
      "end of epoch 2, loss: 1.3817541599273682\n",
      "step 710, loss: 1.2515052556991577\n",
      "step 720, loss: 0.91205233335495\n",
      "step 730, loss: 1.189607858657837\n",
      "step 740, loss: 1.7032915353775024\n",
      "step 750, loss: 0.9474319219589233\n",
      "step 760, loss: 1.202366828918457\n",
      "step 770, loss: 1.484161138534546\n",
      "step 780, loss: 1.1751219034194946\n",
      "step 790, loss: 1.086846947669983\n",
      "step 800, loss: 1.1372277736663818\n",
      "step 810, loss: 1.4941484928131104\n",
      "step 820, loss: 1.255550503730774\n",
      "step 830, loss: 1.1150802373886108\n",
      "step 840, loss: 1.0986747741699219\n",
      "step 850, loss: 1.0389482975006104\n",
      "step 860, loss: 0.8075058460235596\n",
      "step 870, loss: 1.337471604347229\n",
      "step 880, loss: 1.7463476657867432\n",
      "step 890, loss: 1.2130275964736938\n",
      "step 900, loss: 1.0945639610290527\n",
      "step 910, loss: 1.249179482460022\n",
      "step 920, loss: 1.0735129117965698\n",
      "step 930, loss: 1.5494625568389893\n",
      "end of epoch 3, loss: 0.7970250844955444\n",
      "step 940, loss: 1.4359495639801025\n",
      "step 950, loss: 1.1879403591156006\n",
      "step 960, loss: 2.300391674041748\n",
      "step 970, loss: 0.8446869254112244\n",
      "step 980, loss: 0.8319035172462463\n",
      "step 990, loss: 1.1933331489562988\n",
      "step 1000, loss: 1.8231921195983887\n",
      "step 1010, loss: 1.3573522567749023\n",
      "step 1020, loss: 0.9043941497802734\n",
      "step 1030, loss: 1.195645809173584\n",
      "step 1040, loss: 1.4362071752548218\n",
      "step 1050, loss: 1.3853850364685059\n",
      "step 1060, loss: 1.0386937856674194\n",
      "step 1070, loss: 1.0028921365737915\n",
      "step 1080, loss: 0.847939133644104\n",
      "step 1090, loss: 1.1051535606384277\n",
      "step 1100, loss: 1.3518139123916626\n",
      "step 1110, loss: 1.4398505687713623\n",
      "step 1120, loss: 1.0783634185791016\n",
      "step 1130, loss: 0.877159595489502\n",
      "step 1140, loss: 1.3249082565307617\n",
      "step 1150, loss: 1.5888826847076416\n",
      "step 1160, loss: 0.7994749546051025\n",
      "step 1170, loss: 1.0138084888458252\n",
      "end of epoch 4, loss: 1.0138084888458252\n",
      "step 1180, loss: 0.9822435975074768\n",
      "step 1190, loss: 1.191475510597229\n",
      "step 1200, loss: 1.0214300155639648\n",
      "step 1210, loss: 0.5944702625274658\n",
      "step 1220, loss: 0.9473970532417297\n",
      "step 1230, loss: 0.9789690375328064\n",
      "step 1240, loss: 0.7481083869934082\n",
      "step 1250, loss: 0.950119137763977\n",
      "step 1260, loss: 1.3061673641204834\n",
      "step 1270, loss: 1.0805706977844238\n",
      "step 1280, loss: 1.453804612159729\n",
      "step 1290, loss: 1.3145253658294678\n",
      "step 1300, loss: 1.2188024520874023\n",
      "step 1310, loss: 0.947168231010437\n",
      "step 1320, loss: 0.8734796643257141\n",
      "step 1330, loss: 1.176444411277771\n",
      "step 1340, loss: 1.0297681093215942\n",
      "step 1350, loss: 0.8874635696411133\n",
      "step 1360, loss: 0.7907434105873108\n",
      "step 1370, loss: 0.8392777442932129\n",
      "step 1380, loss: 0.6169984340667725\n",
      "step 1390, loss: 1.0291929244995117\n",
      "step 1400, loss: 1.476697564125061\n",
      "end of epoch 5, loss: 1.3264521360397339\n",
      "step 1410, loss: 1.0439099073410034\n",
      "step 1420, loss: 1.1209373474121094\n",
      "step 1430, loss: 0.9737469553947449\n",
      "step 1440, loss: 0.9871097803115845\n",
      "step 1450, loss: 0.6523359417915344\n",
      "step 1460, loss: 1.6851727962493896\n",
      "step 1470, loss: 1.1913409233093262\n",
      "step 1480, loss: 0.8911668658256531\n",
      "step 1490, loss: 1.4050625562667847\n",
      "step 1500, loss: 1.0299358367919922\n",
      "step 1510, loss: 1.0734795331954956\n",
      "step 1520, loss: 0.8770830035209656\n",
      "step 1530, loss: 0.7755833864212036\n",
      "step 1540, loss: 0.989447832107544\n",
      "step 1550, loss: 0.9228471517562866\n",
      "step 1560, loss: 0.7085955739021301\n",
      "step 1570, loss: 0.7856762409210205\n",
      "step 1580, loss: 0.7537503838539124\n",
      "step 1590, loss: 1.0897778272628784\n",
      "step 1600, loss: 0.9470939636230469\n",
      "step 1610, loss: 1.0119942426681519\n",
      "step 1620, loss: 1.6619682312011719\n",
      "step 1630, loss: 0.8096254467964172\n",
      "end of epoch 6, loss: 1.2228782176971436\n",
      "step 1640, loss: 1.068054437637329\n",
      "step 1650, loss: 1.055161476135254\n",
      "step 1660, loss: 0.8785138726234436\n",
      "step 1670, loss: 0.8868212103843689\n",
      "step 1680, loss: 0.6919401288032532\n",
      "step 1690, loss: 1.210607886314392\n",
      "step 1700, loss: 0.8461562991142273\n",
      "step 1710, loss: 1.15382719039917\n",
      "step 1720, loss: 1.3436274528503418\n",
      "step 1730, loss: 0.879304051399231\n",
      "step 1740, loss: 0.9757304787635803\n",
      "step 1750, loss: 0.6266556978225708\n",
      "step 1760, loss: 1.1090333461761475\n",
      "step 1770, loss: 0.8956019878387451\n",
      "step 1780, loss: 1.1288827657699585\n",
      "step 1790, loss: 0.6583304405212402\n",
      "step 1800, loss: 1.0266470909118652\n",
      "step 1810, loss: 0.9376806616783142\n",
      "step 1820, loss: 1.1115477085113525\n",
      "step 1830, loss: 1.207951545715332\n",
      "step 1840, loss: 0.6316691040992737\n",
      "step 1850, loss: 0.6711955070495605\n",
      "step 1860, loss: 0.7742606401443481\n",
      "step 1870, loss: 0.7590256929397583\n",
      "end of epoch 7, loss: 1.157907247543335\n",
      "step 1880, loss: 0.6436214447021484\n",
      "step 1890, loss: 0.866710901260376\n",
      "step 1900, loss: 1.24284827709198\n",
      "step 1910, loss: 1.0974130630493164\n",
      "step 1920, loss: 0.9081850647926331\n",
      "step 1930, loss: 1.5087863206863403\n",
      "step 1940, loss: 1.1373345851898193\n",
      "step 1950, loss: 0.79755038022995\n",
      "step 1960, loss: 1.1013872623443604\n",
      "step 1970, loss: 1.3650705814361572\n",
      "step 1980, loss: 1.0294650793075562\n",
      "step 1990, loss: 1.1379272937774658\n",
      "step 2000, loss: 0.9626792073249817\n",
      "step 2010, loss: 1.248677134513855\n",
      "step 2020, loss: 0.9983401298522949\n",
      "step 2030, loss: 0.9381897449493408\n",
      "step 2040, loss: 0.8168816566467285\n",
      "step 2050, loss: 1.162718415260315\n",
      "step 2060, loss: 0.9049803018569946\n",
      "step 2070, loss: 0.7478306889533997\n",
      "step 2080, loss: 0.6757731437683105\n",
      "step 2090, loss: 1.2125130891799927\n",
      "step 2100, loss: 0.6691006422042847\n",
      "end of epoch 8, loss: 0.6709694862365723\n",
      "step 2110, loss: 1.1290453672409058\n",
      "step 2120, loss: 0.9424196481704712\n",
      "step 2130, loss: 0.6640536785125732\n",
      "step 2140, loss: 0.79476398229599\n",
      "step 2150, loss: 0.9158809185028076\n",
      "step 2160, loss: 0.718204915523529\n",
      "step 2170, loss: 0.898329496383667\n",
      "step 2180, loss: 0.5705631971359253\n",
      "step 2190, loss: 0.8091126680374146\n",
      "step 2200, loss: 0.9701023697853088\n",
      "step 2210, loss: 1.1251661777496338\n",
      "step 2220, loss: 0.8708628416061401\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 2230, loss: 1.1453388929367065\n",
      "step 2240, loss: 0.9828543066978455\n",
      "step 2250, loss: 0.6444646716117859\n",
      "step 2260, loss: 0.9774099588394165\n",
      "step 2270, loss: 1.1849122047424316\n",
      "step 2280, loss: 0.7715990543365479\n",
      "step 2290, loss: 0.9089393615722656\n",
      "step 2300, loss: 0.6883367896080017\n",
      "step 2310, loss: 0.9107192158699036\n",
      "step 2320, loss: 0.986473798751831\n",
      "step 2330, loss: 1.1138514280319214\n",
      "step 2340, loss: 0.7945080995559692\n",
      "end of epoch 9, loss: 0.7945080995559692\n"
     ]
    }
   ],
   "source": [
    "step = 0\n",
    "for epoch in range(n_epochs):\n",
    "    for batch_X1, batch_X2, batch_y in get_batch(X_train, y_train, batch_size):\n",
    "        step += 1\n",
    "        \n",
    "        loss = siamese.train(input1=batch_X1, input2=batch_X2, label=batch_y)\n",
    "        if step % 10 == 0:\n",
    "            print('step {}, loss: {}'.format(step, loss))\n",
    "    \n",
    "    print('end of epoch {}, loss: {}'.format(epoch, loss))\n",
    "    if epoch % 5 == 0:\n",
    "        siamese.save_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "siamese.test_model(input=X_test).tofile('static/embed.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "matplotlib.use('Agg')\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize(embed, labels):\n",
    "\n",
    "    labelset = set(labels.tolist())\n",
    "\n",
    "    fig = plt.figure(figsize=(8,8))\n",
    "    ax = fig.add_subplot(111)\n",
    "\n",
    "    for label in labelset:\n",
    "        indices = np.where(labels == label)\n",
    "        ax.scatter(embed[indices,0], embed[indices,1], label = label, s = 20)\n",
    "    ax.legend()\n",
    "    fig.savefig('embed.jpeg', format='jpeg', dpi=600, bbox_inches='tight')\n",
    "    plt.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed = np.fromfile('static/embed.txt', dtype = np.float32)\n",
    "embed = embed.reshape([-1, 2])\n",
    "\n",
    "visualize(embed, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"static/embed.jpeg\" width=\"500\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}