Embedding/word2vec_skipgram_with_gluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word2Vec - skipgram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Word2Vec 구현을 위해 아래의 링크를 참고"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  - 논문"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-02T09:48:03.543659Z",
     "start_time": "2018-01-02T09:46:40.409Z"
    }
   },
   "source": [
    "- __[Distributed Representations of Words and Phrases and their Compositionality, 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)__\n",
    "\n",
    "- __[Efficient Estimation of Word Representations in Vector Space, 2013](https://arxiv.org/abs/1301.3781)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:13:08.407300Z",
     "start_time": "2018-01-04T07:35:17.446Z"
    }
   },
   "source": [
    "<img src=\"./images/skipgram.JPG\" width=\"900px\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:13:08.408303Z",
     "start_time": "2018-01-04T07:41:18.066Z"
    }
   },
   "source": [
    "### Word2Vec training\n",
    "Word2vec represents each word $w$ in a vocabulary $V$ of size $T$ as a low-dimensional dense vector $v_w$ in an embedding space $\\mathbb{R}^D$. It attempts to learn the continuous word vectors $v_w$, $\\forall w \\in V$ , from a training corpus such that the spatial distance between words then describes the similarity between words, e.g., the closer two words are in the embedding space, the more similar they are semantically and syntactically.  \n",
    "\n",
    "The skipgram architecture tries to predict the context given a word. The problem of predicting context words is framed as a set of independent binary classification tasks. Then the goal is to independently predict the presence (or absence) of context words. For the word at position $t$ we consider all context words as positive examples and sample negatives at random from the dictionary. For a chosen context position $c$, using the binary logistic loss, we obtain the following negative log-likelihood:\n",
    "\n",
    "$$ \\log (1 + e^{-s(w_t, w_c)}) +  \\sum_{n \\in \\mathcal{N}_{t,c}}^{}{\\log (1 + e^{s(w_t, n)})}$$\n",
    "\n",
    "where $w_t$ is a center word, $w_c$ is a context word, $\\mathcal{N}_{t,c}$ is a set of negative examples sampled from the vocabulary. By denoting the logistic loss function $l : x \\mapsto \\log(1 + e^{-x})$, we can re-write the objective as:\n",
    "\n",
    "$$ \\sum_{t=1}^{T}{ \\sum_{c \\in C_t}^{}{ \\big[ l(s(w_t, w_c))} + \\sum_{n \\in \\mathcal{N}_{t,c}}^{}{l(-s(w_t, n))}   \\big]} $$\n",
    "\n",
    "where $s(w_t, w_c) = u_{w_t}^T v_{w_c}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### - git source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-02T09:55:10.814690Z",
     "start_time": "2018-01-02T09:55:10.811682Z"
    }
   },
   "source": [
    "- __[Word2Vec with gluon](https://github.com/saurabh3949/Word2Vec-MXNet/blob/master/Word2vec%2Bwith%2BGluon.ipynb/)__\n",
    "- __[Word2Vec with mxnet](https://github.com/apache/incubator-mxnet/tree/master/example/nce-loss)__\n",
    "- __[Word2Vec with kears](http://adventuresinmachinelearning.com/word2vec-keras-tutorial/)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem with using a full softmax output layer is that it is very computationally expensive. \n",
    "There’s another solution called negative sampling.  It is described in the original Word2Vec paper by Mikolov et al.  It works by reinforcing the strength of weights which link a target word to its context words, but rather than reducing the value of all those weights which aren’t in the context, it simply samples a small number of them – these are called the “negative samples”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll update the weights for the correct label, but only a small number of incorrect labels. This is called “negative sampling”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-02T09:56:17.015684Z",
     "start_time": "2018-01-02T09:56:17.011673Z"
    }
   },
   "source": [
    "To train the embedding layer using negative samples in Keras, we can re-imagine the way we train our network.  Instead of constructing our network so that the output layer is a multi-class softmax layer, we can change it into a simple binary classifier.  For words that are in the context of the target word, we want our network to output a 1, and for our negative samples, we want our network to output a 0. Therefore, the output layer of our Word2Vec Keras network is simply a single node with a sigmoid activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-03T04:19:16.118524Z",
     "start_time": "2018-01-03T04:19:16.114514Z"
    }
   },
   "source": [
    "<img src=\"./images/Negative-sampling-architecture_01.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### nce_loss VS sampled_softmax_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample softmax is all about selecting a sample of the given number and try to get the softmax loss. Here the main objective is to make the result of the sampled softmax equal to our true softmax. So algorithm basically concentrate lot on selecting the those samples from the given distribution. On other hand NCE loss is more of selecting noise samples and try to mimic the true softmax. It will take only one true class and a K noise classes.\n",
    "###### https://stackoverflow.com/questions/42509878/what-is-the-difference-between-sampled-softmax-loss-and-nce-loss-in-tensorflow/43320139"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.026126Z",
     "start_time": "2018-01-04T08:20:21.546852Z"
    }
   },
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "from mxnet import nd\n",
    "from mxnet import gluon, autograd, nd\n",
    "from mxnet.gluon import Block, nn, utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.032142Z",
     "start_time": "2018-01-04T08:20:22.027129Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from datetime import datetime\n",
    "\n",
    "wrk_dir = '.'\n",
    "data_dir = '{}/data/'.format(wrk_dir)\n",
    "today = datetime.today()\n",
    "save_dir = '{}/save/{}'.format(wrk_dir, today.strftime('%Y-%m-%d'))\n",
    "conf_dir = '{}/conf'.format(wrk_dir)\n",
    "os.makedirs(save_dir, exist_ok = True)\n",
    "os.makedirs(conf_dir, exist_ok = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.049187Z",
     "start_time": "2018-01-04T08:20:22.033145Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Karabiner-Elements-App-Profiles',\n",
       " 'test-neg.txt',\n",
       " 'test-pos.txt',\n",
       " 'text8.txt',\n",
       " 'train-neg.txt',\n",
       " 'train-pos.txt',\n",
       " 'train-unsup.txt',\n",
       " 'word2vec-sentiments-master']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.056206Z",
     "start_time": "2018-01-04T08:20:22.050190Z"
    }
   },
   "outputs": [],
   "source": [
    "max_sentence_length = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.064227Z",
     "start_time": "2018-01-04T08:20:22.057210Z"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import logging\n",
    "from optparse import OptionParser\n",
    "\n",
    "import mxnet as mx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.075257Z",
     "start_time": "2018-01-04T08:20:22.065230Z"
    }
   },
   "outputs": [],
   "source": [
    "max_sentence_length = 10000\n",
    "WORD_DIM = 200\n",
    "NEGATIVE_SAMPLES = 5\n",
    "window_size = 5\n",
    "ctx = mx.gpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.083278Z",
     "start_time": "2018-01-04T08:20:22.076261Z"
    }
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 512"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:20:22.094307Z",
     "start_time": "2018-01-04T08:20:22.084281Z"
    }
   },
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:21:49.767385Z",
     "start_time": "2018-01-04T08:21:49.749336Z"
    }
   },
   "outputs": [],
   "source": [
    "def _load_data(name):\n",
    "    buf = open(name).read()\n",
    "    tks = buf.split(' ')\n",
    "    vocab = {}\n",
    "    freq = [0]\n",
    "    data = []\n",
    "    wid_to_word = [\"NA\"]\n",
    "    for tk in tks:\n",
    "        if len(tk) == 0:\n",
    "            continue\n",
    "        if tk not in vocab:\n",
    "            vocab[tk] = len(vocab) + 1\n",
    "            freq.append(0)\n",
    "            wid_to_word.append(tk)\n",
    "        wid = vocab[tk]\n",
    "        data.append(wid)\n",
    "        freq[wid] += 1\n",
    "    negative = []\n",
    "    for i, v in enumerate(freq):\n",
    "        if i == 0 or v < 5:\n",
    "            continue\n",
    "        v = int(math.pow(v * 1.0, 0.75))\n",
    "        negative += [i for _ in range(v)]\n",
    "    return data, negative, vocab, freq, wid_to_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:21:50.372994Z",
     "start_time": "2018-01-04T08:21:50.364973Z"
    }
   },
   "outputs": [],
   "source": [
    "class SimpleBatch(object):\n",
    "    def __init__(self, data_names, data, label_names, label):\n",
    "        self.data = data\n",
    "        self.label = label\n",
    "        self.data_names = data_names\n",
    "        self.label_names = label_names\n",
    "\n",
    "    @property\n",
    "    def provide_data(self):\n",
    "        return [(n, x.shape) for n, x in zip(self.data_names, self.data)]\n",
    "\n",
    "    @property\n",
    "    def provide_label(self):\n",
    "        return [(n, x.shape) for n, x in zip(self.label_names, self.label)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:33:37.516379Z",
     "start_time": "2018-01-04T08:33:37.454214Z"
    }
   },
   "outputs": [],
   "source": [
    "class DataIterWords_(mx.io.DataIter):\n",
    "    def __init__(self, name, batch_size, num_label):\n",
    "        super(DataIterWords_, self).__init__()\n",
    "        self.batch_size = batch_size\n",
    "        self.data, self.negative, self.vocab, self.freq, _ = _load_data(name)\n",
    "        self.vocab_size = 1 + len(self.vocab)\n",
    "        print(\"Vocabulary Size: {}\".format(self.vocab_size))\n",
    "        self.num_label = num_label\n",
    "        self.provide_data = [('data', (batch_size, num_label - 1))]\n",
    "        self.provide_label = [('label', (self.batch_size, num_label))]\n",
    "\n",
    "    def sample_ne(self):\n",
    "        return self.negative[random.randint(0, len(self.negative) - 1)]\n",
    "\n",
    "    def __iter__(self):\n",
    "        batch_data = []\n",
    "        batch_label = []\n",
    "        center_data = []\n",
    "        start = random.randint(0, self.num_label - 1)\n",
    "        for i in range(start, len(self.data) - self.num_label - start, self.num_label):\n",
    "            context = self.data[i: i + self.num_label // 2] \\\n",
    "                      + self.data[i + 1 + self.num_label // 2: i + self.num_label]\n",
    "            target_word = self.data[i + self.num_label // 2]\n",
    "            if self.freq[target_word] < 5:\n",
    "                continue\n",
    "            ## w_t \n",
    "            target = [target_word]\n",
    "            ## w_c + w_n\n",
    "            context = context + [self.sample_ne() for _ in range(self.num_label - 2)]\n",
    "            batch_data.append(context)\n",
    "            batch_label.append(target)\n",
    "            center_data.append([target_word])\n",
    "            if len(batch_data) == self.batch_size:\n",
    "                ## context\n",
    "                data_all = [mx.nd.array(batch_data)]\n",
    "                ## center\n",
    "                label_all = [mx.nd.array(batch_label)]\n",
    "                data_names = ['data']\n",
    "                label_names = ['label']\n",
    "                batch_data = []\n",
    "                batch_label = []\n",
    "                batch_label_weight = []\n",
    "                yield SimpleBatch(data_names, data_all, label_names, label_all)\n",
    "\n",
    "    def reset(self):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:29:39.420405Z",
     "start_time": "2018-01-04T08:29:32.881021Z"
    }
   },
   "outputs": [],
   "source": [
    "data, negative, vocab, freq, wid_to_word =  _load_data(\"{}/text8.txt\".format(data_dir))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:33:45.527677Z",
     "start_time": "2018-01-04T08:33:38.973253Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary Size: 253855\n"
     ]
    }
   ],
   "source": [
    "data_train = DataIterWords_(\"{}/text8.txt\".format(data_dir), BATCH_SIZE, NEGATIVE_SAMPLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:33:45.530685Z",
     "start_time": "2018-01-04T08:33:45.528680Z"
    }
   },
   "outputs": [],
   "source": [
    "VOCAB_SIZE = len(vocab) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:33:45.538706Z",
     "start_time": "2018-01-04T08:33:45.531688Z"
    }
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:32:29.087462Z",
     "start_time": "2018-01-04T08:32:03.029187Z"
    }
   },
   "outputs": [],
   "source": [
    "all_batches = []\n",
    "for idx, batch in enumerate(data_train):\n",
    "    all_batches.append(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:32:29.093478Z",
     "start_time": "2018-01-04T08:32:29.088465Z"
    }
   },
   "outputs": [],
   "source": [
    "def find_most_similar(word_to_index, index_to_word, all_vecs, word):\n",
    "    ans = []\n",
    "    if word not in word_to_index:\n",
    "        print(\"Sorry word not found. Please try another one.\")\n",
    "    else:  \n",
    "        i1 = word_to_index[word]\n",
    "        prod = all_vecs.dot(all_vecs[i1])\n",
    "        i2 = (-prod).argsort()[1:10]\n",
    "        for i in i2:\n",
    "            ans.extend([index_to_word[i]])\n",
    "        return ans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:32:29.110523Z",
     "start_time": "2018-01-04T08:32:29.094480Z"
    }
   },
   "outputs": [],
   "source": [
    "def callback_(model, word_nm_list):\n",
    "    from sklearn.preprocessing import normalize\n",
    "    keys = model.collect_params().keys()\n",
    "    all_vecs = model.collect_params()[list(keys)[0]].data().asnumpy()\n",
    "    all_vecs = normalize(all_vecs, copy=False)\n",
    "    \n",
    "    # Keep only the top 50K most frequent embeddings\n",
    "    top_50k = (-np.array(freq)).argsort()\n",
    "    word_to_index = {}\n",
    "    index_to_word = []\n",
    "    for newid, word_id in enumerate(top_50k):\n",
    "        index_to_word.append(wid_to_word[word_id])\n",
    "        word_to_index[wid_to_word[word_id]] = newid\n",
    "    \n",
    "    for word_nm in word_nm_list:\n",
    "        print (word_nm, ':', ', '.join(find_most_similar(word_to_index, index_to_word, all_vecs, word_nm)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:32:29.129574Z",
     "start_time": "2018-01-04T08:32:29.111526Z"
    }
   },
   "outputs": [],
   "source": [
    "class Model(gluon.HybridBlock):\n",
    "    def __init__(self, **kwargs):\n",
    "        super(Model, self).__init__(**kwargs)\n",
    "        with self.name_scope():\n",
    "            \n",
    "            # Embedding for input words with dimensions VOCAB_SIZE X WORD_DIM\n",
    "            self.center = nn.Embedding(input_dim=VOCAB_SIZE,\n",
    "                                       output_dim=WORD_DIM,\n",
    "                                       weight_initializer=mx.initializer.Uniform(1.0/WORD_DIM))\n",
    "            \n",
    "            # Embedding for output words with dimensions VOCAB_SIZE X WORD_DIM\n",
    "            self.target = nn.Embedding(input_dim=VOCAB_SIZE,\n",
    "                                       output_dim=WORD_DIM,\n",
    "                                       weight_initializer=mx.initializer.Zero())\n",
    "\n",
    "    def hybrid_forward(self, F, center, targets, labels):\n",
    "        \"\"\"\n",
    "        Returns the word2vec skipgram with negative sampling network.\n",
    "        :param F: F is a function space that depends on the type of other inputs. If their type is NDArray, then F will be mxnet.nd otherwise it will be mxnet.sym\n",
    "        :param center: A symbol/NDArray with dimensions (batch_size, 1). Contains the index of center word for each batch.\n",
    "        :param targets: A symbol/NDArray with dimensions (batch_size, negative_samples + 1). Contains the indices of 1 target word and `n` negative samples (n=5 in this example)\n",
    "        :param labels: A symbol/NDArray with dimensions (batch_size, negative_samples + 1). For 5 negative samples, the array for each batch is [1,0,0,0,0,0] i.e. label is 1 for target word and 0 for negative samples\n",
    "        :return: Return a HybridBlock object\n",
    "        \"\"\"\n",
    "        center_vector = self.center(center)\n",
    "        target_vectors = self.target(targets)\n",
    "        pred = F.broadcast_mul(center_vector, target_vectors)\n",
    "        pred = F.sum(data = pred, axis = 2)\n",
    "        sigmoid = F.sigmoid(pred)\n",
    "        loss = F.sum(labels * F.log(sigmoid) + (1 - labels) * F.log(1 - sigmoid), axis=1)\n",
    "        loss = loss * -1.0 / BATCH_SIZE\n",
    "        loss_layer = F.MakeLoss(loss)\n",
    "        return loss_layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:34:35.426331Z",
     "start_time": "2018-01-04T08:34:35.422321Z"
    }
   },
   "outputs": [],
   "source": [
    "labels = nd.zeros((BATCH_SIZE, NEGATIVE_SAMPLES+2), ctx=ctx)\n",
    "labels[:,0:4] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:34:35.898587Z",
     "start_time": "2018-01-04T08:34:35.895579Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(512, 7)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:34:37.188016Z",
     "start_time": "2018-01-04T08:34:37.183001Z"
    }
   },
   "outputs": [],
   "source": [
    "model = Model()\n",
    "model.collect_params().initialize(ctx=ctx)\n",
    "model.hybridize() # Convert to a symbolic network for efficiency.\n",
    "trainer = gluon.Trainer(model.collect_params(), 'SGD', {'learning_rate':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:34:37.568025Z",
     "start_time": "2018-01-04T08:34:37.565017Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import time\n",
    "import numpy as np\n",
    "logging.basicConfig(level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-01-04T08:34:37.951044Z",
     "start_time": "2018-01-04T08:34:37.949038Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model(\n",
      "  (center): Embedding(253855 -> 200, float32)\n",
      "  (target): Embedding(253855 -> 200, float32)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2018-01-04T08:34:38.390Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, batch 4999. Moving avg of loss: 4.2216905019\n",
      "january : contains, nearly, angeles, this, speakers, yeltsin, determined, values, feminism\n",
      "woman : prentice, removal, participate, monroe, iberian, relationships, atlanta, sphere, ultimately\n",
      "0  epochs took 68.0850031375885 seconds\n",
      "Epoch 1, batch 4999. Moving avg of loss: 4.11076397124\n",
      "january : contains, nearly, determined, ending, angeles, this, dna, future, speakers\n",
      "woman : concerned, interstellar, magnitude, acquire, integers, monroe, ess, kandahar, framed\n",
      "1  epochs took 134.55370903015137 seconds\n",
      "Epoch 2, batch 4999. Moving avg of loss: 4.03314016569\n",
      "january : bill, small, nearly, case, future, contains, this, peter, d\n",
      "woman : concerned, magnitude, interstellar, integers, sufficient, ess, airliner, recovery, kandahar\n",
      "2  epochs took 200.85797762870789 seconds\n",
      "Epoch 3, batch 4999. Moving avg of loss: 3.96486449085\n",
      "january : case, bill, small, determined, d, peter, contains, economy, senate\n",
      "woman : airliner, sufficient, concerned, chronicles, frees, prism, interstellar, interpreter, integers\n",
      "3  epochs took 267.5282189846039 seconds\n",
      "Epoch 4, batch 4999. Moving avg of loss: 3.91017865432\n",
      "january : case, determined, d, small, bill, contains, full, senate, economy\n",
      "woman : airliner, frees, kids, chronicles, prism, sufficient, decimated, endeavoured, combinations\n",
      "4  epochs took 333.70915961265564 seconds\n",
      "Epoch 5, batch 4999. Moving avg of loss: 3.86580251193\n",
      "january : determined, case, small, d, contains, von, full, economy, bill\n",
      "woman : kids, airliner, frees, decimated, prism, saving, chronicles, sufficient, endeavoured\n",
      "5  epochs took 399.72566318511963 seconds\n",
      "Epoch 6, batch 4999. Moving avg of loss: 3.82820930731\n",
      "january : determined, small, von, case, economy, d, various, contains, stage\n",
      "woman : kids, airliner, decimated, saving, frees, prism, endeavoured, cryptographers, chronicles\n",
      "6  epochs took 466.61849641799927 seconds\n",
      "Epoch 7, batch 4999. Moving avg of loss: 3.7950436758\n",
      "january : determined, small, von, economy, except, various, case, stage, too\n",
      "woman : kids, airliner, decimated, saving, frees, prism, cryptographers, endeavoured, sue\n",
      "7  epochs took 533.3148074150085 seconds\n",
      "Epoch 8, batch 4999. Moving avg of loss: 3.76498328068\n",
      "january : determined, small, economy, von, except, various, too, stage, case\n",
      "woman : kids, decimated, airliner, saving, integers, endeavoured, cryptographers, frees, prism\n",
      "8  epochs took 600.2646541595459 seconds\n",
      "Epoch 9, batch 4999. Moving avg of loss: 3.73724703281\n",
      "january : determined, economy, except, small, von, various, too, libya, rose\n",
      "woman : kids, decimated, airliner, saving, integers, adrianople, endeavoured, upland, conferencing\n",
      "9  epochs took 667.4993960857391 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "epochs = 10\n",
    "for e in range(epochs):\n",
    "    moving_loss = 0.\n",
    "    for i, batch in enumerate(all_batches):\n",
    "        #mx.nd.waitall()\n",
    "        center_words = batch.data[0].as_in_context(ctx)\n",
    "        target_words = batch.label[0].as_in_context(ctx)\n",
    "        \n",
    "        #if (i == 0):\n",
    "        #    print (center_words)\n",
    "        #    print (target_words)\n",
    "        #    sys.exit('')\n",
    "\n",
    "        with autograd.record():\n",
    "            loss = model(center_words, target_words, labels)\n",
    "        loss.backward()\n",
    "        trainer.step(1)\n",
    "        \n",
    "        #  Keep a moving average of the losses\n",
    "        if (i == 0) and (e == 0):\n",
    "            moving_loss = loss.asnumpy().sum()\n",
    "        else:\n",
    "            moving_loss = .99 * moving_loss + .01 * loss.asnumpy().sum()\n",
    "        if ((i + 1) % 5000 == 0):\n",
    "            print(\"Epoch %s, batch %s. Moving avg of loss: %s\" % (e, i, moving_loss))\n",
    "            \n",
    "        #if i > 15000:\n",
    "        #    break\n",
    "            \n",
    "    callback_(model, ['january','woman'])\n",
    "       \n",
    "    print(e, \" epochs took %s seconds\" % (time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}