Step_2_reduce_dims_TSNE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About\n",
    "In this notebook, we read in some tf-idf data which is essentially a giant matrix and then we reduce its dimensionality so that we can visualise its features.  Note that there's actually a lot more that can be done with the tf-idf data - this is all just for the purposes of visualisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------\n",
      "Step 2:  t-SNE dimensional reduction for visualisation\n",
      "2018-02-16 12:47:03.997663\n",
      "------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print('------------------------------------------------------')\n",
    "print('Step 2:  t-SNE dimensional reduction for visualisation')\n",
    "from datetime import datetime as dt\n",
    "print(dt.now())\n",
    "print('------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# import dependencies\n",
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.decomposition import TruncatedSVD, PCA\n",
    "from datetime import datetime as dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# configure\n",
    "from config import Config as c\n",
    "working_data = c.working_data\n",
    "doi_datapath = c.dois_pkl\n",
    "# word_datapath = c.word_datapath\n",
    "tfidf_datapath = c.tfidf_datapath\n",
    "# cosine_sims_datapath = c.cosine_sims_datapath\n",
    "vectorizer_datapath = c.vectorizer_datapath\n",
    "tsne_data = c.tsne_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# load data\n",
    "t_start=dt.now()\n",
    "tfidf = pickle.load(open(tfidf_datapath,'rb')).todense() # should this be todense?\n",
    "# cosine_distances = pickle.load(open(cosine_sims_datapath,'rb'))\n",
    "dois = pickle.load(open(doi_datapath,'rb'))\n",
    "data = pd.read_csv(working_data,index_col = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heuristics\n",
    "This section is far from ideal.  We are using t-SNE to reduce a high dimensional dataset down to 2 dimensions.  The correct settings are dependent on the size of our dataset, its structure, AND on each other - so changing one setting requires changing others as well to get the desired effect.  Furthermore, the process is not completely deterministic and won't always give the same result with the same inputs.  Tuning t-SNE is not something that can be done programmatically as far as I am aware.  \n",
    "\n",
    "Below, I have simply guessed at some approximate starting figures which I think will give OK results.  It may be necessary to modify them in config.py for each specific visualisation in order to get better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Perplexity set to: 80\n",
      "Number of iterations set to:  1000\n",
      "Number of iterations without progress set to:  50\n",
      "Learning rate set to:  100\n"
     ]
    }
   ],
   "source": [
    "# configure t-sne parameters\n",
    "n_rows = data.shape[0]\n",
    "\n",
    "if c.perplexity == None:\n",
    "    perplexity = int(0.07*n_rows) # choosing 7 percent of dataset size for perplexity\n",
    "    print('Warning: perplexity not set in config.py choosing default of: ',perplexity)\n",
    "else:\n",
    "    perplexity = c.perplexity\n",
    "    print('Perplexity set to:',perplexity)\n",
    "\n",
    "if c.n_iter == None:\n",
    "    n_iter = int(2*n_rows)\n",
    "    print('Warning: n_iter not set in config.py choosing default of: ',n_iter)\n",
    "else:\n",
    "    n_iter = c.n_iter\n",
    "    print('Number of iterations set to: ',n_iter)\n",
    "    \n",
    "if c.n_iter_without_progress == None:\n",
    "    n_iter_without_progress = max(n_iter//100 , 25)\n",
    "    print('Warning: n_iter_without_progress not set in config.py choosing default of: ',n_iter_without_progress)\n",
    "else:\n",
    "    n_iter_without_progress = c.n_iter_without_progress\n",
    "    print('Number of iterations without progress set to: ',n_iter_without_progress)\n",
    "\n",
    "if c.learning_rate == None:\n",
    "    learning_rate = int(7*n_rows)\n",
    "    print('Warning: learning_rate not set in config.py choosing default of: ',learning_rate)\n",
    "else:\n",
    "    learning_rate = c.learning_rate\n",
    "    print('Learning rate set to: ',learning_rate)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7713, 23)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DI</th>\n",
       "      <th>PY</th>\n",
       "      <th>WD</th>\n",
       "      <th>AU</th>\n",
       "      <th>AF</th>\n",
       "      <th>SO</th>\n",
       "      <th>SC</th>\n",
       "      <th>SN</th>\n",
       "      <th>EI</th>\n",
       "      <th>TC</th>\n",
       "      <th>...</th>\n",
       "      <th>DOI</th>\n",
       "      <th>Link</th>\n",
       "      <th>field_citation_ratio</th>\n",
       "      <th>highly_cited_1</th>\n",
       "      <th>highly_cited_10</th>\n",
       "      <th>highly_cited_5</th>\n",
       "      <th>recent_citations</th>\n",
       "      <th>relative_citation_ratio</th>\n",
       "      <th>times_cited</th>\n",
       "      <th>Citations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2237</th>\n",
       "      <td>10.1523/JNEUROSCI.4627-13.2014</td>\n",
       "      <td>2014</td>\n",
       "      <td>Cortical and Thalamic Excitation Mediate the M...</td>\n",
       "      <td>Doig, NM; Magill, PJ; Apicella, P; Bolam, JP; ...</td>\n",
       "      <td>Doig, Natalie M.; Magill, Peter J.; Apicella, ...</td>\n",
       "      <td>JOURNAL OF NEUROSCIENCE</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0270-6474</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31</td>\n",
       "      <td>...</td>\n",
       "      <td>10.1523/JNEUROSCI.4627-13.2014</td>\n",
       "      <td>http://dx.doi.org10.1523/JNEUROSCI.4627-13.2014</td>\n",
       "      <td>8.32</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>27.0</td>\n",
       "      <td>3.64</td>\n",
       "      <td>41.0</td>\n",
       "      <td>41.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4328</th>\n",
       "      <td>10.1016/S1474-4422(12)70311-3</td>\n",
       "      <td>2013</td>\n",
       "      <td>A multilevel intervention to increase communit...</td>\n",
       "      <td>Scott, PA; Meurer, WJ; Frederiksen, SM; Kalbfl...</td>\n",
       "      <td>Scott, Phillip A.; Meurer, William J.; Frederi...</td>\n",
       "      <td>LANCET NEUROLOGY</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>1474-4422</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26</td>\n",
       "      <td>...</td>\n",
       "      <td>10.1016/S1474-4422(12)70311-3</td>\n",
       "      <td>http://dx.doi.org10.1016/S1474-4422(12)70311-3</td>\n",
       "      <td>6.86</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>32.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1401</th>\n",
       "      <td>10.1523/JNEUROSCI.1034-14.2014</td>\n",
       "      <td>2014</td>\n",
       "      <td>Motion Direction Biases and Decoding in Human ...</td>\n",
       "      <td>Wang, HX; Merriam, EP; Freeman, J; Heeger, DJ</td>\n",
       "      <td>Wang, Helena X.; Merriam, Elisha P.; Freeman, ...</td>\n",
       "      <td>JOURNAL OF NEUROSCIENCE</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0270-6474</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>10.1523/JNEUROSCI.1034-14.2014</td>\n",
       "      <td>http://dx.doi.org10.1523/JNEUROSCI.1034-14.2014</td>\n",
       "      <td>3.00</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.02</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4670</th>\n",
       "      <td>10.1177/1352458512462920</td>\n",
       "      <td>2013</td>\n",
       "      <td>Characterization of anti-natalizumab antibodie...</td>\n",
       "      <td>Lundkvist, M; Engdahl, E; Holmen, C; Moverare,...</td>\n",
       "      <td>Lundkvist, M.; Engdahl, E.; Holmen, C.; Movera...</td>\n",
       "      <td>MULTIPLE SCLEROSIS JOURNAL</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>1352-4585</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>10.1177/1352458512462920</td>\n",
       "      <td>http://dx.doi.org10.1177/1352458512462920</td>\n",
       "      <td>2.32</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.33</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  DI    PY  \\\n",
       "2237  10.1523/JNEUROSCI.4627-13.2014  2014   \n",
       "4328   10.1016/S1474-4422(12)70311-3  2013   \n",
       "1401  10.1523/JNEUROSCI.1034-14.2014  2014   \n",
       "4670        10.1177/1352458512462920  2013   \n",
       "\n",
       "                                                     WD  \\\n",
       "2237  Cortical and Thalamic Excitation Mediate the M...   \n",
       "4328  A multilevel intervention to increase communit...   \n",
       "1401  Motion Direction Biases and Decoding in Human ...   \n",
       "4670  Characterization of anti-natalizumab antibodie...   \n",
       "\n",
       "                                                     AU  \\\n",
       "2237  Doig, NM; Magill, PJ; Apicella, P; Bolam, JP; ...   \n",
       "4328  Scott, PA; Meurer, WJ; Frederiksen, SM; Kalbfl...   \n",
       "1401      Wang, HX; Merriam, EP; Freeman, J; Heeger, DJ   \n",
       "4670  Lundkvist, M; Engdahl, E; Holmen, C; Moverare,...   \n",
       "\n",
       "                                                     AF  \\\n",
       "2237  Doig, Natalie M.; Magill, Peter J.; Apicella, ...   \n",
       "4328  Scott, Phillip A.; Meurer, William J.; Frederi...   \n",
       "1401  Wang, Helena X.; Merriam, Elisha P.; Freeman, ...   \n",
       "4670  Lundkvist, M.; Engdahl, E.; Holmen, C.; Movera...   \n",
       "\n",
       "                              SO                         SC         SN   EI  \\\n",
       "2237     JOURNAL OF NEUROSCIENCE  Neurosciences & Neurology  0270-6474  NaN   \n",
       "4328            LANCET NEUROLOGY  Neurosciences & Neurology  1474-4422  NaN   \n",
       "1401     JOURNAL OF NEUROSCIENCE  Neurosciences & Neurology  0270-6474  NaN   \n",
       "4670  MULTIPLE SCLEROSIS JOURNAL  Neurosciences & Neurology  1352-4585  NaN   \n",
       "\n",
       "      TC    ...                                 DOI  \\\n",
       "2237  31    ...      10.1523/JNEUROSCI.4627-13.2014   \n",
       "4328  26    ...       10.1016/S1474-4422(12)70311-3   \n",
       "1401   8    ...      10.1523/JNEUROSCI.1034-14.2014   \n",
       "4670  17    ...            10.1177/1352458512462920   \n",
       "\n",
       "                                                 Link  field_citation_ratio  \\\n",
       "2237  http://dx.doi.org10.1523/JNEUROSCI.4627-13.2014                  8.32   \n",
       "4328   http://dx.doi.org10.1016/S1474-4422(12)70311-3                  6.86   \n",
       "1401  http://dx.doi.org10.1523/JNEUROSCI.1034-14.2014                  3.00   \n",
       "4670        http://dx.doi.org10.1177/1352458512462920                  2.32   \n",
       "\n",
       "     highly_cited_1 highly_cited_10  highly_cited_5 recent_citations  \\\n",
       "2237          False            True           False             27.0   \n",
       "4328          False            True           False             12.0   \n",
       "1401          False           False           False              8.0   \n",
       "4670          False           False           False              6.0   \n",
       "\n",
       "     relative_citation_ratio times_cited  Citations  \n",
       "2237                    3.64        41.0       41.0  \n",
       "4328                    2.18        32.0       32.0  \n",
       "1401                    1.02        12.0       12.0  \n",
       "4670                    1.33        17.0       17.0  \n",
       "\n",
       "[4 rows x 23 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sample(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many articles do we have data for in our set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "JOURNAL OF NEUROSCIENCE       3243\n",
       "NEUROIMAGE                    1822\n",
       "NEURON                         644\n",
       "NEUROLOGY                      499\n",
       "BRAIN                          489\n",
       "MULTIPLE SCLEROSIS JOURNAL     412\n",
       "ANNALS OF NEUROLOGY            329\n",
       "ACTA NEUROPATHOLOGICA          184\n",
       "LANCET NEUROLOGY                91\n",
       "Name: SO, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['SO'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Check__ How many rows of tf-idf data did we create in the last step?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7711"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensional reduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dimensional reduction with TSVD\n",
      "Choosing  50  dimensions for initial step.  t-SNE will do the rest.\n",
      "TSVD took 0:00:02.322022\n"
     ]
    }
   ],
   "source": [
    "method = 'TSVD'\n",
    "\n",
    "print('Starting dimensional reduction with',method)\n",
    "# Peform dimensional reduction\n",
    "vectors = tfidf\n",
    "# truncated svd\n",
    "t_start = dt.now()\n",
    "# n of dimensions for initial step\n",
    "n_components = 50\n",
    "\n",
    "print('Choosing ',n_components,' dimensions for initial step.  t-SNE will do the rest.')\n",
    "\n",
    "if method =='TSVD':\n",
    "    X_reduced = TruncatedSVD(n_components = n_components,\n",
    "                            random_state = 0).fit_transform(vectors)\n",
    "else:\n",
    "    # PCA Option... seems to be slower than TSVD\n",
    "    X_reduced = PCA(n_components = n_components).fit_transform(vectors)\n",
    "\n",
    "dr_t = dt.now() - t_start\n",
    "print(method,'took', dr_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Continuing dimensional reduction with t-SNE\n",
      "[t-SNE] Computing 241 nearest neighbors...\n",
      "[t-SNE] Indexed 7711 samples in 0.084s...\n",
      "[t-SNE] Computed neighbors for 7711 samples in 9.873s...\n",
      "[t-SNE] Computed conditional probabilities for sample 1000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 2000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 3000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 4000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 5000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 6000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 7000 / 7711\n",
      "[t-SNE] Computed conditional probabilities for sample 7711 / 7711\n",
      "[t-SNE] Mean sigma: 0.019389\n",
      "[t-SNE] Computed conditional probabilities in 1.435s\n",
      "[t-SNE] Iteration 50: error = 79.7311020, gradient norm = 0.0000001 (50 iterations in 26.263s)\n",
      "[t-SNE] Iteration 50: gradient norm 0.000000. Finished.\n",
      "[t-SNE] KL divergence after 50 iterations with early exaggeration: 79.731102\n",
      "[t-SNE] Iteration 100: error = 2.6182425, gradient norm = 0.0032148 (50 iterations in 26.000s)\n",
      "[t-SNE] Iteration 150: error = 1.8156415, gradient norm = 0.0008747 (50 iterations in 26.211s)\n",
      "[t-SNE] Iteration 200: error = 1.5974711, gradient norm = 0.0004618 (50 iterations in 33.062s)\n",
      "[t-SNE] Iteration 250: error = 1.4863393, gradient norm = 0.0003077 (50 iterations in 31.698s)\n",
      "[t-SNE] Iteration 300: error = 1.4202746, gradient norm = 0.0002232 (50 iterations in 31.326s)\n",
      "[t-SNE] Iteration 350: error = 1.3785369, gradient norm = 0.0001594 (50 iterations in 31.618s)\n",
      "[t-SNE] Iteration 400: error = 1.3496603, gradient norm = 0.0001248 (50 iterations in 32.159s)\n",
      "[t-SNE] Iteration 450: error = 1.3287495, gradient norm = 0.0001053 (50 iterations in 31.170s)\n",
      "[t-SNE] Iteration 500: error = 1.3132448, gradient norm = 0.0000860 (50 iterations in 32.413s)\n",
      "[t-SNE] Iteration 550: error = 1.3018241, gradient norm = 0.0000763 (50 iterations in 31.336s)\n",
      "[t-SNE] Iteration 600: error = 1.2934388, gradient norm = 0.0000661 (50 iterations in 31.320s)\n",
      "[t-SNE] Iteration 650: error = 1.2871790, gradient norm = 0.0000601 (50 iterations in 31.043s)\n",
      "[t-SNE] Iteration 700: error = 1.2825731, gradient norm = 0.0000558 (50 iterations in 33.505s)\n",
      "[t-SNE] Iteration 750: error = 1.2789820, gradient norm = 0.0000530 (50 iterations in 32.071s)\n",
      "[t-SNE] Iteration 800: error = 1.2763858, gradient norm = 0.0000488 (50 iterations in 31.308s)\n",
      "[t-SNE] Iteration 850: error = 1.2738099, gradient norm = 0.0000577 (50 iterations in 31.206s)\n",
      "[t-SNE] Iteration 900: error = 1.2718959, gradient norm = 0.0000476 (50 iterations in 31.858s)\n",
      "[t-SNE] Iteration 950: error = 1.2705246, gradient norm = 0.0000452 (50 iterations in 31.245s)\n",
      "[t-SNE] Iteration 1000: error = 1.2694076, gradient norm = 0.0000486 (50 iterations in 32.079s)\n",
      "[t-SNE] Error after 1000 iterations: 1.269408\n",
      "Dimensional reduction complete!\n"
     ]
    }
   ],
   "source": [
    "print('Continuing dimensional reduction with t-SNE')\n",
    "# tsne\n",
    "X_embedded = TSNE(n_components = 2,  \n",
    "                  perplexity = perplexity, \n",
    "                  verbose = 3,\n",
    "                 n_iter = n_iter,\n",
    "                 n_iter_without_progress = n_iter_without_progress,\n",
    "                 learning_rate = learning_rate,\n",
    "                 ).fit_transform(X_reduced)\n",
    "print('Dimensional reduction complete!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((7713, 23), (7711, 2))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(data), np.shape(X_embedded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['DI', 'PY', 'WD', 'AU', 'AF', 'SO', 'SC', 'SN', 'EI', 'TC', 'Z9', 'DOI',\n",
       "       'Link', 'field_citation_ratio', 'highly_cited_1', 'highly_cited_10',\n",
       "       'highly_cited_5', 'recent_citations', 'relative_citation_ratio',\n",
       "       'times_cited', 'Citations', 'TSNE1', 'TSNE2'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try:\n",
    "    data = data.drop(['TSNE1','TSNE2'], axis=1)#.head()\n",
    "except:\n",
    "    pass\n",
    "data = pd.concat([data,pd.DataFrame(X_embedded)], axis =1)\n",
    "# test_df.head()\n",
    "data.rename(columns={0:'TSNE1',1:'TSNE2'}, inplace=True)\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# add PCA data as a check to see how closely TSNE matches it\n",
    "\n",
    "# X_embedded2 = PCA(n_components = 2).fit_transform(X_reduced)\n",
    "# X_embedded2\n",
    "# try:\n",
    "#     data = data.drop(['PCA1','PCA2'], axis=1).head()\n",
    "# except:\n",
    "#     pass\n",
    "# data = pd.concat([data,pd.DataFrame(X_embedded2)], axis =1)\n",
    "# # test_df.head()\n",
    "# data.rename(columns={0:'PCA1',1:'PCA2'}, inplace=True)\n",
    "# data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DI</th>\n",
       "      <th>PY</th>\n",
       "      <th>WD</th>\n",
       "      <th>AU</th>\n",
       "      <th>AF</th>\n",
       "      <th>SO</th>\n",
       "      <th>SC</th>\n",
       "      <th>SN</th>\n",
       "      <th>EI</th>\n",
       "      <th>TC</th>\n",
       "      <th>...</th>\n",
       "      <th>field_citation_ratio</th>\n",
       "      <th>highly_cited_1</th>\n",
       "      <th>highly_cited_10</th>\n",
       "      <th>highly_cited_5</th>\n",
       "      <th>recent_citations</th>\n",
       "      <th>relative_citation_ratio</th>\n",
       "      <th>times_cited</th>\n",
       "      <th>Citations</th>\n",
       "      <th>TSNE1</th>\n",
       "      <th>TSNE2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6107</th>\n",
       "      <td>10.1016/j.neuroimage.2013.02.070</td>\n",
       "      <td>2013</td>\n",
       "      <td>Spontaneous EEG alpha oscillation interacts wi...</td>\n",
       "      <td>Mayhew, SD; Ostwald, D; Porcaro, C; Bagshaw, AP</td>\n",
       "      <td>Mayhew, Stephen D.; Ostwald, Dirk; Porcaro, Ca...</td>\n",
       "      <td>NEUROIMAGE</td>\n",
       "      <td>Neurosciences &amp; Neurology; Radiology, Nuclear ...</td>\n",
       "      <td>1053-8119</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>...</td>\n",
       "      <td>6.12</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.15</td>\n",
       "      <td>32.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>31.124981</td>\n",
       "      <td>-27.585726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7513</th>\n",
       "      <td>10.1016/j.neuron.2013.07.002</td>\n",
       "      <td>2013</td>\n",
       "      <td>Distinct Representations of Cognitive and Moti...</td>\n",
       "      <td>Matsumoto, M; Takada, M</td>\n",
       "      <td>Matsumoto, Masayuki; Takada, Masahiko</td>\n",
       "      <td>NEURON</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0896-6273</td>\n",
       "      <td>1097-4199</td>\n",
       "      <td>47</td>\n",
       "      <td>...</td>\n",
       "      <td>9.75</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>28.0</td>\n",
       "      <td>2.86</td>\n",
       "      <td>51.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>-29.711529</td>\n",
       "      <td>-12.323181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5677</th>\n",
       "      <td>10.1016/j.neuroimage.2013.09.057</td>\n",
       "      <td>2014</td>\n",
       "      <td>Dynamic and static contributions of the cerebr...</td>\n",
       "      <td>Tak, S; Wang, DJJ; Polimeni, JR; Yan, LR; Chen...</td>\n",
       "      <td>Tak, Sungho; Wang, Danny J. J.; Polimeni, Jona...</td>\n",
       "      <td>NEUROIMAGE</td>\n",
       "      <td>Neurosciences &amp; Neurology; Radiology, Nuclear ...</td>\n",
       "      <td>1053-8119</td>\n",
       "      <td>1095-9572</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>3.25</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.76</td>\n",
       "      <td>16.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.684785</td>\n",
       "      <td>-32.776920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7094</th>\n",
       "      <td>10.1016/j.neuron.2014.10.014</td>\n",
       "      <td>2014</td>\n",
       "      <td>Cortical fosGFP Expression Reveals Broad Recep...</td>\n",
       "      <td>Jouhanneau, JS; Ferrarese, L; Estebanez, L; Au...</td>\n",
       "      <td>Jouhanneau, Jean-Sebastien; Ferrarese, Leiron;...</td>\n",
       "      <td>NEURON</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0896-6273</td>\n",
       "      <td>1097-4199</td>\n",
       "      <td>13</td>\n",
       "      <td>...</td>\n",
       "      <td>3.25</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>16.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>19.252451</td>\n",
       "      <td>-5.271960</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    DI    PY  \\\n",
       "6107  10.1016/j.neuroimage.2013.02.070  2013   \n",
       "7513      10.1016/j.neuron.2013.07.002  2013   \n",
       "5677  10.1016/j.neuroimage.2013.09.057  2014   \n",
       "7094      10.1016/j.neuron.2014.10.014  2014   \n",
       "\n",
       "                                                     WD  \\\n",
       "6107  Spontaneous EEG alpha oscillation interacts wi...   \n",
       "7513  Distinct Representations of Cognitive and Moti...   \n",
       "5677  Dynamic and static contributions of the cerebr...   \n",
       "7094  Cortical fosGFP Expression Reveals Broad Recep...   \n",
       "\n",
       "                                                     AU  \\\n",
       "6107    Mayhew, SD; Ostwald, D; Porcaro, C; Bagshaw, AP   \n",
       "7513                            Matsumoto, M; Takada, M   \n",
       "5677  Tak, S; Wang, DJJ; Polimeni, JR; Yan, LR; Chen...   \n",
       "7094  Jouhanneau, JS; Ferrarese, L; Estebanez, L; Au...   \n",
       "\n",
       "                                                     AF          SO  \\\n",
       "6107  Mayhew, Stephen D.; Ostwald, Dirk; Porcaro, Ca...  NEUROIMAGE   \n",
       "7513              Matsumoto, Masayuki; Takada, Masahiko      NEURON   \n",
       "5677  Tak, Sungho; Wang, Danny J. J.; Polimeni, Jona...  NEUROIMAGE   \n",
       "7094  Jouhanneau, Jean-Sebastien; Ferrarese, Leiron;...      NEURON   \n",
       "\n",
       "                                                     SC         SN         EI  \\\n",
       "6107  Neurosciences & Neurology; Radiology, Nuclear ...  1053-8119        NaN   \n",
       "7513                          Neurosciences & Neurology  0896-6273  1097-4199   \n",
       "5677  Neurosciences & Neurology; Radiology, Nuclear ...  1053-8119  1095-9572   \n",
       "7094                          Neurosciences & Neurology  0896-6273  1097-4199   \n",
       "\n",
       "      TC    ...      field_citation_ratio highly_cited_1 highly_cited_10  \\\n",
       "6107  27    ...                      6.12          False           False   \n",
       "7513  47    ...                      9.75          False            True   \n",
       "5677  11    ...                      3.25          False           False   \n",
       "7094  13    ...                      3.25          False           False   \n",
       "\n",
       "      highly_cited_5 recent_citations relative_citation_ratio times_cited  \\\n",
       "6107           False             18.0                    2.15        32.0   \n",
       "7513            True             28.0                    2.86        51.0   \n",
       "5677           False              7.0                    1.76        16.0   \n",
       "7094           False             12.0                    1.00        16.0   \n",
       "\n",
       "      Citations      TSNE1      TSNE2  \n",
       "6107       32.0  31.124981 -27.585726  \n",
       "7513       51.0 -29.711529 -12.323181  \n",
       "5677       16.0   0.684785 -32.776920  \n",
       "7094       16.0  19.252451  -5.271960  \n",
       "\n",
       "[4 rows x 23 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sample(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7713, 23)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DI</th>\n",
       "      <th>PY</th>\n",
       "      <th>WD</th>\n",
       "      <th>AU</th>\n",
       "      <th>AF</th>\n",
       "      <th>SO</th>\n",
       "      <th>SC</th>\n",
       "      <th>SN</th>\n",
       "      <th>EI</th>\n",
       "      <th>TC</th>\n",
       "      <th>...</th>\n",
       "      <th>field_citation_ratio</th>\n",
       "      <th>highly_cited_1</th>\n",
       "      <th>highly_cited_10</th>\n",
       "      <th>highly_cited_5</th>\n",
       "      <th>recent_citations</th>\n",
       "      <th>relative_citation_ratio</th>\n",
       "      <th>times_cited</th>\n",
       "      <th>Citations</th>\n",
       "      <th>TSNE1</th>\n",
       "      <th>TSNE2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7408</th>\n",
       "      <td>10.1016/j.neuron.2013.09.028</td>\n",
       "      <td>2013</td>\n",
       "      <td>Causal Evidence of Performance Monitoring by N...</td>\n",
       "      <td>Heilbronner, SR; Platt, ML</td>\n",
       "      <td>Heilbronner, Sarah R.; Platt, Michael L.</td>\n",
       "      <td>NEURON</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0896-6273</td>\n",
       "      <td>1097-4199</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>4.78</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.16</td>\n",
       "      <td>20.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>41.914577</td>\n",
       "      <td>27.885458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4463</th>\n",
       "      <td>10.1177/1352458513509507</td>\n",
       "      <td>2014</td>\n",
       "      <td>Hypovitaminosis-D and EBV: no interdependence ...</td>\n",
       "      <td>Ramien, C; Pachnio, A; Sisay, S; Begum, J; Lee...</td>\n",
       "      <td>Ramien, Caren; Pachnio, Annette; Sisay, Sofia;...</td>\n",
       "      <td>MULTIPLE SCLEROSIS JOURNAL</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>1352-4585</td>\n",
       "      <td>1477-0970</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>2.00</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.90</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>-24.931896</td>\n",
       "      <td>-47.509399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7605</th>\n",
       "      <td>10.1016/j.neuron.2013.03.004</td>\n",
       "      <td>2013</td>\n",
       "      <td>Plum, an Immunoglobulin Superfamily Protein, R...</td>\n",
       "      <td>Yu, XMM; Gutman, I; Mosca, TJ; Iram, T; Ozkan,...</td>\n",
       "      <td>Yu, Xiaomeng M.; Gutman, Itai; Mosca, Timothy ...</td>\n",
       "      <td>NEURON</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0896-6273</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>3.03</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.06</td>\n",
       "      <td>22.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>16.197462</td>\n",
       "      <td>33.142780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2561</th>\n",
       "      <td>10.1523/JNEUROSCI.3029-13.2013</td>\n",
       "      <td>2013</td>\n",
       "      <td>The Transcription Factor Serum Response Factor...</td>\n",
       "      <td>Stern, S; Haverkamp, S; Sinske, D; Tedeschi, A...</td>\n",
       "      <td>Stern, Sina; Haverkamp, Stephanie; Sinske, Dan...</td>\n",
       "      <td>JOURNAL OF NEUROSCIENCE</td>\n",
       "      <td>Neurosciences &amp; Neurology</td>\n",
       "      <td>0270-6474</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>3.08</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.07</td>\n",
       "      <td>20.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.311239</td>\n",
       "      <td>1.133781</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  DI    PY  \\\n",
       "7408    10.1016/j.neuron.2013.09.028  2013   \n",
       "4463        10.1177/1352458513509507  2014   \n",
       "7605    10.1016/j.neuron.2013.03.004  2013   \n",
       "2561  10.1523/JNEUROSCI.3029-13.2013  2013   \n",
       "\n",
       "                                                     WD  \\\n",
       "7408  Causal Evidence of Performance Monitoring by N...   \n",
       "4463  Hypovitaminosis-D and EBV: no interdependence ...   \n",
       "7605  Plum, an Immunoglobulin Superfamily Protein, R...   \n",
       "2561  The Transcription Factor Serum Response Factor...   \n",
       "\n",
       "                                                     AU  \\\n",
       "7408                         Heilbronner, SR; Platt, ML   \n",
       "4463  Ramien, C; Pachnio, A; Sisay, S; Begum, J; Lee...   \n",
       "7605  Yu, XMM; Gutman, I; Mosca, TJ; Iram, T; Ozkan,...   \n",
       "2561  Stern, S; Haverkamp, S; Sinske, D; Tedeschi, A...   \n",
       "\n",
       "                                                     AF  \\\n",
       "7408           Heilbronner, Sarah R.; Platt, Michael L.   \n",
       "4463  Ramien, Caren; Pachnio, Annette; Sisay, Sofia;...   \n",
       "7605  Yu, Xiaomeng M.; Gutman, Itai; Mosca, Timothy ...   \n",
       "2561  Stern, Sina; Haverkamp, Stephanie; Sinske, Dan...   \n",
       "\n",
       "                              SO                         SC         SN  \\\n",
       "7408                      NEURON  Neurosciences & Neurology  0896-6273   \n",
       "4463  MULTIPLE SCLEROSIS JOURNAL  Neurosciences & Neurology  1352-4585   \n",
       "7605                      NEURON  Neurosciences & Neurology  0896-6273   \n",
       "2561     JOURNAL OF NEUROSCIENCE  Neurosciences & Neurology  0270-6474   \n",
       "\n",
       "             EI  TC    ...      field_citation_ratio highly_cited_1  \\\n",
       "7408  1097-4199  16    ...                      4.78          False   \n",
       "4463  1477-0970   7    ...                      2.00          False   \n",
       "7605        NaN  18    ...                      3.03          False   \n",
       "2561        NaN  20    ...                      3.08          False   \n",
       "\n",
       "     highly_cited_10  highly_cited_5 recent_citations relative_citation_ratio  \\\n",
       "7408           False           False             13.0                    1.16   \n",
       "4463           False           False              9.0                    0.90   \n",
       "7605           False           False              6.0                    1.06   \n",
       "2561           False           False             11.0                    1.07   \n",
       "\n",
       "     times_cited  Citations      TSNE1      TSNE2  \n",
       "7408        20.0       20.0  41.914577  27.885458  \n",
       "4463        11.0       11.0 -24.931896 -47.509399  \n",
       "7605        22.0       22.0  16.197462  33.142780  \n",
       "2561        20.0       20.0   1.311239   1.133781  \n",
       "\n",
       "[4 rows x 23 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.sample(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# write to file\n",
    "data.to_csv(working_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done in 0:10:37.121142\n"
     ]
    }
   ],
   "source": [
    "print('Done in '+str(dt.now()-t_start))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}