1_tensor_basics.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using mxnet backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorly as tl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors are a natural one to represent multi-dimensional data\n",
    "\n",
    "![tensor_illustration](img/tensor_cartoon.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.Creating a tensor\n",
    "\n",
    "A tensor is nothing more than a multi-dimensional array.\n",
    "\n",
    "Let's take for this example the tensor $\\tilde X$ defined by its frontal slices:\n",
    "\n",
    "$$\n",
    "   X_1 = \n",
    "   \\left[\n",
    "   \\begin{matrix}\n",
    "   0  & 2  & 4  & 6\\\\\n",
    "   8  & 10 & 12 & 14\\\\\n",
    "   16 & 18 & 20 & 22\n",
    "   \\end{matrix}\n",
    "   \\right]\n",
    "$$\n",
    "\n",
    "and \n",
    "\n",
    "$$\n",
    "   X_2 =\n",
    "   \\left[\n",
    "   \\begin{matrix}\n",
    "   1  & 3  & 5  & 7\\\\\n",
    "   9  & 11 & 13 & 15\\\\\n",
    "   17 & 19 & 21 & 23\n",
    "   \\end{matrix}\n",
    "   \\right]\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "In Python, this array can be expressed as a numpy array::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = tl.tensor(np.arange(24).reshape((3, 4, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[[  0.   1.]\n",
       "  [  2.   3.]\n",
       "  [  4.   5.]\n",
       "  [  6.   7.]]\n",
       "\n",
       " [[  8.   9.]\n",
       "  [ 10.  11.]\n",
       "  [ 12.  13.]\n",
       "  [ 14.  15.]]\n",
       "\n",
       " [[ 16.  17.]\n",
       "  [ 18.  19.]\n",
       "  [ 20.  21.]\n",
       "  [ 22.  23.]]]\n",
       "<NDArray 3x4x2 @cpu(0)>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view the frontal slices by fixing the last axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[  0.   2.   4.   6.]\n",
       " [  8.  10.  12.  14.]\n",
       " [ 16.  18.  20.  22.]]\n",
       "<NDArray 3x4 @cpu(0)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:, :, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[  1.   3.   5.   7.]\n",
       " [  9.  11.  13.  15.]\n",
       " [ 17.  19.  21.  23.]]\n",
       "<NDArray 3x4 @cpu(0)>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:, :, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.Setting the backend "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mxnet.ndarray.NDArray"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using numpy backend.\n"
     ]
    }
   ],
   "source": [
    "tl.set_backend('numpy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = tl.tensor(np.arange(24).reshape((3, 4, 2)))\n",
    "type(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using mxnet backend.\n"
     ]
    }
   ],
   "source": [
    "tl.set_backend('mxnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mxnet.ndarray.NDArray"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = tl.tensor(np.arange(24).reshape((3, 4, 2)))\n",
    "type(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.Basic tensor operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Unfolding\n",
    "\n",
    "Also called **matrization**, **unfolding** a tensor is done by reading the element in a given way as to obtain a matrix instead of a tensor.\n",
    "\n",
    "It is done by stacking the **fibers** of the tensor into a matrix.\n",
    "\n",
    "![tensor_illustration](img/fibers.png)\n",
    "Illustration: *Nonnegative Matrix and Tensor Factorizations*, Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-ichi Amari, John Wiley & Sons, 2009.\n",
    "\n",
    "\n",
    "\n",
    "### Definition\n",
    "For a tensor of size $(I_1, I_2, \\cdots, I_n)$, the k-mode unfolding of this tensor will be of size $(I_k, I_1 \\times \\cdots \\times I_{k-1} \\times I_{k+1} \\cdots \\times I_n)$.\n",
    "\n",
    "\n",
    "   Given a tensor $\\tilde X \\in \\mathbb{R}^{I_1 \\times I_2 \\times \\cdots \\times I_N}$, the\n",
    "   mode-n unfolding of $\\tilde X$ is a matrix $\\mathbf{X}_{[n]} \\in \\mathbb{R}^{I_n, I_M}$,\n",
    "   with $M = \\prod_{\\substack{k=1,\\\\k \\neq n}}^N I_k$ and is defined by\n",
    "   the mapping from element $(i_1, i_2, \\cdots, i_N)$ to $(i_n, j)$, with\n",
    "\n",
    "\n",
    "$$\n",
    "    j = \\sum_{\\substack{k=1,\\\\k \\neq n}}^N i_k \\times \\prod_{m=k+1}^N I_m.\n",
    "$$\n",
    "\n",
    "### Convention\n",
    "\n",
    "   Traditionally, mode-1 unfolding denotes the unfolding along the first dimension.\n",
    "   However, to be consistent with the Python indexing that always starts at zero,\n",
    "   in tensorly, unfolding also starts at zero!\n",
    "\n",
    "   Therefore ``unfold(tensor, 0)`` will unfold said tensor along its first dimension!\n",
    "   \n",
    "\n",
    "### Example\n",
    "\n",
    "For instance, using the $\\tilde X$ previously defined:\n",
    "$$\n",
    "   X_1 = \n",
    "   \\left[\n",
    "   \\begin{matrix}\n",
    "   0  & 2  & 4  & 6\\\\\n",
    "   8  & 10 & 12 & 14\\\\\n",
    "   16 & 18 & 20 & 22\n",
    "   \\end{matrix}\n",
    "   \\right]\n",
    "$$\n",
    "\n",
    "and \n",
    "\n",
    "$$\n",
    "   X_2 =\n",
    "   \\left[\n",
    "   \\begin{matrix}\n",
    "   1  & 3  & 5  & 7\\\\\n",
    "   9  & 11 & 13 & 15\\\\\n",
    "   17 & 19 & 21 & 23\n",
    "   \\end{matrix}\n",
    "   \\right]\n",
    "$$\n",
    "\n",
    "The 0-mode unfolding of $\\tilde X$:\n",
    "\n",
    "$$\n",
    "   \\tilde X_{[0]} =\n",
    "   \\left[ \\begin{matrix}\n",
    "      0 & 1 & 2 & 3 & 4 & 5 & 6 & 7\\\\\n",
    "      8 & 9 & 10 & 11 & 12 & 13 & 14 & 15\\\\\n",
    "      16 & 17 & 18 & 19 & 20 & 21 & 22 & 23\\\\\n",
    "   \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "The 1-mode unfolding is given by:\n",
    "\n",
    "$$\n",
    "   \\tilde X_{[1]} =\n",
    "   \\left[ \\begin{matrix}\n",
    "      0 & 1 & 8 & 9 & 16 & 17\\\\\n",
    "      2 & 3 & 10 & 11 & 18 & 19\\\\\n",
    "      4 & 5 & 12 & 13 & 20 & 21\\\\\n",
    "      6 & 7 & 14 & 15 & 22 & 23\\\\\n",
    "   \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "Finally, the 2-mode unfolding is the unfolding along the last axis:\n",
    "\n",
    "$$\n",
    "    \\tilde X_{[2]} =\n",
    "   \\left[ \\begin{matrix}\n",
    "      0 & 2 & 4 & 6 & 8 & 10 & 12 & 14 & 16 & 18 & 20 & 22\\\\\n",
    "      1 & 3 & 5 & 7 & 9 & 11 & 13 & 15 & 17 & 19 & 21 & 23\\\\\n",
    "   \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "### In TensorLy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[  0.   1.   2.   3.   4.   5.   6.   7.]\n",
       " [  8.   9.  10.  11.  12.  13.  14.  15.]\n",
       " [ 16.  17.  18.  19.  20.  21.  22.  23.]]\n",
       "<NDArray 3x8 @cpu(0)>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl.unfold(X, mode=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[  0.   1.   8.   9.  16.  17.]\n",
       " [  2.   3.  10.  11.  18.  19.]\n",
       " [  4.   5.  12.  13.  20.  21.]\n",
       " [  6.   7.  14.  15.  22.  23.]]\n",
       "<NDArray 4x6 @cpu(0)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl.unfold(X, mode=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[  0.   2.   4.   6.   8.  10.  12.  14.  16.  18.  20.  22.]\n",
       " [  1.   3.   5.   7.   9.  11.  13.  15.  17.  19.  21.  23.]]\n",
       "<NDArray 2x12 @cpu(0)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tl.unfold(X, mode=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Folding\n",
    "\n",
    "Folding is the inverse operation: you can **fold** an unfolded tensor back from matrix to full tensor using the ``tensorly.fold`` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[[  0.   1.]\n",
       "  [  2.   3.]\n",
       "  [  4.   5.]\n",
       "  [  6.   7.]]\n",
       "\n",
       " [[  8.   9.]\n",
       "  [ 10.  11.]\n",
       "  [ 12.  13.]\n",
       "  [ 14.  15.]]\n",
       "\n",
       " [[ 16.  17.]\n",
       "  [ 18.  19.]\n",
       "  [ 20.  21.]\n",
       "  [ 22.  23.]]]\n",
       "<NDArray 3x4x2 @cpu(0)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unfolding = tl.unfold(X, 1)\n",
    "original_shape = X.shape\n",
    "tl.fold(unfolding, mode=1, shape=original_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 n-mode product\n",
    "\n",
    "This is a natural generalization of matrix-vector and matrix-matrix product. A tensor has mode than two dimensions, so when multiplying a tensor by a matrix or a vector, we now have to specify the **mode** $n$ along which to take the product. It is a way of *contracting* along a specific mode.\n",
    "\n",
    "### Tensor times matrix\n",
    "\n",
    "In that case we are doing an operation analogous to a matrix multiplication on the $n$-th mode. Given a tensor $\\tilde X$ of size $(I_1, I_2, \\cdots, I_N)$, and a matrix $M$ of size $(D, I_n)$, the $n$-mode product of $\\tilde X$ by $M$ is written $\\tilde X \\times_n M$ and is of size $(I_k, I_1 \\times \\cdots \\times I_{n-1} \\times D \\times I_{n+1} \\cdots \\times I_n)$.\n",
    "\n",
    "### Tensor times vector\n",
    "\n",
    "In that case we are contracting over the $n$-th mode by multiplying it with a vector. Given a tensor $\\tilde X$ of size $(I_1, I_2, \\cdots, I_N)$, and a vector $v$ of size $(I_n)$, the $n$-mode product of $\\tilde X$ by $v$ is written $\\tilde X \\times_n v$ and is of size $(I_k, I_1 \\times \\cdots \\times I_{n-1} \\times I_{n+1} \\cdots \\times I_n)$.\n",
    "\n",
    "![tensor_illustration](img/tensor_contraction.png)\n",
    "\n",
    "\n",
    "### Example\n",
    "\n",
    "With the tensor $\\tilde X$ of size (3, 4, 2) we defined previously, let's define a matrix M of size (5, 4) to multiply along the second mode (mode 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 4)\n"
     ]
    }
   ],
   "source": [
    "M = tl.tensor(np.arange(4*5).reshape((5, 4)))\n",
    "print(M.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorLy, all the tensor algebra functions are in the `tensorly.tenalg` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "res = tl.tenalg.mode_dot(X, M, mode=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected the result is of shape (3, 5, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 5, 2)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, for a vector of size 4 and contract along the mode 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4,)\n"
     ]
    }
   ],
   "source": [
    "v = tl.tensor(np.arange(4))\n",
    "print(v.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "res = tl.tenalg.mode_dot(X, v, mode=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have multiplied by a vector, we have effectively contracted out one mode of the tensor so the result is a matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 2)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}