intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center'>Dropout as Regularization and Bayesian Approximation</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <h3 align='center'>Weidong Xu, Zeyu Zhao, Tianning Zhao</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "Deep neural network is a very powerful tool in machine learning. Multiple non-linear hidden layers enable the model to learn complicated relationships between input and output. However, when the training set is small, there are different parameter settings that would fits training set perfectly, but the one complex parameter setting tends to perform poorly on the test dataset, ie we got the problem of overfitting. One way to solve this problem is by averaging predictions of different neural networks , but this becomes computationally expensive when applied to large datasets. The alternative that makes it possible to train a huge number of different networks in a reasonable time is dropout, which randomly omits some hidden units i.e. feature detectors to prevent co-adaption. The idea of dropout model can be shown as below.\n",
    "![Dropout_model.png](Dropout_model.png)\n",
    "ref: Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.\n",
    "\n",
    "## 2. Model Description\n",
    "With dropout, the feed-forward operation of neural networks can be described as :\n",
    "\n",
    "$$r_j^{(l)} \\sim Bernoulli(p),$$\n",
    "$$\\tilde y^{(l)} = r^{(l)} \\times y^{(l)},$$\n",
    "$$z_i^{(l+1)} = w_i^{(l+1)}\\tilde y^{(l)} + b_i^{(l+1)},$$\n",
    "$$y_i^{(l+1)} = f(z_i^{(l+1)})$$\n",
    "\n",
    "where $l$ is the index the hidden layers of the network, $z^{(l)}$ denote\n",
    "the vector of inputs into layer $l$, $y^{(l)}$ denote the vector of outputs from layer $l$ ($y^{(0)} = x$ is the input). $w^{(l)}$ and $b^{(l)}$ are the weights and biases at layer $l$. f is any activation function.\n",
    "\n",
    "With this, the standard stochastic gradient descent procedure was used for training the dropout neural networks on mini-batches of training cases.  \n",
    "\n",
    "For testing, we use the “mean network” that contains all of the hidden units with their outgoing weights halved instead of averaging over a large number of dropout networks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}