msan694_disco/MSAN694-Lec1-Notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slides 1-3 : About Diane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents \n",
    "\n",
    "1. Class Overview\n",
    "2. Motivation \n",
    "3. What is Distributed Computing?\n",
    "4. Spark\n",
    "5. RDD Creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Class Overview\n",
    "#### Slides 6-11 : about the course "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Spark Interview Questions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What is Apache Spark?\n",
    "2. Explain the key features of Spark. What is RDD?\n",
    "3. How to create RDD.\n",
    "4. What is ”partitions”?\n",
    "5. Types of RDD operations?\n",
    "6. What is “transformation”?\n",
    "7. What is “action”?\n",
    "8. Functions of “spark core”?\n",
    "9. What is “spark context”?\n",
    "10. What is an “RDD lineage”?\n",
    "11. Which file systems does Spark support?\n",
    "12. List the various types of “Cluster Managers” in Spark.\n",
    "13. What is “YARN”?\n",
    "14. What is “Mesos”?\n",
    "15. What is a “worker node”? \n",
    "16. What is an “accumulator”? \n",
    "17. What is “Spark SQL” (Shark)? What is “SparkStreaming”? \n",
    "18. What is “GraphX”?\n",
    "19. What is “MLlib”?\n",
    "20. What are the advantages of using Apache Spark over Hadoop MapReduce for big data processing?\n",
    "21. What are the languages supported by Apache Spark for developing big data applications?\n",
    "22. Can you use Spark to access and analyze data stored in Cassandra databases?\n",
    "23. Is it possible to run Apache Spark on Apache Mesos?\n",
    "24. How can you minimize data transfers when working with Spark?\n",
    "25. Why is there a need for broadcast variables?\n",
    "26. Name a few companies that use Apache Spark in production.\n",
    "27. What are the various data sources available in SparkSQL?\n",
    "28. What is the advantage of a Parquet file?\n",
    "29. What do you understand by Pair RDD?\n",
    "30. Is Apache Spark a good fit for Reinforcement learning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Motivation: Why Distributed Computing? (Slide 14)\n",
    "- How to make a line move faster\n",
    "- Counting or Ordering Cards\n",
    "- Searching for the max numbers in multiple sets of numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: What is Distributed Computing? (Slide 19)\n",
    "\n",
    "** cluster ** : collection of systems that work together to perform functions\n",
    "\n",
    "** node ** : individual servers within a cluster\n",
    "\n",
    "** scale up ** : getting a faster performing complex machine\n",
    "\n",
    "** scale out **: getting more simple machines to work in parallel \n",
    "\n",
    "** why out instead of up **?\n",
    "1. ** Cheaper**: Easier to collect many cheaper machines instead of a super high performing machine\n",
    "2. ** Reliable**: If a node fails, another node can assume the workload or the responsibility of the faulty node\n",
    "2. ** Faster** : with current tech, a single machine can only run fast, in contrast, an infinite number of parallel computers can be added via networking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Map Reduce\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Big Idea** - the data will be split up over multiple nodes (computers), and tasks will be **mapped** and run on each data subsets. Afterwards, the data will be shffled and collected and **reduced**.\n",
    "\n",
    "\n",
    "** map **: apply a function to each key-value pair over a subset of the data. This happens in parallel on different nodes, simultaneously. *Example: filtering for even numbers in each node's data subset*\n",
    "\n",
    "** reduce **: return only 1 key-value pair from multiple pairs of data. This is usually done with an aggregate function such as sum, count, etc.\n",
    "\n",
    "\n",
    "Walmart Order\n",
    "OrderID: 1\n",
    "Customer : Diane\n",
    "Timestamp : 2017-08-15 05:04:32 PST\n",
    "Items:\n",
    "\n",
    "#### Step 1: Orders distributed over multiple nodes\n",
    "\n",
    "|Order No| Node | item number | data |\n",
    "|--------|--|---|-------------|\n",
    "|101|A|1 |{ProductName : San Francisco Giants Hat M, {Qty: 1, UnitPrice : 10, Price : 10}}|\n",
    "|102|B|1 |{ProductName : San Francisco Giants Hat M, {Qty: 2, UnitPrice : 10, Price : 20}}|\n",
    "|102|B|2|{ProductName : San Francisco Giants Hat S,{ Qty: 3, UnitPrice : 8, Price : 24}} Shipping: Corte Madera|\n",
    "\n",
    "#### Step 2: Map a function - pull out only the product and quantity information\n",
    "\n",
    "|Node| item number | data |\n",
    "|-|---|-------------|\n",
    "|A|1 |{San Francisco Giants Hat M, {Qty:1, Price : 10} |\n",
    "|B|1 |{San Francisco Giants Hat M, {Qty:2, Price : 20} |\n",
    "|B|2 |{San Francisco Giants Hat S, {Qty:3, Price : 24}|\n",
    "\n",
    "#### Step 3: Reduce the Key Value pairs (condense M hats)\n",
    "\n",
    "| item number | data |\n",
    "|---|-------------|\n",
    "|1 |{San Francisco Giants Hat M, {Qty:3, Price : 30} |\n",
    "|2 |{San Francisco Giants Hat S, {Qty:3, Price : 24}|\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Hadoop Map Reduce\n",
    "\n",
    "**Description**: Open source, distributed, Java computation framework consisting of\n",
    "- Hadoop common\n",
    "- Hadoop Distributed File System (HDFS)\n",
    "- YARN\n",
    "- MapReduce\n",
    "\n",
    "Solved issues of:\n",
    "- Distribution\n",
    "- Parallelism\n",
    "- Fault Tolerance\n",
    "\n",
    "Limitations:\n",
    "- **Slow**: MapReduce Jobs need to be stored in disk before used by another job. Slow with iterative algorithms\n",
    "- **MapReduce not always a good fit** many kinds of problems dont easily fit Map Reduce's two step paradigm\n",
    "- **Low-level framework**: other tools have been made to work with it, but leads to tool fragmentation and increased complexity to use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Spark (Slide 24)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** A quick comparison**\n",
    "\n",
    "||HadoopMapReuce|Spark|\n",
    "|-------|---------|-------------------|\n",
    "|Speed|Decently fast|100 times faster than Hadoop|\n",
    "|Ease of Use|No interactive modes and Hard to learn|Provides interactive modes and Easy to learn|\n",
    "|$Costs|Open source|Open source|\n",
    "|Data Processing|Batch Processing|Batch Processing + Streaming|\n",
    "|Fault Tolerence|Fault Torelent |Fault Torelent|\n",
    "|Security|Kerberos authentication|Password authentification|\n",
    "\n",
    "||HadoopMapReuce|Spark|\n",
    "|-------|---------|-------------------|\n",
    "|Operations|Will keep datasubsets on Disk|will keep data subsets in RAM|\n",
    "\n",
    "![](https://www.packtpub.com/sites/default/files/Article-Images/B05195_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Spark Benefits & Similiarities to Mapreduce\n",
    "\n",
    "- Write distributed programs in a similiar way to writing local programs (in python, R, java .. etc)\n",
    "- Spark combines the following:\n",
    "    - batch processing\n",
    "    - real time data processing\n",
    "    - SQL-like handling\n",
    "    - graph algorithms\n",
    "    - machine learning\n",
    "    \n",
    "** Main data object used in Spark: RDD, Resilient Distributed Datasets **\n",
    "\n",
    "| Aspect | Structure |\n",
    "|------|------------|\n",
    "|Speed | Runs in memory, uses DAG, and cyclical data flow |\n",
    "|Ease of use | Offers 80+ high level operators (treated like libraries, that are developer friendly). Also has shells for Scala, Python and R.\n",
    "| Code lines | Spark code often only requires 1/3 # of lines\n",
    "| Generality| <ul><li>Spark SQL</li><li>Spark Streaming</li><li> MLib (machine learning)</li> <li>GraphX</li></ul>\n",
    "| Runs On| <ul><li>Standalone</li><li>YARN</li><li>Mesos</li></ul>|\n",
    "| Can Access| <ul><li>HDFS</li><li>Cassandra</li><li>HBASE</li><li>HIVE</li><li>Tachyon</li></ul>|\n",
    "    \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Spark Stack (components)\n",
    "\n",
    "![](https://www.safaribooksonline.com/library/view/learning-spark/9781449359034/assets/lnsp_0101.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note on common data objects in spark: DataFrames vs. RDD\n",
    "So, a `DataFrame` has additional metadata due to its tabular format, which allows Spark to run certain optimizations on the finalized query. ** Added in Spark 2.0 **\n",
    "\n",
    "An `RDD`, on the other hand, is merely a Resilient Distributed Dataset that is more of a blackbox of data that cannot be optimized as the operations that can be performed against it, are not as constrained.  ** Since Spark 1.0 **\n",
    "\n",
    "#### Spark Core:\n",
    "\n",
    "1. **Data abstraction & logic:** Connects RDDs to underlying distributed file system, such as S3, HDFS, GlusterFS\n",
    "2. **Fundamental Functions : ** Networking, security, job scheduling, data shuffling\n",
    "\n",
    "#### Spark SQL:\n",
    "\n",
    "1. **Provides Data Manipulation** for structured datasets\n",
    "2. **Operates on DataFrames + Datasets** - transforms these operations into operations on RDDs ( see spark core)\n",
    "3. **Database compatibility : ** Hive, JSON stores, relational databases, NoSQL, and Parquet ( for additional datasets)\n",
    "\n",
    "#### Spark Streaming:\n",
    "\n",
    "1. **Ingest real-time data compatibility**: \n",
    "    - HDFS\n",
    "    - Kafka\n",
    "    - Flume\n",
    "    - Twitter\n",
    "    - ZeroMQ\n",
    "    - many more...\n",
    "2. ** Automatic recovery **\n",
    "3. ** Provides DStreams = periodic RDDs per timing window **\n",
    "4. ** Compatible with other Spark components **\n",
    "    - Core\n",
    "    - ML\n",
    "    - Mllib\n",
    "    - GraphX\n",
    "    - SQL\n",
    "    \n",
    "#### Spark MLlib + ML\n",
    "\n",
    "1. **MLlib** : RDD-based APIs (spark core)\n",
    "2. **Spark ML** : DataFrame-based APIs (spark SQL)\n",
    "3. Library of machine learning algorithms designed to run on a distributed dataset.\n",
    "    - logistic regression\n",
    "    - naive bayes\n",
    "    - support vector machines\n",
    "    - decision trees\n",
    "    - random forests\n",
    "    - linear regression\n",
    "    - k-mean clustering\n",
    "\n",
    "#### Spark GraphX\n",
    "\n",
    "1. ** Graph functions: ** such as EdgeRDD and VertexRDD\n",
    "2. ** Graph Algos: ** including\n",
    "    - page rank\n",
    "    - connected components\n",
    "    - shortest paths\n",
    "    - SVD++\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Spark Example Operations\n",
    "\n",
    "- Extract-transformation-load (ETL) operations\n",
    "- Predictive analytics\n",
    "- Machine learning\n",
    "- Data access operation (SQL queries and visualizations) Text mining and text processing\n",
    "- Real-time event processing Graph applications\n",
    "- Pattern Recognition Recommendation engines\n",
    "- And many more.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Spark installation - slide 43-48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}