SetupJupyter.md

Setup Jupyter with PySpark and Parallel IPython

Install Jupyter Notebook at once!

fab install-jupyter

Only jupyter

# This will install in /usr/local/bin (share with all user)
sudo pip3 install jupyter
# This will install in ~/.local/bin
pip3 install --user jupyter

Jupyter Notebook Server

• Jupyter Notebook Doc - Running a notebook server

jupyter notebook --generate-config

# Setup password
jupyter notebook password

Start the server

jupyter notebook --ip <your_LAN_ip> --port 8888

• https://stackoverflow.com/questions/39155953/exposing-python-jupyter-on-lan

Either use password or token or ... to login

PySpark with Jupyter Notebook

• Get Started with PySpark and Jupyter Notebook in 3 Minutes
• How to set up PySpark for your Jupyter notebook

Interactive Parallel Computing with IPython

Haven't tried yet

• ipyparallel Github
• ipyparallel documents

Online Jupyter Notebook Server

Google Colaboratory

# Put at the start of the notebook
!apt-get -y install openjdk-8-jre-headless
!pip install pyspark
from pyspark.sql import SparkSession
from pyspark import SparkContext
spark = SparkSession.builder.master("local").getOrCreate()
sc = SparkContext.getOrCreate()

• 使用 Google Colaboratory 跑 PySpark
• Apache Spark on Google Colaboratory

AWS SageMaker

• Build Amazon SageMaker notebooks backed by Spark in Amazon EMR