Use apache Spark with jupyter notebook (IPython notebook)

Conclusion

Just run the following command:

PYSPARK_PYTHON=/Users/username/.pyenv/shims/python PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ~/somewhere/spark-1.5.2/bin/pyspark --master local[4]

Premise

--jupyter notebook and Apache Spark are installed

starting method

Conclusion

--Basically, just start it as described in https://spark.apache.org/docs/latest/programming-guide.html#using-the-shell.

--Just set the environment variables for $ {PYSPARK_DRIVER_PYTHON} and $ {PYSPARK_DRIVER_PYTHON_OPTS} correctly and start pyspark. --This time, add the --master local [4] option to test on the node at hand. --Also, to make sure that the master and worker use the same version of python, I specified the python path in $ {PYSPARK_PYTHON}. --You can find the path to python in your environment by looking up which python. --The above commands are summarized below.

PYSPARK_PYTHON=/Users/username/.pyenv/shims/python PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ~/somewhere/spark-1.5.2/bin/pyspark --master local[4]

Recommended Posts

Use apache Spark with jupyter notebook (IPython notebook)
Use pip with Jupyter Notebook
Use Cython with Jupyter Notebook
Play with Jupyter Notebook (IPython Notebook)
Use Bokeh with IPython Notebook
Use markdown with jupyter notebook (with shortcut)
Use nb extensions with Anaconda's Jupyter notebook
Use Jupyter Lab and Jupyter Notebook with EC2
How to use jupyter notebook with ABCI
Rich cell output with Jupyter Notebook (IPython)
Use nim with Jupyter
Graph drawing with jupyter (ipython notebook) + matplotlib + vagrant
Try Apache Spark on Jupyter Notebook (on local Docker
Parallel computing with iPython notebook
Using Graphviz with Jupyter Notebook
How to use IPython Notebook
EC2 provisioning with Vagrant + Jupyter (IPython Notebook) on Docker
Run Apache-Spark with IPython Notebook
Graph drawing with IPython Notebook
How to use Jupyter Notebook
[Machine learning] Start Spark with iPython Notebook and try MLlib
I want to use a virtual environment with jupyter notebook!
Allow external connections with jupyter notebook
Formatting with autopep8 on Jupyter notebook
Visualize decision trees with jupyter notebook
Make a sound with Jupyter notebook
Add more kernels with Jupyter Notebook
Build IPython Notebook environment with boot2docker
Convenient analysis with Pandas + Jupyter notebook
Easy to use Jupyter notebook (Python3.5)
Spark play with WSL anaconda jupyter (2)
How to use jupyter notebook without polluting your environment with Docker
I want to blog with Jupyter Notebook
Try SVM with scikit-learn on Jupyter Notebook
[Explanation with image] Use pickle with Kaggle's NoteBook
Use BigQuery from your local Jupyter Notebook
Linking python and JavaScript with jupyter notebook
"LIVE" HTML presentation with IPython 3.0.0-dev, IPython Notebook
Jupyter Notebook Basics of how to use
Use vim keybindings on Docker-launched Jupyter Notebook
[Jupyter Notebook memo] Display kanji with matplotlib
How to use Jupyter notebook [Super Basic]
I wanted to use jupyter notebook with docker in pip environment (opticspy)
Specify the browser to use with Jupyter Notebook. Especially Mac. (And Vivaldi)
Use Jupyter Notebook with Visual Studio Code on Windows 10 + Python + Poetry + pyenv-win
Use jupyter notebook by connecting from another host
Jupyter Notebook memo
Introducing Jupyter Notebook
When Html cannot be output with Jupyter Notebook
Analytical environment construction with Docker (jupyter notebook + PostgreSQL)
Data analysis for improving POG 2 ~ Analysis with jupyter notebook ~
Launch and use IPython notebook on the network
Verify NLC accuracy with Watson Studio's Jupyter Notebook
Powerful Jupyter Notebook
Enable Jupyter Notebook with conda on remote server
Try using conda virtual environment with Jupyter Notebook
Use emacs or vim keybindings in IPython notebook
Fill the browser with the width of Jupyter Notebook
Jupyter notebook password
Jupyter Notebook memo
ipython notebook installation