[PYTHON] Machine Learning with Spark (86) with Spark (1) "Machine Learning with Spark" By Rajdeep Dua, Manpreet Singh Ghotra, Nick Pentreath
1. For those who want to use it immediately (as soon as)
「Machine Learning with Spark」 By Rajdeep Dua, Manpreet Singh Ghotra, Nick Pentreath
http://shop.oreilly.com/product/9781785889936.do
docker Please install docker and start docker on Windows and Mac. On Windows, docker may not start unless Intel Virtualization is enabled in the Bios. In addition, security warnings may appear.
docker run
$ docker pull kaizenjapan/anaconda2-spark
$ docker run -it -p 8888:8888 kaizenjapan/anaconda2-sparkk /bin/bash
In the shell session below (base) root @ f19e2f06eabb: / # is the comman prompt. Actually, the number part may be different. Enter the right side of # on this line. The other lines are output. If there are any errors or differences in the output, please let us know in the comments. Navigate to the folder for each chapter.
If the display in docker and the shell of the OS that started docker are similar, you may make a mistake as to which one you are investigating. Watch out for docker's comman prompt.
File sharing or copying
In the OS that started docker and docker, please share the file or copy the file and display the generated file on the browser etc. The URL of the method is described in the reference column.
I'm looking for a good way to organize the disks of the OS that started docker. Some methods have shared settings from the beginning.
In the case of copying, the OS side command that started docker was executed. Replace it with your docker number. I displayed the copied file on the browser and checked the contents.
3 parts
python python2 program
java
2. For those who build docker by themselves
From here on down, I will record what kind of policy and procedure I made the docker that was pulled above. It is a reference material for using the above docker. Not needed to run the rest of the book. This is the procedure for building docker / anaconda on your own. It is not a way to create a docker file. sorry. docker
A mechanism that allows Linux such as ubuntu and debian to be used in common from linux, windows and mac os. It is good to be able to use it without changing the settings of the OS to be used. With the same specifications, it can be used by a large number of people. Both software officially supported by the software developer and those conveniently tailored by the user are available. This time, we will tailor what we have officially distributed so that it can be used by others. python
I went to Phthon for deep learning training. The reason for using python is that many machine learning mechanisms can be used in python, and statistical analysis mechanisms such as R can also be easily used from python. anaconda
There are differences between the 2 and 3 versions of python and the distribution method. I've been using python3 with Anaconda for the last year and a half. The reason I used Anaconda is that it comes with a library of statistical analysis and Jupyter Notebook from the beginning.
official docker distribution
There are official distribution of OS such as ubuntu and debian, and official distribution of languages such as gcc and anaconda. By using these and registering on docker-hub, you can check the quality of official distribution and share a wide range of information including change rights. It does not mean that docker officially distributes it, but that it is officially distributed by each software provider. docker pull
The use of docker official distribution is realized by pulling from the URL. docker Anaconda
Use the one officially distributed by anaconda.
$ docker pull kaizenjapan/anaconda-keras
Using default tag: latest
latest: Pulling from continuumio/anaconda3
Digest: sha256:e07b9ca98ac1eeb1179dbf0e0bbcebd87701f8654878d6d8ce164d71746964d1
Status: Image is up to date for continuumio/anaconda3:latest
$ docker run -it -p 8888:8888 continuumio/anaconda3 /bin/bash
Pull other pushes that actually used keras and tensorflow
apt-get
(base) root@d8857ae56e69:/# apt-get update
(base) root@d8857ae56e69:/# apt-get install -y procps
(base) root@d8857ae56e69:/# apt-get install -y vim
(base) root@d8857ae56e69:/# apt-get install -y apt-utils
(base) root@d8857ae56e69:/# apt-get install sudo
(base) root@d8857ae56e69:/# apt-get upgrade
Source git
(base) root@f19e2f06eabb:/# git clone https://github.com/PacktPublishing/Machine-Learning-with-Spark
pip
pip install pyspark
Register with docker hub
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
caef766a99ff continuumio/anaconda3 "/usr/bin/tini -- /b…" 10 hours ago Up 10 hours 0.0.0.0:8888->8888/tcp sleepy_bassi
$ docker commit caef766a99ff kaizenjapan/anaconda2-spark
$ docker push kaizenjapan/anaconda2-spark
Reference
spark
Machine learning
Why machine learning with docker Book / source list is being created (Goal 100) https://qiita.com/kaizen_nagoya/items/ddd12477544bf5ba85e2
Machine learning with docker (1) with anaconda (1) "Deep Learning from scratch-The theory and implementation of deep learning learned with Python" by Yasuki Saito https://qiita.com/kaizen_nagoya/items/a7e94ef6dca128d035ab
Machine learning with docker (2) with anaconda (2) "Deep Learning from scratch 2 Natural language processing" by Yasuki Saito https://qiita.com/kaizen_nagoya/items/3b80dfc76933cea522c6
Machine learning with docker (3) with anaconda (3) "Intuition Deep Learning" Antonio Gulli, Sujit Pal Chapter 1, Chapter 2 https://qiita.com/kaizen_nagoya/items/483ae708c71c88419c32
Machine learning with docker (71) Environment construction (1) docker Somehow, no matter what, there are only errors. https://qiita.com/kaizen_nagoya/items/690d806a4760d9b9e040
Machine learning with docker (72) Environment construction (2) Docker for Windows https://qiita.com/kaizen_nagoya/items/c4daa5cf52e9f0c2c002
Machine learning with docker (73) Environment construction (3) docker / linux / macos bash script, ms-dos batch file https://qiita.com/kaizen_nagoya/items/3f7b39110b7f303a5558
Machine learning with docker (74) Environment construction (4) R How many difficulties? https://qiita.com/kaizen_nagoya/items/5fb44773bc38574bcf1c
Machine learning with docker (75) Environment construction (5) Management of docker related files https://qiita.com/kaizen_nagoya/items/4f03df9a42c923087b5d
I tried to run OpenCV with Python and was told that libGL.so was missing, but I solved it. https://qiita.com/toshitanian/items/5da24c0c0bd473d514c8
Drawing tips with matplotlib on the server side https://qiita.com/TomokIshii/items/3a26ee4453f535a69e9e
Copy files between host and container with Docker https://qiita.com/gologo13/items/7e4e404af80377b48fd5
Use file sharing with Docker for Mac https://qiita.com/seijimomoto/items/1992d68de8baa7e29bb5
"Nagoya's IoT is Nagoya's OS" How can I use Docker? TOPPERS / FMP on RaspberryPi with Macintosh 5 barriers https://qiita.com/kaizen_nagoya/items/9c46c6da8ceb64d2d7af
Road to 64bit CPU and / or 64 year old determination https://qiita.com/kaizen_nagoya/items/cfb5ffa24ded23ab3f60
Deep Learning 2 Natural Language Processing from Zero How to proceed with a book club (example) https://qiita.com/kaizen_nagoya/items/025eb3f701b36209302e
Try using NVIDIA Docker on Ubuntu 16.04 LTS https://blog.amedama.jp/entry/2017/04/03/235901
Document history
ver. 0.10 First draft 20181024 ver. 0.11 push 20181028
Recommended Posts