EC2 provisioning with Vagrant + Jupyter (IPython Notebook) on Docker

Overview

(First half) EC2 provisioning using Vagrant (Second half) Start Jupyter (IPython Notebook) using Docker

I will write to Qiita for the first time. During the New Year holidays, I tried Deep learning from scratch and, hopefully, study TensorFlow, and study environment construction as well. , It is an introduction that also serves as a memorandum. Since the first and second half of this time are independent, I think it is possible to refer to only one of them.

:)

• EC2 makes it easy to raise or lower specifications
• Vagrant makes EC2 start / stop easy-> saves It minimizes mistakes that you forget to drop after work.
• Improved portability with Docker I also study here. The impression that it is convenient to set up a data analysis environment here and there.

:(

• EC2 costs money
• The meaning of biting Docker is weak If you decide on the environment from the beginning like studying books, I feel that you can build an environment with Ansible etc. simply and throughout.

Construction procedure

As a premise

• Install Vagrant
• Credential settings for IAM users with EC2 launch rights
• VPC and subnet settings to boot EC2

Is required.

I think this will be helpful.

• Install Vagrant
• INSTALLING VAGRANT
• Procedure to install Vagrant on Mac OS X
• Credential settings
• Deploying a Development Environment in Amazon EC2 Using the AWS Command Line Interface
• Settings to manage and handle access keys of multiple accounts with AWS CLI

(First half) EC2 provisioning using Vagrant

Refer to HashiCorp's GitHub page: Vagrant AWS Provider to configure the settings.

First, install the plug-in to use AWS.

$ vagrant plugin install vagrant-aws

It is a BOX equivalent to a machine image, but since a dummy box is open to the public for EC2, I will use this. (Officially, the box name is dummy, but it's a bit vague, so I changed it to ec2.)

$ vagrant box add ec2 https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

Next, move to the working directory and create a Vagrantfile.

$ mkdir jupyter_ec2
$ cd jupyter_ec2
$ vim Vagrantfile

Edit the Vagrantfile and set various parameters. This time, we have specified booting with a VPC created in advance.

There seem to be quite a few other parameters, and if you want to specify them, you can use the README or AWS API documentation. I think it is better to refer to).

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "ec2" # dummy box name

  config.vm.provider :aws do |aws, override|
    #AWS Credentials
    aws.aws_dir = ENV['HOME'] + "/.aws/"
    aws.aws_profile = "YOUR_PROFILE"
    aws.keypair_name = "YOUR_AWS_ACCESS_KEY"
    
    #SSH credentials
    override.ssh.username = "ec2-user"
    override.ssh.private_key_path = "SSH_KEY_PATH"

    #VPC settings
    aws.region = 'ap-northeast-1'
    aws.availability_zone = 'ap-northeast-1a'
    aws.subnet_id = 'subnet-XXXXXXXX'
    aws.security_groups = 'sg-XXXXXXXX'

    #EC2 settings
    aws.ami = "ami-9f0c67f8" # Amazon Linux 2016.09.01
    aws.instance_type = 'm4.large'
    aws.elastic_ip = true
    aws.associate_public_ip = true
    aws.tags = { 'Name' => 'NAME_TAGS_VALUE' }

    #EBS settings
    aws.block_device_mapping = [
      {
        'DeviceName' => "/dev/xvda",
        'Ebs.VolumeSize' => 50,
        'Ebs.DeleteOnTermination' => true,
        'Ebs.VolumeType' => 'standard',
      }
    ]
  end
end

After editing, start EC2.

$ vagrant up --provider=aws

When it doesn't work,

$ export VAGRANT_LOG=DEBUG

Then re-execute vagrant up and it will display a detailed error message.

If you want to undo the error message ʻexport VAGRANT_LOG = WARN `

Make sure that the security group SSH is also open properly. If the upper limit is not relaxed, the upper limit of the number of EIPs tends to be caught.

When it finishes starting

$ vagrant ssh

Check if you can SSH login with.

$ vagrant halt

You can stop the instance with, so when you are not working, stop it as appropriate to save money.

(Second half) Start Jupyter (IPython Notebook) using Docker

First, install Docker on EC2 by referring to Installing Docker.

$ sudo yum update -y
$ sudo yum install -y docker

Start the docker daemon so that it will start automatically even after the instance is restarted.

$ sudo service docker start
$ sudo chkconfig docker on

Add ec2-user to the docker group so that you can add docker commands without sudo privileges.

$ sudo usermod -a -G docker ec2-user

Once you exit and log in again, docker will start without sudo.

Next, describe the Dockerfile.

$ vim Dockerfile

This time, for the sake of simplicity, we will install TensorFlow based on the image of anaconda. In CMD, specify that jupyter should be started at docker run.

FROM continuumio/anaconda:latest

RUN mkdir /opt/notebooks
RUN pip install tensorflow
# Another installations comes here.

CMD ["/opt/conda/bin/jupyter", "notebook", "--notebook-dir=/opt/notebooks", "--ip='*'", "--port=8888", "--no-browser"]

After editing, create a container image from the Dockerfile.

$ docker build -t myjupyter .

Finally start the container. Specify volume and mount it so that the notebook can be made persistent and can be seen from the host side. Also, the --restart option causes the docker daemon to automatically start the container even after the instance is stopped / started.

$ mkdir notebooks
$ docker run -p 8888:8888 -v /home/ec2-user/notebooks:/opt/notebooks --restart=always myjupyter

After allowing TCP port 8888 communication in the security group, access [Elastic IP]: 8888 with a browser to confirm that Jupyter has started.

Only when studying with this, learn on the browser with vagrant up, stop charging with vagrant halt when quitting, build from the same Dockerfile when you want to try locally, easy porting if you get ipythonnb via Vagrant It became an environment ...! It should be, so I will correct it as appropriate while studying books.