[PYTHON] [Introduction to Azure for kaggle users] Comparison of how to start and use Azure Notebooks and Azure Notebooks VM

the aim

For those who are feeling the limits of their computational resources, but don't know how to use cloud resources, here is a summary of how to use Azure Notebooks and Azure Notebooks VM. For details, I will introduce the official Azure tutorial, but I would like to follow about what to do and how much it can be used as a computational resource.

Target person

• Those who participate in competitions such as kaggle and want to use the environment such as GPU usage and memory increase in the cloud
• Those who feel that the calculation environment at hand is insufficient for business etc.

Prerequisites

• Email accounts such as hotmail.com and ʻoutlook.com` (for private use)
• Azure subscription (free version is also available)
• Create Azure ML workspace (you can create it according to the linked manual)

How to Use

1. Azure Notebooks Free compute
2. Azure Notebooks VM
3. Combined use of the above two

1. Use Azure Notebooks Free compute

specification

• Usage amount: Free
• Machine specs: CPU 1 core, 4GB RAM, 7GB Storage
• Language: R, Python

Pros

• You can use Jupyternote environment with Azure related packages (tensorflow, Keras, Pytorch, of course) installed for free.
• Starts up quickly and can be started immediately

Disadvantages

• Environment with lower specifications than the average PC these days
• Occasionally, if you freeze or hit Run repeatedly, you will be banned from the system for several hours due to security, so it seems better to download files frequently or edit the code locally.
• R users also work in the Jupyter environment and are not available in the familiar R-Studio

How to get started

• Sign in at Official Link
• When you log in for the first time, you will be asked if you want to start the project by copying the .ipynb file of the tutorial about AzureML from Github (the tutorial is Python Ver only).
• There is an unfamiliar unit called Project, but it's just a directory, so there's no problem at first.
• • Button → Notebook → Select [R] or [Python] to start

2. Use Azure Notebooks VM

specification

• Usage amount: https://azure.microsoft.com/ja-jp/pricing/details/virtual-machines/windows/
• Machine specs: Freely select from the link above
• Language: R, Python

Pros

• You can use Jupyternote environment with Azure related packages (of course tensorflow, Keras, Pytorch) installed.
• You can freely choose specifications such as GPU usage and large capacity RAM
• R users can also choose the R-Studio environment

Disadvantages

• Occasionally, if you freeze or hit Run repeatedly, you will be banned from the system for several hours due to security, so it seems better to download files frequently or edit the code locally.

How to get started

• Log in to Azure Portal and go to the workspace screen created with the prerequisites → Select [Calculate] on the left toolbar → [Notebook VMs ] Create by selecting the name and specifications from New

• It takes about 10 min to create
• Status: When Running, select from JupyterLab, Jupyter, R-Studio from the adjacent Application URL and execute.

3. Combined use of Free compute and VM

specification

• Usage amount: https://azure.microsoft.com/ja-jp/pricing/details/virtual-machines/windows/
• Machine specs: Freely select from the link above
• Language: R, Python

Pros

• Data preprocessing is Free compute, and deep learning learning etc. uses GPU environment, etc. You can cut costs by using VMs that require only the parts that require specifications.
• Azure AutoML is available, so you can automatically select the best model for modeling various methods and their ensembles (average, stacking).

Disadvantages

• Occasionally, if you freeze or hit Run repeatedly, you will be banned from the system for several hours due to security, so it seems better to download files frequently or edit the code locally.
• There is a dedicated coding, and it is troublesome to remember
• Preprocessing is often not very cost effective in terms of time, so I feel that it is enough to do it with a VM from beginning to end.

How to get started

• Python : https://docs.microsoft.com/ja-jp/azure/machine-learning/service/tutorial-1st-experiment-sdk-setup
• R : https://docs.microsoft.com/ja-jp/azure/machine-learning/service/tutorial-1st-r-experiment