[PYTHON] Use the Kaggle API inside a Docker container

things to do

--Start container using Kaggle official image (kaggle / python) --Download the csv file with the kaggle ~ command --Download the public notebook with the kaggle ~ command

First summary

1. DL kaggle.json (kaggle.com> My Account> Create New API Token button)
2. $ docker run -it --rm --mount type=bind,src=pwd,dst=/root/dev kaggle/python
3. $ pip install kaggle
4. $ mkdir ~/.kaggle
5. $ cp /root/dev/kaggle.json ~/.kaggle
6. $ chmod 600 ~/.kaggle/kaggle.json
7. $ kaggle competitions download -c titanic -p input/titanic
8. $ unzip input/titanic/titanic.zip input/titanic
9. $ kaggle kernels pull arthurtok/introduction-to-ensembling-stacking-in-python -p ./

The dataset is downloaded from the Titanic competition If you have no problem, you can close it here: relaxed:

environment

macOS 10.14.6 (Mojave) Docker 19.03.4

Why use Docker?

--notebook unstable --I want to write code in VS Code --I don't want to pollute the global environment --Collision between pip and conda --Reliable because there is an official Kaggle image

What is the Kaggle API?

A guy who can do the operations on Kaggle's site from the command line

For example?

--Dataset download --Submit --List of available competitions --Download leaderboard

etc…

For more information, visit Official Repository

Command list

kaggle competitions {list, files, download, submit, submissions, leaderboard}
kaggle datasets {list, files, download, create, version, init}
kaggle kernels {list, init, push, pull, output, status}
kaggle config {view, set, unset}

I will make such a configuration

kaggle_titanic
├── input
│    └── titanic  <-Win if you can download the csv file here
└── working  <-Win if you can download the ipynb file here

Get Kaggle API Token

From My Account on the upper right

This is in the middle

The kaggle.json will be downloaded, so save it in the kaggle_titanic directory. The contents are like this

kaggle.json

{"username":"anata_no_namae","key":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}

By the way

You can invalidate the token here When you press something like "I accidentally pushed kaggle.json to GitHub!"

Container startup

Official image of Kaggle Use kaggle / python The following command in the kaggle_titanic directory

docker run -it --rm --mount type=bind,src=`pwd`,dst=/root/dev kaggle/python

The explanation of the option is written here, if you like -> [[Explanation with image] Create Anaconda environment with Docker and use VS Code in container](https://qiita.com/komiya_____/items/96c14485eb035701e218#%E3%82%B3%E3%83%B3 % E3% 83% 86% E3% 83% 8A% E8% B5% B7% E5% 8B% 95)

From here on the shell on the container side

Move to the mounted directory

cd /root/dev

OK if the contents are synchronized

ls

input  kaggle.json  working

Install kaggle package

pip install kaggle

Check the version to see if it works

kaggle -v

Traceback (most recent call last):
  File "/opt/conda/bin/kaggle", line 7, in <module>
    from kaggle.cli import main
  File "/opt/conda/lib/python3.6/site-packages/kaggle/__init__.py", line 23, in <module>
    api.authenticate()
  File "/opt/conda/lib/python3.6/site-packages/kaggle/api/kaggle_api_extended.py", line 149, in authenticate
    self.config_file, self.config_dir))
OSError: Could not find kaggle.json. Make sure it's located in /root/.kaggle. Or use the environment method.

I'm so angry You're saying something like kaggle.json can't be found

Place kaggle.json

Create a ~ / .kaggle / directory and copy kaggle.json into it ** * The ~ / .kaggle / directory may have been created at the timing of $ kaggle -v, but this is not a problem **

mkdir ~/.kaggle # <-You may be told that you already have a directory, but don't worry
cp /root/dev/kaggle.json ~/.kaggle

this time

kaggle -v

Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /root/.kaggle/kaggle.json'
Kaggle API 1.5.6

I can use it, but I'm still angry Other users can read the API Key, so change the permissions

Change permissions

chmod 600 ~/.kaggle/kaggle.json 

Following what is

kaggle -v
Kaggle API 1.5.6

Huh

It's annoying to do it every time

The series of processing here may be executed collectively in the cell There is also a strategy to create a file and reuse it

kaggle_settings.ipynb

!pip install kaggle
!mkdir ~/.kaggle
!cp /root/dev/kaggle.json ~/.kaggle
!chmod 600 ~/.kaggle/kaggle.json

Let's download the data

Specify the competition name with -c -p Specify save destination path

kaggle competitions download -c titanic -p input/titanic

ls input/titanic

titanic.zip

titanic.zip has been downloaded

Use the notation on the URL for the competition name

For example, in the Severstal: Steel Defect Detection competition, this is

Defrost

Unzip the downloaded titanic.zip

unzip input/titanic/titanic.zip -d input/titanic

ls input/titanic

gender_submission.csv  test.csv  titanic.zip  train.csv

There are 3 items, gender_submission.csv, test.csv, and train.csv. Now you can download the dataset

Next, try pulling the notebook

Notation of notebook is here

kaggle kernels pull arthurtok/introduction-to-ensembling-stacking-in-python -p ./working

ls ./working

introduction-to-ensembling-stacking-in-python.ipynb

Winner because it is pulled properly

It may be convenient to use VS Code together

It is good to be able to write code with intellisense or your own key binding

[Explanation with image] Create Anaconda environment with Docker and use VS Code in container

[Explanation with image] Convert VS Code to Jupyter

For those who want to use jupyter

Added -p 8888: 8888 option when starting container (associate host side and container side ports)

docker run -p 8888:8888 -it --rm --mount type=bind,src=`pwd`,dst=/root/dev kaggle/python

jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=''

Open localhost: 8888 from the address bar of your browser and it's OK

The end

Thank you for reading to the end