[PYTHON] [Colab] How to copy a huge dataset
Background
PyTorch's DCGAN Tutorial requires a huge dataset (1GB, about 220,000 images) ↓ Learning is slow & memory shortage when running on local Jupyter Lab ↓ Let's learn on ** Google Colaboratory ** (hereinafter Colab) that can also use GPU ↓ Problems occur while moving hands
** How to copy a dataset to Colab? ** **
problem
――Can Colab refer to Google Drive files? ――Can ZIP be decompressed on Colab?
Can Colab refer to Google Drive files?
You can refer to it by mounting Google Drive. Create a new notebook and run the following code.
from google.colab import drive
drive.mount('/content/drive')
A link to generate an authentication code will be attached, so access it. Select the account that uses Colab in the account selection. Google Drive File Stream will ask for access, so allow it. An authentication code will be issued, so copy it, paste it, and enter.
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=xxx
Enter your authorization code:
Can ZIP be decompressed on Colab?
You can use the unzip command on Colab. First, upload the ZIP file of the dataset to Google Drive and copy it to Colab.
cp "./drive/My Drive/Colab Notebooks/data/celeba/img_align_celeba.zip" "."
After that, use the unzip command to unzip it on Colab.
!unzip "img_align_celeba.zip"
Try to display the unzipped image
from PIL import Image
Image.open('img_align_celeba/000001.jpg')
I was able to display it safely.
Summary
- ZIP upload your dataset to Google Drive
- Mount Google Drive on Colab
- Copy the ZIP file onto Colab
- Unzip the ZIP file on Colab
Postscript
If you just want to upload a file to Colab, you can select a local file with the following code. However, I feel that it takes a long time to upload a large file. I feel that I can upload faster via Google Drive in this article.
from google.colab import files
files.upload()
Recommended Posts