[PYTHON] Deep Learning from the mathematical basics Part 2 (during attendance)

Purpose of this article

MPSYOKOHAMA 10th python-mnist, a library of handwritten digit character training data and teacher data used in the DNN implementation, runs in the windows environment. I didn't have it, so a memo when I forced it to move

Background

-Install python-mnist 0.3 with pip ・ From MINST DATABASE site, train-images-idx3-ubyte.gz: train-labels-idx1-ubyte.gz: download -Create a folder named mnist in the directory where python is running. -Unzip the above two files and save them. In other words Execution environment> mninst> train-images-idx3-ubyte> train-images.idx3-ubyte (file) Execution environment> mninst> train-labels-idx1-ubyte> train-labels.idx1-ubyte (file) I made the situation. So, when I try to read it, I first specify the mnist folder under the execution environment, but "" This is \ (half-width) for windows.

from mnist import MNIST
mndata = MNIST('.\mnist')

And then to read the file

train_img, train_label = mndata.load_training()

When I executed it, it was rejected due to a permission problem. .. .. I tried playing with the permissions of various files, but I couldn't solve it.

PermissionError: [Errno 13] Permission denied: '.\\mnist\\train-labels-idx1-ubyte'

However, if you directly insert import os and read it, you can access the file. Hmmm.

with open('.\mnist\\train-labels-idx1-ubyte\\train-labels.idx1-ubyte', 'rb') as f:
    line = f.readline()
    print(line)
>>>b'\x00\x00\x08\x01\x00\x00\xea....

It can't be helped, so let's load it directly while referring to the library.

C:\Python34\Lib\site-packages\mnist

With reference to loader.py here, I decided to write a code that can only do what I need this time by imitating it.

Implementation

I really wanted to use something like os.join to make it work in any storage location, but I gave up because of lack of ability. I put a file directly under mnist and read it. Execution environment> mninst> train-images.idx3-ubyte (file) Execution environment> mninst> train-labels.idx1-ubyte (file)

WMINST.py
# coding: utf-8

import os
import struct
from array import array

class MNIST(object):
    def __init__(self):
        self.train_img_fname = 'train-images-idx3-ubyte'
        self.train_lbl_fname = 'train-labels-idx1-ubyte'
        
        self.train_images = []
        self.train_labels = []
        
    def load_training(self): #Create a folder called mninst in the working folder and train in it-images.idx3-ubyte,train-labels.idx1-I put ubyte.
        ims, labels = self.load(('.\mnist\\train-images.idx3-ubyte'),
                                 ('.\mnist\\train-labels.idx1-ubyte'))
        
        self.train_images = ims
        self.train_labels = labels
        
        return ims, labels
        
    @classmethod
    def load(cls, path_img, path_lbl):
        with open(path_lbl, 'rb') as file:
                    magic, size = struct.unpack(">II", file.read(8))
                    if magic != 2049:
                        raise ValueError('Magic number mismatch, expected 2049,'
                                         'got {}'.format(magic))

                    labels = array("B", file.read())

        with open(path_img, 'rb') as file:
                    magic, size, rows, cols = struct.unpack(">IIII", file.read(16))
                    if magic != 2051:
                        raise ValueError('Magic number mismatch, expected 2051,'
                                         'got {}'.format(magic))

                    image_data = array("B", file.read())

        images = []
        for i in range(size):
            images.append([0] * rows * cols)

        for i in range(size):
            images[i][:] = image_data[i * rows * cols:(i + 1) * rows * cols]

        return images, labels

I was addicted to specifying the file path like `'. \ Mnist \\ train-labels.idx1-ubyte'```, and after the first mnist folder \\ `` Connect with two. (``` \ is \ (half-width))

I gave this file an appropriate name (WMINST.py), saved it in the working directory, and read it, and it worked. The reason why it becomes MNIST () and not (. \ Mnist) is because I wrote it so that the file path can be loaded directly.

from WMNIST import MNIST
mndata = MNIST()
train_img, train_label = mndata.load_training()

Apparently it worked.

Recommended Posts

Deep Learning from the mathematical basics Part 2 (during attendance)
Deep Learning from mathematical basics (during attendance)
[Part 4] Use Deep Learning to forecast the weather from weather images
[Part 1] Use Deep Learning to forecast the weather from weather images
[Part 2] Use Deep Learning to forecast the weather from weather images
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
Mathematical statistics from the basics Random variables
Deep Learning from scratch
[Deep Learning from scratch] I implemented the Affine layer
Othello ~ From the tic-tac-toe of "Implementation Deep Learning" (4) [End]
Deep Learning from scratch 1-3 chapters
Learning from the basics Artificial intelligence textbook Chapter 5 Chapter end problems
I tried to implement Perceptron Part 1 [Deep Learning from scratch]
"Deep Learning from scratch" Self-study memo (Part 8) I drew the graph in Chapter 6 with matplotlib
Deep Learning / Deep Learning from Zero Chapter 3 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 5 Memo
Lua version Deep Learning from scratch Part 6 [Neural network inference processing]
Try deep learning with TensorFlow Part 2
Deep learning from scratch (cost calculation)
Deep Learning / Deep Learning from Zero 2 Chapter 7 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 8 Memo
Deep Learning / Deep Learning from Zero Chapter 5 Memo
Deep Learning / Deep Learning from Zero Chapter 4 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 3 Memo
Deep Learning memos made from scratch
Deep Learning / Deep Learning from Zero 2 Chapter 6 Memo
(python) Deep Learning Library Chainer Basics Basics
Deep learning tutorial from environment construction
About the order of learning programming languages (from beginner to intermediate) Part 2
Voice processing by deep learning: Let's identify who the voice actor is from the voice
Reinforcement learning to learn from zero to deep
[Learning memo] Deep Learning made from scratch [Chapter 7]
Deep learning from scratch (forward propagation edition)
Othello-From the tic-tac-toe of "Implementation Deep Learning" (3)
Learning notes from the beginning of Python 1
Deep learning / Deep learning from scratch 2-Try moving GRU
Deep learning / Deep learning made from scratch Chapter 6 Memo
[Learning memo] Deep Learning made from scratch [Chapter 6]
Visualize the effects of deep learning / regularization
Image alignment: from SIFT to deep learning
Python: Gender Identification (Deep Learning Development) Part 1
Python: Gender Identification (Deep Learning Development) Part 2
"Deep Learning from scratch" in Haskell (unfinished)
Deep learning / Deep learning made from scratch Chapter 7 Memo
[Windows 10] "Deep Learning from scratch" environment construction
Learning record of reading "Deep Learning from scratch"
Learning notes from the beginning of Python 2
[Deep Learning from scratch] About hyperparameter optimization
Othello-From the tic-tac-toe of "Implementation Deep Learning" (2)
[Learning memo] Deep Learning made from scratch [~ Chapter 4]
Prepare the environment for O'Reilly's book "Deep Learning from scratch" with apt-get (Debian 8)
Deep Learning from scratch The theory and implementation of deep learning learned with Python Chapter 3
Lua version Deep Learning from scratch Part 5.5 [Making pkl files available in Lua Torch]
Deep Learning
"Deep Learning from scratch" self-study memo (unreadable glossary)
The story of doing deep learning with TPU
"Deep Learning from scratch" Self-study memo (9) MultiLayerNet class
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
Basics of Supervised Learning Part 1-Simple Regression- (Note)
Deep Learning from scratch Chapter 2 Perceptron (reading memo)
Video frame interpolation by deep learning Part1 [Python]