[PYTHON] Machine learning beginners tried RBM

at first...

This is a Qiita beginner's post. It's hard to see ... or anything like this, anyone can do it! Please forgive me ... To the last, it was done by Mr. Zero, who has no knowledge of machine learning and artificial intelligence. (At first, what was epoch? What was Accuracy? Ww)

What is RBM?

RBM(Restricted Boltzmann machine) It is a kind of pre-training method in deep learning, and it is a kind of model that makes a perfect match with AutoEncoder, which is often heard. The model was invented between 1984 and 1986, hoping for an edge in statistical mechanics. Unlike Autoencoder, which receives input and outputs deterministically, it is a generative model that can be discussed on a probability distribution, so it is known as a highly convenient model.

Deep Learning from RBM ~ With Black Magic ~

I'm not sure ...

I searched a lot!

Then ... There is something called BernoulliRBM in scikit learn! I have to do it

Implementation

For the time being, run it on a dataset called MNIST!

RBM.ipynb

import numpy as np
import matplotlib.pyplot as plt

from scipy.ndimage import convolve
from sklearn import linear_model, datasets, metrics
from sklearn.model_selection import train_test_split
from sklearn.neural_network import BernoulliRBM
from sklearn.pipeline import Pipeline
from sklearn.base import clone

#Data set used
from keras.datasets import fashion_mnist
from keras.datasets import mnist

from keras.layers import Input, Dense
from keras.models import Model
from keras import layers, models

import time
import numpy

For the time being, import looks like the above

Next, do as shown in the example (many parts do not know how to process)

RBM.ipynb

def nudge_dataset(X, Y):
    """
    This produces a dataset 5 times bigger than the original one,
    by moving the 8x8 images in X around by 1px to left, right, down, up
    """
    direction_vectors = [
        [[0, 1, 0],
         [0, 0, 0],
         [0, 0, 0]],

        [[0, 0, 0],
         [1, 0, 0],
         [0, 0, 0]],

        [[0, 0, 0],
         [0, 0, 1],
         [0, 0, 0]],

        [[0, 0, 0],
         [0, 0, 0],
         [0, 1, 0]]]

    def shift(x, w):
        return convolve(x.reshape((8, 8)), mode='constant', weights=w).ravel()

    X = np.concatenate([X] +
                       [np.apply_along_axis(shift, 1, X, vector)
                        for vector in direction_vectors])
    Y = np.concatenate([Y for _ in range(5)], axis=0)
    return X, Y


# Load Data
# (x_train, y_train), (x_test, y_test) = mnist.load_data()
# X, y = mnist.load_data()
# X = np.asarray(X, 'float32')
# X, Y = nudge_dataset(X, y)
# X = (X - np.min(X, 0)) / (np.max(X, 0) + 0.0001)  # 0-1 scaling

# X_train, X_test, Y_train, Y_test = train_test_split(
#     X, Y, test_size=0.2, random_state=0)

# (X_train, Y_train), (X_test, Y_test) = fashion_mnist.load_data()
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.astype('float32') / 255.
X_test = X_test.astype('float32') / 255.
X_train = X_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
X_test = X_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

# Models we will use
logistic = linear_model.LogisticRegression(solver='newton-cg', tol=1)
rbm = BernoulliRBM(random_state=0, verbose=True)

rbm_features_classifier = Pipeline(
    steps=[('rbm', rbm), ('logistic', logistic)])

# #############################################################################
# Training

# Hyper-parameters. These were set by cross-validation,
# using a GridSearchCV. Here we are not performing cross-validation to
# save time.
rbm.learning_rate = 0.06
rbm.n_iter = 10
# More components tend to give better prediction performance, but larger
# fitting time
rbm.n_components = 100
logistic.C = 6000

# Training RBM-Logistic Pipeline
rbm_features_classifier.fit(X_train, Y_train)

# Training the Logistic regression classifier directly on the pixel
raw_pixel_classifier = clone(logistic)
raw_pixel_classifier.C = 100.
raw_pixel_classifier.fit(X_train, Y_train)

# #############################################################################
# Evaluation

Y_pred = rbm_features_classifier.predict(X_test)
print("Logistic regression using RBM features:\n%s\n" % (
    metrics.classification_report(Y_test, Y_pred)))

Y_pred = raw_pixel_classifier.predict(X_test)
print("Logistic regression using raw pixel features:\n%s\n" % (
    metrics.classification_report(Y_test, Y_pred)))

Sorry for the dirty source code ... I have to comment out ... It's difficult to write beautiful code ...

I am using Bernoulli RBM from the sklearn library. In this example, we are building a classification pipeline using a Bernoulli RBM feature extractor and a Logistic Regression classifier. For comparison, we present a logistic regression for raw pixel values.

[RBM source code (sklearn)](https://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification -py)

result

MNIST...Accuracy: 0.97 Fashion-MNIST...Accuracy: 0.79

If you increase the number of epochs, it will take a long time, so set the number of epochs to 10 (no time to submit the assignment ...)

Conclusion

I tried to copy and implement the source code, but it took a long time to find it ... I have to understand more! I thought. After that, a graph? I thought it would be easier to understand if I could display the loss function and so on.

I will post it little by little from now on. I want to post docker and web related ... I look forward to working with you.