[PYTHON] Keras usage example: Input (Embedding + Flatten)

1. Install Keras

Since the installation method of Tensorflow and Theano is introduced on various sites, detailed explanation is omitted here. Installation of Keras

pip install Keras

Then just play around with the config file and select Theano or Tensorflow. If you are using Tensorflow on the backend, it will automatically use the GPU.

When using Tensorflow with Keras, as of January 2017, the following error may occur.

AttributeError: 'module' object has no attribute 'control_flow_ops'

It seems that such an error occurs because python.control_flow_ops does not exist in tensorflow. StackOverflow, git discussion / fchollet / keras / issues / 3857). If you define it at the beginning of the script as follows, it will be cured.

import tensorflow as tf
tf.python.control_flow_ops = tf

2. Input(Embedding + Flatten) + Layer + Dropout + Output

This is the sample code introduced in Kaggle Summary: Redhat. The original problem is a binary classification problem of whether or not an unknown customer purchases a service from about 100 variables based on customer information. Among them, qianqian posted an analysis method using Keras. ** Normally, variables are engineered and preprocessed and then put into the input layer as they are, but qianqian passes each variable independently through the Embedding layer and Flatten layer and then merges them. ** **

Code of author (qianqian) or [Code here](https: // Please use gist.github.com/TomHortons/dd672a4323f42aed59316c9f56f72574).

Here is a diagram that visualizes the created model with visualize_util. (See "Other" in this article)

Here, we will focus on the explanation of Embedding + Flatten processing from the reference source code.

First, let's look at the input creation part. Input is created with the following code.

flatten_layers = []
inputs = []

for c in columns:

    inputs_c = Input(shape=(1,), dtype='int32')

    num_c = len(np.unique(data[c].values))

    embed_c = Embedding(
                    num_c,
                    dim,
                    dropout=0.2,
                    input_length=1
                    )(inputs_c)
    flatten_c= Flatten()(embed_c)

    inputs.append(inputs_c)
    flatten_layers.append(flatten_c)

flatten = merge(flatten_layers,mode='concat')

columns is a list that stores column names. As you can see from the first figure, each variable is not entered as it is. Input-> Embedding-> flatten is executed for each variable, and all 1600 variables are merged to make it an input layer. In other words, one-dimensional numerical data is expanded to 32 dimensions and input.

fc1 = Dense(hidden,activation='relu')(flatten)
dp1 = Dropout(0.5)(fc1)

outputs = Dense(1,activation='sigmoid')(dp1)

After that, one intermediate layer (fc1) and one dropbou (dp1) are set as output.

As you can see in this article, Deep Language Modeling for Question Answering using Keras Embedding seems to be used as Word2Vec in language learning. It was a mystery why Embedding was used for numerical data that was not language learning, so I checked Kaggle's forum.

It looks like this when I explain it roughly.

The Embedding layer transforms each category into a different feature space. Finding a representation that encodes the best features is not easy, and can all of this be trained using this technique? .. The best neural consists only of the embedding layer with a score of 0.9877. By adding more features to it, it will improve to 0.989. It is difficult to determine each parameter (number of hidden layers, dimension of embedding, method of merging additional features, batch size, number of train epochs) in the first place.

The Embedding layer has the following functions. For category data [1,2,3]

1. Perform encoding, 1: [1,0,0], 2: [0,1,0], 3: [0,0,1]
2. Multiply the weight matrix W whose shape is (3, k).
3. Finally, a one-hot (specific value is 1 and then zero) matrix with shape (4, k) is completed.

Where k is the dimension of the potential feature set by the user. In other words, the embedding layer is performing a linear transformation of the category data.

Furthermore, there is the following explanation about the flat ten layer.

Since there is only one type of input sequence (the input is one-dimensional), the output of the embedding layer is N * 1 * 32. Therefore, it is necessary to transform the shape of output to N * 32. Where N is the number of input data (maybe). If M is the number of category features, it will have M * 32 dimensional input. For the above reasons, the flattened output is concatenated. The input of Keras model is the input layer expressed in a list.

Very kindly, he rewrote the relationship between the embedding layer and the flatten layer so that you can see the output of the middle layer.

inputs = [model.layers[0].input,K.learning_phase()]
outputs = [model.layers[3].output]
func = K.function(inputs, outputs)

sample_size = data.shape[0]
data_ae = []
batches = make_batches(sample_size, batch_size)
for batch_index, (batch_start, batch_end) in enumerate(batches):
    X_0 = data[batch_start:batch_end].toarray()
    yy = func([X_0,0])[0]
    print yy.shape
    data_ae.append(yy)


data_ae = np.vstack(data_ae)

Other

Visualize each layer of the model created by Keras.

Install pydot-ng and

pip install pydot-ng

Execute as follows.

from keras.utils.visualize_util import plot
plot(model, to_file="model.png ", show_shapes=True)

Check the output of the middle layer

When designing a Keras model, there are times when you want to check the behavior of the middle layer. In Document, the output of the intermediate layer is visualized by describing the Keras function.

from keras import backend as K

# with a Sequential model
get_3rd_layer_output = K.function([model.layers[0].input],
                                  [model.layers[3].output])
layer_output = get_3rd_layer_output([X])[0]

If there is a dropout layer, it will work if you set the learning phase flag.

get_3rd_layer_output = K.function([model.layers[0].input, K.learning_phase()],
                                  [model.layers[3].output])

# output in test mode = 0
layer_output = get_3rd_layer_output([X, 0])[0]

# output in train mode = 1
layer_output = get_3rd_layer_output([X, 1])[0]