[PYTHON] Parallel learning of deep learning by Keras and Kubernetes

Parallel learning of deep learning by Keras and Kubernetes

I wrote earlier that Keras supports CNTK.

Run Keras with CNTK backend from CentOS

Keras can now use different frameworks in the same program with Theano, TensorFlow and CNTK as backends. I think the advantage of increasing the Keras backend is that you can change the framework and learn by simply changing the environment variables. If you change the framework, the neural network model itself will (probably) not change if the program is the same, but the accuracy and speed will change slightly. Alternatively, you can ensemble and infer the models learned in each framework. I came up with an ensemble of multiple frameworks, so I actually tried it.

The program and Kubernetes yml can be found below. https://github.com/shibuiwilliam/keras_multibackend

Learn multiple Keras backends in parallel

It's boring to try multiple deep learning frameworks sequentially from Keras with the same program, so I tried to train each framework in parallel using Kubernetes. The image is as follows.

It's boring just to learn sequentially.

Let's learn in parallel.

Overall picture

Let Kubernetes learn deep learning in parallel on a Docker container. The overall picture is as follows.

Both platform and program configurations are required.

platform

Prepare 3 virtual machines. This time it is CentOS 7.3. Make VM1 an NFS server and mount the shared directories on VM2 and VM3. Place the Keras learning and inference programs in the shared directory. VM1,2,3 will all be able to access Keras learning and inference programs.

Form a Kubernetes cluster with VM1,2,3. Build VM1,2,3 with Keras, Theano, TensorFlow, CNTK, and other installed Docker images.

The Dockerfile used this time is below.

https://github.com/shibuiwilliam/keras_multibackend

Kubernetes performs Cifar 10 CNN learning by each framework as a job. We will prepare 3 types of Job yml. Define the values of each environment variable KERAS_BACKEND as Theano, TensorFlow, and CNTK.

The Job definition yml of Kubernetes is as follows. First of all, it is for learning. Although CNTK is illustrated, Tensorflow and Theano only change the value of Keras_Backend.

apiVersion: batch/v1
kind: Job
metadata:
  name: kerascntk
spec:
  template:
    spec:
      restartPolicy: Never
      volumes:
        - name: nfstest
          nfs:
            server: 10.249.20.65
            path: "/tmp/nfstest/"
      containers:
        - name: kerascntk
          image: jupyter:1.52
          volumeMounts:
              - name: nfstest
                mountPath: "/tmp/nfstest/"
          env:
              - name: KERAS_BACKEND
                value: "cntk"
          command: ["python"]
          args: ["/tmp/nfstest/cifar_train.py"]

For inference. Inference is summed with 3 backend inferences in Python, so you only need one below.

apiVersion: batch/v1
kind: Job
metadata:
  name: pred
spec:
  template:
    spec:
      restartPolicy: Never
      volumes:
        - name: nfstest
          nfs:
            server: 10.249.20.65
            path: "/tmp/nfstest/"
      containers:
        - name: pred
          image: jupyter:1.52
          volumeMounts:
              - name: nfstest
                mountPath: "/tmp/nfstest/"
          command: ["python"]
          args: ["/tmp/nfstest/cifar_pred.py"]

program

Prepare a Keras learning program and an inference program. It is not necessary to create each framework separately. Since the environment variable KERAS_BACKEND is defined and divided in the Kubernetes Job yml, there is one program for learning and one for inference.

Each program is as follows. First of all, it is for learning. I haven't made any special efforts. It's an ordinary VGG-like CNN. The number of epochs is 50, but since Early Stopping is also set, it is supposed to end in about 20 epochs. Also, 100 images of test data are excluded from Evaluation for use in inference.

#For learning

# see its environment variable
import os
kerasBKED = os.environ["KERAS_BACKEND"]
print(kerasBKED)

# imports
import keras
from keras.models import load_model
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import EarlyStopping, ModelCheckpoint
import pickle
import numpy as np

# variables
batch_size = 32
num_classes = 10
epochs = 50

# loading and reshaping cifar10 data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

# leave out 100 images for prediction
x_test1 = x_test[100:,:]
x_test2 = x_test[:100,:]
y_test1 = y_test[100:,:]
y_test2 = y_test[:100,:]
print(x_test1.shape, x_test2.shape, y_test1.shape, y_test2.shape)
# only test1 data is used for validation

# VGG-like neural network model
model = Sequential()
model.add(Conv2D(32, (3, 3), padding='same',
                 input_shape=x_train.shape[1:]))
model.add(Activation('relu'))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Conv2D(64, (3, 3), padding='same'))
model.add(Activation('relu'))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes))
model.add(Activation('softmax'))

opt = keras.optimizers.adam()

model.compile(loss='categorical_crossentropy',
              optimizer=opt,
              metrics=['accuracy'])


# Add checkpoint and earlystopping
chkpt = '/tmp/nfstest/cifar_weights/' + kerasBKED + '_weights.{epoch:02d}-{loss:.2f}-{val_loss:.2f}.hdf5'
cp_cb = ModelCheckpoint(filepath = chkpt, monitor='val_loss', verbose=1, save_best_only=True, mode='auto')
es_cb = EarlyStopping(monitor='val_loss', patience=1, verbose=1, mode='auto')

# fit the model
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test1, y_test1),
                    callbacks=[cp_cb, es_cb],
                    shuffle=True)

# score and save the model
score = model.evaluate(x_test1, y_test1, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

saveDir = "/tmp/nfstest/cifar_model/"
if not os.path.isdir(saveDir):
    os.makedirs(saveDir)
modelName = "{0}_{1}_model.hdf5".format(kerasBKED, score[1])
model_path = os.path.join(saveDir, modelName)
model.save(model_path)

At the end of training, save the model to an NFS directory. As a result, three types of models are stored in the NFS directory. Inference uses these three models. For inference, use 100 images extracted from the test data. Infer by calling the three types of model files (Theano, Tensorflow, CNTK) created by learning. Since there is a difference in the accuracy of learning in each model, the inference is multiplied by the accuracy and finally divided by the sum of the accuracy. It looks like an ensemble model, but it doesn't have any theoretical backing.

#For reasoning

# import
import keras
from keras.models import load_model
from keras.datasets import cifar10
import os

# variables
batch_size = 32
num_classes = 10

# load and reshape data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

x_test1 = x_test[100:,:]
x_test2 = x_test[:100,:]
y_test1 = y_test[100:,:]
y_test2 = y_test[:100,:]
# test2 data is used for prediction

# get a list of model files
backList = ["cntk", "tensorflow", "theano"]
saveDir = "/tmp/nfstest/cifar_model/"
files = os.listdir(saveDir)
filesList = [f for f in files if os.path.isfile(os.path.join(saveDir, f))]
filesList

# predict using each model
prediction = 0
sumAcc = 0
for bk in backList:
    os.environ["KERAS_BACKEND"] = bk
    kerasBKED = os.environ["KERAS_BACKEND"]
    modelName = [m for m in filesList if bk in m][0]
    modelAcc = float(modelName.split("_")[1])
    print("MODELNAME {0} \t for BACKEND {1} \t with ACCURACY {2}".format(modelName, kerasBKED, modelAcc))
    model = load_model(os.path.join(saveDir, modelName))
    prediction += model.predict(x_test2, batch_size=batch_size, verbose=0) * modelAcc
    sumAcc += modelAcc
prediction = prediction / sumAcc
print(prediction)

# output predictions to csv
import csv
with open("/tmp/nfstest/cifar_model/preds.csv", 'w') as f:
    writer = csv.writer(f, lineterminator='\n')
    writer.writerows(prediction)


predlist = []
for i in prediction:
    pred = max(enumerate(i), key=lambda x: x[1])[0]
    predlist.append([pred, i[pred]])
print(predlist)


# output overall summary to csv
import csv
with open("/tmp/nfstest/cifar_model/pred.csv", 'w') as f:
    writer = csv.writer(f, lineterminator='\n')
    writer.writerow(predlist)

The result of trying

In learning, all three frameworks settled on almost the same Accuracy.

It is natural because we are learning with the same neural network.

However, when it comes to inference, there is a slight difference in the back end. Below is a list of 10 test data inferred by each backend. (In the above program, we infer from 100 image data, but since it is too much, we will limit it to 10.) For the first image data, CNTK infers 4 to be 49.498%, while Tensorflow and Theano infer 3 to 55.225% and 67.820%, respectively. Multiplying each backend's inference and Accuracy and dividing it by the Accuracy's total results in inferring 3 to 57.855%. The correct answer for each image data is listed as answer to the left of ensemble.

\frac{0.49498 * 0.771 + 0.55225 * 0.772 + 0.67820 * 0.785}{0.771 + 0.772 + 0.785} \\
= 0.57855

The second image data is 8 for CNTK, Tensorflow, and Theano. However, in the third data, CNTK is 0 at 41.421%, Tensorflow is 8 at 72.917%, and Theano is 1 at 52.288%.

What is interesting to look at is that even if the individual backends are incorrect, the ensemble will give the correct answer. For example, in image 3, CNTK, Tensorflow, and Theano make separate inferences, but since Tensorflow infers 8 with the highest probability, the ensemble is also 8.

Summary

In the learning phase, both backends are just as accurate. However, if you try to infer, each may give different results. Further ensemble seems to give the correct answer with higher accuracy than inferring with one backend.

I succeeded in parallel learning with Kubernetes. It's very convenient because Kubernetes Job will execute the specified program in the cluster. Kubernetes also has a Cron Job, and it seems that Job can be executed in a timed manner. When planning a deep learning study, I find it useful to automate it with Kubernetes' Cron Job. So, next time, I will verify the automatic execution of deep learning learning by Kubernetes Cron Job.