[PYTHON] Try Common Representation Learning with chainer

Overview

"[Joint multi-modal representations for e-commerce catalog search driven by visual attributes](https://kddfashion2016.mybluemix.net/kddfashion_finalSubmissions/Joint%20multi-modal%]" from IBM Research, India at the KDD2016 workshop. 20representations% 20for% 20e-commerce% 20catalog% 20search% 20driven% 20by% 20visual% 20attributes.pdf) ”is to be implemented by chainer.

The original dissertation method is here. The Implementation of this paper is written in Theano.

The content of the dissertation feels like a rough read, and when text and images are given as a pair, the common space is found using Neural Net (method name is "Correlational Neural Net, abbreviated as Corr Net") and is useful for search engines It is something like that.

I think that it is common to use CCA when seeking a common space of two different modals, but in practice scikit-learn CCA takes time to train as soon as the data size becomes large & it may not be usable due to Memory Error. There is, and this time I tried to implement it with chainer quickly.

The code written in jupyter notebook in python3 is ** here **.

It seems that the point is to train the correlation coefficient so that it becomes large in the common space.

Image diagram スクリーンショット 2017-03-27 11.29.47.png

The loss function consists of the loss of restoring both when two modals are given, the loss of restoring both when one modal is given, and the loss that makes the correlation in the hidden layer high. I have. スクリーンショット 2017-03-27 11.35.52.png

Data used

I wanted to try it easily, so I used MNIST to find a common space for 28x28 images and label information in one-hot-vector format.

I will try different data in the future.

result

Original image

スクリーンショット 2017-03-24 17.07.01.png

Image after restoration ("Image" → "Image")

スクリーンショット 2017-03-24 17.06.36.png

It seems that it can be restored successfully

Image after restoration using label ("Label" → "Image")

スクリーンショット 2017-03-24 17.06.52.png

Image after restoration of (0 and 8) by combining label information ("label" → "image")

スクリーンショット 2017-03-27 11.52.26.png

An image that looks like the middle between 0 and 8 is generated properly.

Transition of loss

Total

Detail

Learning is progressing so that the correlation is high in the hidden layer properly!

Consideration

It seems that you can restore just from the label information!

In any case, chainer is much easier to write than Theano and can output logs, which is convenient ^^

Recommended Posts

Try Common Representation Learning with chainer

Try deep learning with TensorFlow

Try implementing RBM with chainer.

Try Deep Learning with FPGA

Try machine learning with Kaggle

Now, let's try face recognition with Chainer (learning phase)

Try Deep Learning with FPGA-Select Cucumbers

Reinforcement learning 13 Try Mountain_car with ChainerRL.

Try deep learning with TensorFlow Part 2

Try horse racing prediction with Chainer

[Chainer] Learning XOR with multi-layer perceptron

Try machine learning with scikit-learn SVM

Reinforcement learning 8 Try using Chainer UI

Classify anime faces with deep learning with Chainer

Try Bitcoin Price Forecasting with Deep Learning

Try deep learning of genomics with Kipoi

Reinforcement learning 11 Try OpenAI acrobot with ChainerRL.

Seq2Seq (1) with chainer

Try scraping with Python.

Learning Python with ChemTHEATER 03

"Object-oriented" learning with python

Learning Python with ChemTHEATER 05-1

Use tensorboard with Chainer

Try to predict forex (FX) with non-deep machine learning

Now, let's try face recognition with Chainer (prediction phase)

Learning Python with ChemTHEATER 02

Learning Python with ChemTHEATER 01

Try SNN with BindsNET

Use scikit-learn training dataset with chainer (for learning / prediction)

Try regression with TensorFlow

Try to build a deep learning / neural network with scratch

[Evangelion] Try to automatically generate Asuka-like lines with Deep Learning

I tried to implement ListNet of rank learning with Chainer

Try to predict if tweets will burn with machine learning

Let's move word2vec with Chainer and see the learning progress

Machine learning learned with Pokemon

Try to factorial with recursion

Try function optimization with Optuna

Try using PythonTex with Texpad.

Play with reinforcement learning with MuZero

Try edge detection with OpenCV

Ensemble learning summary! !! (With implementation)

Try Google Mock with C

Try using matplotlib with PyCharm

Try programming with a shell!

Try GUI programming with Hy

Try Python output with Haxe 3.2

Try matrix operation with NumPy

Reinforcement learning 6 First Chainer RL

Reinforcement learning starting with Python

Try implementing XOR with PyTorch

Learn elliptical orbits with Chainer

Try running CNN with ChainerRL

About learning with google colab

Try various things with PhantomJS

Deep Kernel Learning with Pyro

Reinforcement learning 5 Try programming CartPole?

Use chainer with Jetson TK1

Linux fastest learning with AWS

Machine learning Minesweeper with PyTorch

Try running Python with Try Jupyter