[PYTHON] GPy kernel function comparison

Purpose We often see cases of using GPy's Radial Basis Function (RBF) in optimization on the net, but there are several other kernel functions. Even though there was a mathematical explanation for these functions, I couldn't figure out how to use them properly, so I compared them by displaying them in a graph with the one-dimensional default settings. It was created with reference to the script of Some experiments in Gaussian Processes Regression. I just tried to write kernel functions in parallel.

environment

script

# Support for maths
import numpy as np
# Plotting tools
from matplotlib import pyplot as plt
# we use the following for plotting figures in jupyter
#%matplotlib inline  #When using Jupyter notebook

import warnings
warnings.filterwarnings('ignore')

# GPy: Gaussian processes library
import GPy
#from IPython.display import display

# Create a 1-D RBF kernel with default parameters
k01 = GPy.kern.RBF(1)
k02 = GPy.kern.Matern32(1)
k03 = GPy.kern.Matern52(1)
k04 = GPy.kern.ExpQuad(1)
k05 = GPy.kern.RatQuad(1)
k06 = GPy.kern.Exponential(1)
k07 = GPy.kern.Cosine(1)
k08 = GPy.kern.White(1)
k09 = GPy.kern.Linear(1)
k10 = GPy.kern.sde_Brownian(1)
k11 = GPy.kern.Integral(1)
k12 = GPy.kern.Poly(1)
k13 = GPy.kern.Bias(1)
k14 = GPy.kern.StdPeriodic(1)
k15 = GPy.kern.PeriodicExponential(1)

# Our sample space: 100 samples in the interval [-4,4]
X = np.linspace(-4.,4.,100)[:, None] # we need [:, None] to reshape X into a column vector for use in GPy

# First, sample kernel at x' = 0
K01 = k01.K(X, np.array([[0.]])) # k(x,0)
K02 = k02.K(X, np.array([[0.]])) # k(x,0)
K03 = k03.K(X, np.array([[0.]])) # k(x,0)
K04 = k04.K(X, np.array([[0.]])) # k(x,0)
K05 = k05.K(X, np.array([[0.]])) # k(x,0)
K06 = k06.K(X, np.array([[0.]])) # k(x,0)
K07 = k07.K(X, np.array([[0.]])) # k(x,0)
K08 = k08.K(X, np.array([[0.]])) # k(x,0)
K09 = k09.K(X, np.array([[0.]])) # k(x,0)
K10 = k10.K(X, np.array([[0.]])) # k(x,0)
K11 = k11.K(X, np.array([[0.]])) # k(x,0)
K12 = k12.K(X, np.array([[0.]])) # k(x,0)
K13 = k13.K(X, np.array([[0.]])) # k(x,0)
K14 = k14.K(X, np.array([[0.]])) # k(x,0)
K15 = k15.K(X, np.array([[0.]])) # k(x,0)

plt.axes([0.1, 0.1, 0.6, 0.8])

plt.plot(X, K01, label="RBF")
plt.plot(X, K02, label="Matern32")
plt.plot(X, K03, label="Matern52")
plt.plot(X, K04, label="ExpQuad")
plt.plot(X, K05, label="RatQuad")
plt.plot(X, K06, label="Exponential")
plt.plot(X, K07, label="Cosine")
plt.plot(X, K08, label="White")
plt.plot(X, K09, label="Linear")
plt.plot(X, K10, label="sde_Brownian")
plt.plot(X, K11, label="Integral")
plt.plot(X, K12, label="Poly")
plt.plot(X, K13, label="Bias")
plt.plot(X, K14, label="StdPeriodic")
plt.plot(X, K15, label="PeriodicExponential")

plt.title("$\kappa_{Covariance Function}(x,x')$");
plt.legend(bbox_to_anchor=(1.05,1), loc='upper left', borderaxespad=0, fontsize=8)
plt.show()

Explanation Lines 16-30: In Official Gpy Description, search for "covariance function" by keyword and find a method that seems to work. Defined in one dimension. Line 52: Set the graph (Axes) range to the lower left (0.1,0.1) and upper right (0.6,0.8). Line 71: With bbox_to_anchor = (1.05,1), place the legend in the upper right corner, and with borderaxespad = 0, place the legend outside the graph. bbox_to_anchor = (1,1) matches the upper right corner of the graph (axes).

result ** Figure 1** Of FIG. 1, the one having a peak near Y = 1 at X = 0 and having a peak close to the Gaussian distribution is displayed in FIG. ** Figure 2** Exponential is the sharpest function, but other than that, it is similar, and there is a difference in the degree of concentration near 0 with X = ± 2 as the boundary.

** Personal conclusion ** For the time being, it seems better to build an optimization framework with RBF, check the operation results of that framework several times, and change the kernel function as appropriate in some cases.