2

I have a set of data (X,y) where X is my input and it is 2-dimensional, y is my output and it is 1-dimensional, and each pair (X,y) has a corresponding non-uniform noise term. Here is a working example where I apply Gaussian process regression:

import numpy as np 
from sklearn.gaussian_process import GaussianProcessRegressor 
from sklearn.gaussian_process.kernels import RBF, WhiteKernel, ConstantKernel as C  

lenX = 20
X1min = 0.
X1max = 1.
X2min = 0.
X2max = 2.
X1 = np.linspace(X1min,X1max,lenX)
X2 = np.linspace(X2min,X2max,lenX)
time_spacing = X2[1] - X2[0]

X = []
i = 0
while i < lenX:
    j = 0
    while j < lenX:
        X.append([X1[i],X2[j]])
        j = j + 1
    i = i + 1

X = np.array(X)

def fun_y(X):
    y = 5.*((np.sin(X[:,0]))**2.)*(np.e**(-(X[:,1]**2.)))
    y[y < 0.001] = 0.0
    return y

y = fun_y(X)
noise = 0.1*y #0.2/y + 0.2#*np.linspace(1.0,0.1,len(X))

len_x1 = 10
len_x2 = 100
x1_min = X1min
x2_min = X2min
x1_max = X1max
x2_max = X2max
x1 = np.linspace(x1_min, x1_max, len_x1)
x2 = np.linspace(x2_min, x2_max, len_x2) 

i = 0 
inputs_x = []
while i < len(x1):
    j = 0
    while j < len(x2):
        inputs_x.append([x1[i],x2[j]])
        j = j + 1
    i = i + 1
inputs_x_array = np.array(inputs_x)   #simply a set of inputs I want to predict at

kernel = C(1.0, (1e-10, 1000)) * RBF(length_scale = [1., 1.], length_scale_bounds=[(1e-5, 1e5),(1e-7, 1e7)]) \
        + WhiteKernel(noise_level=1, noise_level_bounds=(1e-10, 1e10)) #\

gp = GaussianProcessRegressor(kernel=kernel, alpha=noise ** 2, n_restarts_optimizer=100) 

# Fit to data using Maximum Likelihood Estimation of the parameters
gp.fit(X, y.reshape(-1,1)) #removing reshape results in a different error 

y_pred_index, y_pred_sigma_index = gp.predict(inputs_x_array, return_std=True)

Despite working with numerous variants of kernels, I continue observing convergence errors when trying to find an optimal fit of the hyperparameters to the data:

/.local/lib/python3.6/site-packages/sklearn/gaussian_process/gpr.py:481: ConvergenceWarning: fmin_l_bfgs_b terminated abnormally with the  state: {'grad': array([ 3.89194489e-03,  9.32690036e-03, -0.00000000e+00,  6.42836597e+01]), 'task': b'ABNORMAL_TERMINATION_IN_LNSRCH', 'funcalls': 128, 'nit': 26, 'warnflag': 2}
  ConvergenceWarning)

I've attempted to add/multiply RBF kernels, vary the bounds for the hyperparameters, and include WhiteNoise, but none of my approaches appear to work. Any thoughts on what I can do to avoid this error and select a good kernel for fitting the data?

1 Answer 1

1

I'm not certain that this is a good kernel for your data, but just by limiting the hyperparameter bounds I did manage to get rid of the ConvergenceWarning:

kernel = C(1.0, (1e-3, 1e3)) * RBF(length_scale = [.1, .1], length_scale_bounds=[(1e-2, 1e2),(1e-2, 1e2)]) \
        + WhiteKernel(noise_level=1e-5, noise_level_bounds=(1e-10, 1e-4))

Asking for gp.kernel_.get_params(deep=True) yields

{'k1': 1.51**2 * RBF(length_scale=[0.843, 1.15]),
 'k1__k1': 1.51**2,
 'k1__k1__constant_value': 2.275727769273166,
 'k1__k1__constant_value_bounds': (0.001, 1000.0),
 'k1__k2': RBF(length_scale=[0.843, 1.15]),
 'k1__k2__length_scale': array([0.84331346, 1.15091614]),
 'k1__k2__length_scale_bounds': [(0.01, 100.0), (0.01, 100.0)],
 'k2': WhiteKernel(noise_level=1.4e-08),
 'k2__noise_level': 1.403204609548082e-08,
 'k2__noise_level_bounds': (1e-10, 0.0001)}

See also this remark.

Sign up to request clarification or add additional context in comments.

1 Comment

When attempting to predict using your solution, e.g. gp.predict(np.array([0.,0.]).reshape(1, -1), return_std=True), the following error is returned: \Local\Continuum\anaconda3\lib\site-packages\sklearn\gaussian_process\gpr.py:336: UserWarning: Predicted variances smaller than 0. Setting those variances to 0. warnings.warn("Predicted variances smaller than 0. It appears there is still an underlying problem in the training/fitting process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.