How to use deep learning for interpolation properly

Hi!

What I do wrong? I try to apply deep learning for interpolation function sin. And It learns not good. Insted of sin its just show line function. This is my code

clear all;

clc;

close all;

N=100;

T=2*pi;

x=0:T/(N-1):T;

y=sin(x);

%----- normalization ---------------

y=y+abs(min(y));

y=y./max(y);

x=x./max(x);

x=x;

y=y;

%---------- end normalization -----

plot(x,y,'o')

%------------------------

layers = [

featureInputLayer(1)

fullyConnectedLayer(10)

fullyConnectedLayer(1)

regressionLayer

];

options = trainingOptions('adam', ...

'MaxEpochs',20, ...

'MiniBatchSize',10, ...

'Shuffle','every-epoch', ...

'Plots','training-progress', ...

'Verbose',false);

[net inf] = trainNetwork(x',y',layers,options);

y1 = predict(net,x');

figure(2); hold on;

plot(x,y,'oblack','MarkerSize',20)

plot(x,y1,'x','MarkerSize',20)

Answer 1

[Abolfazl Chaman Motlagh]

Hi,

The Sin(x) function is a complete nonlinear function, on the other hand your network is too simple to handle such a nonlinearity. for overcoming this problem you can:

• Increase number of Layers
• Increase number of Parameters
• add an Activation layer to handle nonlinearity
• you might also give more iterate to network for learning , maxepoch=2000 since your data and network are small.

here a simple solution for your model:

layers = [
featureInputLayer(1)
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
fullyConnectedLayer(1)
regressionLayer
];

the result is (5 fully connected layer with 20 parameters)

here are some other examples:

(2 fully connected layer each with 200 parameters) ( you can see still not converge correctly)

(3 fully connected layer each with 5 parameters)

so you can change this parameters (even change layers parameters indivisually) to see what is best option for your application.

[Alexey Kozhakin]

Thank you for your answer!

What does it mean reluLayer after fullyConnectedLayer(20)? Is it mean that 20 neurons have relu function type?

layers = [

featureInputLayer(1)

fullyConnectedLayer(20);

reluLayer;

fullyConnectedLayer(20);

reluLayer;

fullyConnectedLayer(20);

fullyConnectedLayer(1)

regressionLayer

];

This structure mean that net has 1 input layer than layer 20 hidden neurons, then next layer of 20 hidden neurons, then again layer of 20 hidden neurons and then output, right? What tipe of function of these 20 hidden neurons, relu function?

[Abolfazl Chaman Motlagh]

the relulayer, apply relu function on output of a layer :

https://en.wikipedia.org/wiki/Rectifier_(neural_networks).

So the Strycture of network become 1 input (as a number), then at next layer 20 neurons, then output given to relu function, output of that will become input of next layer with 20 nuerons and so on ... . the relu here is used for activation function after each layer :

for --> fullyConnectedLayer(20);reluLayer

[Alexey Kozhakin]

Could you send me your code please, because I tried to do what you recommended

• Increase number of Layers
• Increase number of Parameters
• add an Activation layer to handle nonlinearity
• you might also give more iterate to network for learning , maxepoch=2000 since your data and network are small.

and it is still remain linear

[Alexey Kozhakin]

According this

for --> fullyConnectedLayer(20);reluLayer

I have a question how can I apply activation function for each neuron in layer. Because now, according formula it looks like each neuron is linear function, but I want something like this in each layer

I want each neuron from 20 in the layer has activation function not just linear function

Is it possible to do it in Matlab?

[Abolfazl Chaman Motlagh]

for first question, here is some example:

N=100;
T=2*pi;
x=0:T/(N-1):T;
y=sin(x);
y=y+abs(min(y));
y=y./max(y);
x=x./max(x);

layers = [
featureInputLayer(1)
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
reluLayer;
fullyConnectedLayer(20);
fullyConnectedLayer(1)
regressionLayer
];

options = trainingOptions('sgdm', ...
'MaxEpochs',600, ...
'MiniBatchSize',10, ...
'Shuffle','every-epoch', ...
'Plots','training-progress', ...
'InitialLearnRate',0.01, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropFactor',0.1, ...
'LearnRateDropPeriod',200, ...
'Verbose',false);

[net inf] = trainNetwork(x',y',layers,options);
y1 = predict(net,x');

plot(x,y,'o','LineWidth',2)
hold on
plot(x,y1,'-','LineWidth',2)
legend('Sin(x)','Predicted')

[Abolfazl Chaman Motlagh]

for second question:

by putting relu function after one layer, all outputs pass over the relu function. so just by adding reluLayer in layers list, you are using this function on every neurons.

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.relulayer.html

there are some standart implementation of other common activation function in matlab, like

• relu
• leakyRelu
• clippedRelu
• elu
• tanh
• swish

https://www.mathworks.com/help/deeplearning/ug/list-of-deep-learning-layers.html

but if you want you can define your own activation function using matlab:

functionLayer:

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.functionlayer.html

or even complete Deep Network Layer structure and operation:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html