Imported keras model gives a different(wrong) output in MATLAB

Hi, I'm working on both tensorflow keras with python and MATLAB for my research. However, I found that there was a critical issue when I import a keras model into MATLAB deep learning layers. It's not a simple compatibility issue like unsupported layers or something. Even for the supported layers, the original keras model and imported MATAL one give different results for an identical input. Here's the thing...

In the first test, TensorFlow keras model is defined as below:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(2*2*1, input_shape=(2,)),
    tf.keras.layers.ReLU()
])
model.build()
model.save('testDenseLayer.h5')
weights = model.weights
W0 = weights[0]
print(W0)
b0 = weights[1]
print(b0)
z = tf.ones([1,2])
x = model(z)
print(x)

and it gives

<tf.Variable 'dense/kernel:0' shape=(2, 4) dtype=float32, numpy=
array([[ 0.7081969 , -0.63268185, -0.848845  ,  0.11187315],
       [-0.36178374,  0.02817345,  0.95137477,  0.11700463]],
      dtype=float32)>
<tf.Variable 'dense/bias:0' shape=(4,) dtype=float32, numpy=array([0., 0., 0., 0.], dtype=float32)>
tf.Tensor([[0.34641314 0.         0.10252976 0.22887778]], shape=(1, 4), dtype=float32)

The following imported MATLAB model

layers = importKerasLayers('testDenseLayers.h5', 'ImportWeights', true);
W0 = layers(2).Weights' % basically, the column vector is the default in MATLAB while row vector is the default in python
b0 = layers(2).Bias'
lgraph = layerGraph(layers);
dlnet = dlnetwork(lgraph);
dlz = dlarray(ones(1, 2, 1, 1, 'single'), 'SSCB');
dlx = predict(dlnet, dlz);
x = extractdata(dlx)'

also give the same weights.

W0 =
  2×4 single matrix
   0.7081969  -0.6326818  -0.8488450   0.1118731
  -0.3617837   0.0281734   0.9513748   0.1170046
b0 =
  1×4 single row vector
     0     0     0     0
x =
  1×4 single row vector
   0.3464131           0   0.1025298   0.2288778

It's obvious except some round-off error in x.

After adding reshape layer,

   model = tf.keras.Sequential([
    tf.keras.layers.Dense(2*2*1, input_shape=(2,)),
    tf.keras.layers.ReLU(),
    tf.keras.layers.Reshape((2, 2, 1))
])
model.build()
model.save('testDenseReshapeLayers.h5')
z = tf.ones([1,2])
x = model(z)
print(x[0,:,:,0])

it gives

tf.Tensor(
[[0.         0.        ]
 [0.69579005 0.        ]], shape=(2, 2), dtype=float32)

In MATLAB, I have to replace the reshape layer with custom layer I defined:

layers = importKerasLayers('testDenseReshapeLayers.h5', 'ImportWeights', true);
lgraph = layerGraph(layers);
lgraph = replaceLayer(lgraph, 'reshape', reshapeLayer('new_reshape', 2, 2, 1));
dlz = dlarray(ones(1, 2, 1, 1, 'single'), 'SSCB');
dlx = predict(dlnet, dlz);
x = extractdata(dlx)'

it gives

x =
  2×2 single matrix
           0           0
   0.6957901           0

So far so good....

Here comes a problem when a convolutional layer is included.

model = tf.keras.Sequential([
    tf.keras.layers.Dense(2*2*1, input_shape=(2,)),
    tf.keras.layers.ReLU(),
    tf.keras.layers.Reshape((2, 2, 1)),
    tf.keras.layers.Conv2D(1, 3, padding='same')
])
model.build()
model.save('testDenseReshapeConvLayers.h5')
weights = model.weights
W1 = weights[2]
b1 = weights[3]
print(W1.shape)
print(W1[:,:,0,0])
print(b1)
z = tf.ones([1,2])
x = model(z)
print(x[0,:,:,0])

Is gives

(3, 3, 1, 1)
tf.Tensor(
[[-0.4860901  -0.03003603 -0.03072345]
 [-0.22509798  0.07996976 -0.2171354 ]
 [ 0.35814303  0.27246052 -0.31091872]], shape=(3, 3), dtype=float32)
<tf.Variable 'conv2d/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>
tf.Tensor(
[[-0.3222635   0.11868786]
 [-0.0455985  -0.04457825]], shape=(2, 2), dtype=float32)

In MATLAB,

layers = importKerasLayers('testDenseReshapeConvLayers.h5', 'ImportWeights', true);
W1 = layers(5).Weights'
b1 = layers(5).Bias'
lgraph = replaceLayer(lgraph, 'reshape', reshapeLayer('new_reshape', 2, 2, 1));
dlnet = dlnetwork(lgraph);
dlz = dlarray(ones(1, 2, 1, 1, 'single'), 'SSCB');
dlx = predict(dlnet, dlz);
x = extractdata(dlx)'

but is gives

W1 =
  3×3 single matrix
  -0.4860901  -0.2250980   0.3581430
  -0.0300360   0.0799698   0.2724605
  -0.0307235  -0.2171354  -0.3109187
b1 =
  single
     0
x =
  2×2 single matrix
   0.4043748   0.5315413
   0.1186879  -0.3340813

Somehow, the weight matrix has been transposed and the result x is totally different from python's one, not a round-off error.

In my best knowledge, the column vector is the default in MATLAB while the row vector is the default in python. So, I have to carefully consider which dimension goes where in a tensor. However, for the compatibility, the importKerasLayers function has to do automatically in my opinion.

The point is, the imported keras model in MATLAB gives different results from in python and it's a huge problem.

Do I need to modify or something for the compatibility?

What can I do to get the same output?

Answer 1

[Srivardhan Gadila]

layers = importKerasLayers('testDenseReshapeConvLayers.h5', 'ImportWeights', true);
W1 = layers(5).Weights'
b1 = layers(5).Bias'
lgraph = replaceLayer(lgraph, 'reshape', reshapeLayer('new_reshape', 2, 2, 1));
dlnet = dlnetwork(lgraph);
dlz = dlarray(ones(1, 2, 1, 1, 'single'), 'SSCB');
dlx = predict(dlnet, dlz);
x = extractdata(dlx)'

From the above code of yours, I see that in lines 2,3 and 8 you used the transpose operator ' (i.e., extractdata(dlx)') and I think that it mght be the reason for the transposed output.

The activations/outputs obtained from a layer will be the same irrespective of how individual frameworks store their weights( ordering of filter weights). Once try defining network with two convolutional layers only and check the output of weights of 1st convolution layer and Output from 2nd convolution layer for a given input as follows:

n1 is the network imported from keras (Since I don't know your implementation for your reshape layer in matlab, I defined a network with 2 convolution layers with no activation layers like relu etc.)

>> n1 = importKerasLayers('testDenseReshapeConvLayers.h5',"ImportWeights",1)
n1 = 

  3x1 Layer array with layers:

     1   'ImageInputLayer'   Image Input   6x6x1 images
     2   'conv2d_2'          Convolution   2 3x3x1 convolutions with stride [1  1] and padding 'same'
     3   'conv2d_3'          Convolution   3 3x3x2 convolutions with stride [1  1] and padding 'same'
     
>> n2 = SeriesNetwork(n1);
>> W1 = n1(2).Weights

  3×3×1×2 single array

W1(:,:,1,1) =

    0.2237    0.1550    0.2994
   -0.3774   -0.1723   -0.4106
   -0.4185   -0.0205   -0.4234


W1(:,:,1,2) =

   -0.1033    0.0642    0.0711
   -0.4454    0.2370   -0.1090
   -0.2108   -0.3168   -0.0470

>> W1(1,:,:,1)

ans =

  1×3 single row vector

    0.2237    0.1550    0.2994
>> z = activations(n2,ones(6,6),'conv2d_3');
>> z(1,:,1)

ans =

  1×6 single row vector

   -0.2704   -0.2422    0.1859    0.1859    0.1456    0.3901

Define the network similar to above in keras and check for the following:

1. Weights of the first convolution layer: W1
2. Portion of weights of first convolution layer as follows: W1[0,:,:,0] (W1(1,:,:,1) in matlab)
3. Give input x as a matrix of ones of size [1,6,6,1] and check the output z of 2nd convolution layer as follows: z[0,:,0] (z(1,:,1) in matlab)

You can observe that the weights will be stored in different ordering but the output z will be the same from both the frameworks.