Facing an error when attempting to pass my training data to train a network.

im new to running ML models in matlab so please try to explain it in simple words,i am attempting to run a tcn model on a dataset which has 8 features which includes the timestamps necessary as well. when attempting to train the model i get the following error when i attempt to reshape the data: "The training sequences are of feature dimension 2795 10 but the input layer expects sequences of feature dimension 9."

here is the code:
clc;
clear;
close all;
trainData = readtable('train_net_gen.csv');
testData = readtable('test_net_gen.csv');
trainData.Timestamp = datetime(trainData.Timestamp, 'InputFormat', 'yyyy-MM-dd HH:mm:ss');
testData.Timestamp = datetime(testData.Timestamp, 'InputFormat', 'yyyy-MM-dd HH:mm:ss');
trainData.month = month(trainData.Timestamp);
trainData.day_of_month = day(trainData.Timestamp);
trainData.hour = hour(trainData.Timestamp);
trainData.day_of_week = weekday(trainData.Timestamp) - 1;
trainData.minute_of_day = trainData.hour * 60 + minute(trainData.Timestamp);
testData.month = month(testData.Timestamp);
testData.day_of_month = day(testData.Timestamp);
testData.hour = hour(testData.Timestamp);
testData.day_of_week = weekday(testData.Timestamp) - 1;
testData.minute_of_day = testData.hour * 60 + minute(testData.Timestamp);
% Moving averages and differences
trainData.moving_avg_3_production = movmean(trainData.next_day_from_system_kwh, [2, 0]);
trainData.diff_production = [NaN; diff(trainData.next_day_from_system_kwh)];
testData.moving_avg_3_production = movmean(testData.next_day_from_system_kwh, [2, 0]);
testData.diff_production = [NaN; diff(testData.next_day_from_system_kwh)];
% Remove NaNs
trainData = rmmissing(trainData);
testData = rmmissing(testData);
% Prepare features and target variables
sequenceLength = 10;
featureDimension = 9;
X_train = [trainData.minute_of_day, trainData.from_grid_kwh, trainData.net_consumption, ...
trainData.month, trainData.day_of_month, trainData.hour, trainData.day_of_week, ...
trainData.moving_avg_3_production, trainData.diff_production];
y_train = trainData.next_day_from_system_kwh;
X_test = [testData.minute_of_day, testData.from_grid_kwh, testData.net_consumption, ...
testData.month, testData.day_of_month, testData.hour, testData.day_of_week, ...
testData.moving_avg_3_production, testData.diff_production];
y_test = testData.next_day_from_system_kwh;
% Standardize features
X_train = (X_train - mean(X_train)) ./ std(X_train);
y_train = (y_train - mean(y_train)) ./ std(y_train);
X_test = (X_test - mean(X_test)) ./ std(X_test);
y_test = (y_test - mean(y_test)) ./ std(y_test);
% Calculate the exact number of samples to ensure all data is used
numTrainSamples = floor(size(X_train, 1) / sequenceLength);
numTestSamples = floor(size(X_test, 1) / sequenceLength);
% Reshape to ensure correct input dimensions: [numSamples, sequenceLength, featureDimension]
X_train_seq = reshape(X_train(1:numTrainSamples * sequenceLength, :), [numTrainSamples, sequenceLength, featureDimension]);
y_train_seq = y_train(1:numTrainSamples);
X_test_seq = reshape(X_test(1:numTestSamples * sequenceLength, :), [numTestSamples, sequenceLength, featureDimension]);
y_test_seq = y_test(1:numTestSamples);
% Verify the dimensions
fprintf('Size of X_train_seq: %s\n', mat2str(size(X_train_seq)));
fprintf('Size of y_train_seq: %s\n', mat2str(size(y_train_seq)));
layers = [
sequenceInputLayer(featureDimension, "Name", "input")
convolution1dLayer(3, 64, "Padding", "causal", "Name", "conv1")
layerNormalizationLayer("Name", "layernorm1")
reluLayer("Name", "relu1")
convolution1dLayer(3, 128, "Padding", "causal", "Name", "conv2")
layerNormalizationLayer("Name", "layernorm2")
reluLayer("Name", "relu2")
convolution1dLayer(3, 128, "Padding", "causal", "Name", "conv3")
layerNormalizationLayer("Name", "layernorm3")
reluLayer("Name", "relu3")
fullyConnectedLayer(1, "Name", "output")
regressionLayer("Name", "outputLayer")
];
options = trainingOptions("adam", ...
"MaxEpochs", 100, ...
"MiniBatchSize", 16, ...
"Shuffle", "never", ...
"ValidationData", {X_test_seq, y_test_seq}, ...
"Plots", "training-progress", ...
"Verbose", 1);
try

net = trainNetwork(X_train_seq, y_train_seq, layers, options);
disp("Training successful with simplified sequence-to-one TCN structure!");
catch ME
disp("Error during training:");
disp(ME.message);
end
if exist("net", "var")
y_pred = predict(net, X_test_seq);
rmse = sqrt(mean((y_pred - y_test_seq).^2));
disp(["Production RMSE: ", num2str(rmse)]);
figure;
plot(y_test_seq, "b-", "LineWidth", 1.5);
hold on;
plot(y_pred, "r--", "LineWidth", 1.5);
xlabel("Sample");
ylabel("Standardized kWh");
legend("Actual", "Predicted");
grid on;
end
output:
Size of X_train_seq: [2795 10 9]
Size of y_train_seq: [2795 1]
Error during training:
The training sequences are of feature dimension 2795 10 but the input layer expects sequences of feature dimension 9.
I would like any suggestions on why this error occurs as even when i use it for a subset of features such as 5 features, it still reads the feature dimensions as 10.
Thank you.

Answer 1

[Shishir Reddy]

Hey Shreyas

The error you're encountering indicates a mismatch between the expected input dimensions for the network and the dimensions of the data you're providing. Specifically, the network expects input sequences with a feature dimension of 9, but the reshaped data seems to have an incorrect dimension.

You are trying to reshape ‘X_train’ and ‘X_test’ into sequences of shape [‘numSamples’, ‘sequenceLength’, ‘featureDimension’]. So, it should be ensured that the product of ‘numTrainSamples’, ‘sequenceLength’, and ‘featureDimension’ should match the number of elements in ‘X_train’.

Kindly refer to the following snippet with additional checks and debugging prints which will help to understand the error you’re encountering.

numTrainSamples = floor(size(X_train, 1) / sequenceLength);
numTestSamples = floor(size(X_test, 1) / sequenceLength);
 
% Size before reshaping
fprintf('Size of X_train before reshaping: %s\n', mat2str(size(X_train)));
  
X_train_seq = reshape(X_train(1:numTrainSamples * sequenceLength, :), [numTrainSamples, sequenceLength, featureDimension]);
y_train_seq = y_train(1:numTrainSamples);
  

% Size after reshaping
fprintf('Size of X_train_seq after reshaping: %s\n', mat2str(size(X_train_seq)));
fprintf('Size of y_train_seq: %s\n', mat2str(size(y_train_seq)));
  
%Same process for test data
X_test_seq = reshape(X_test(1:numTestSamples * sequenceLength, :), [numTestSamples, sequenceLength, featureDimension]);
y_test_seq = y_test(1:numTestSamples);
  
 
fprintf('Size of X_test_seq: %s\n', mat2str(size(X_test_seq)));
fprintf('Size of y_test_seq: %s\n', mat2str(size(y_test_seq)));

These debugging prints can be used to verify the size of the data at different stages to catch mismatches if any.

I hope this resolves the issue.

[Shreyas Suresh]

Hi, so i have implemented your suggestion however i am still recieving the same error when i attempt to run it

this is the console output:

Size of X_train before reshaping: [27954 9]

Size of X_train_seq after reshaping: [2795 10 9]

Size of y_train_seq: [2795 1]

Size of X_test_seq: [698 10 9]

Size of y_test_seq: [698 1]

Error during training:

The training sequences are of feature dimension 2795 10 but the input layer expects sequences of feature dimension 9.

I would appreciate it, if you could help me with the further steps i would need to take.

[Shishir Reddy]

The console output shows that the reshaping seems to be correct, as the dimensions of 'X_train_seq' and 'X_test_seq' match the expected ['numSamples', 'sequenceLength', 'featureDimension'] format. However, the error message suggests a misunderstanding by MATLAB about the input dimensions. This might be due to how the data is being fed into the network.

Potential solutions -

• Confirm that X_train_seq is indeed a 3D array with dimensions [2795, 10, 9]. MATLAB might misinterpret if the data is not correctly structured.
• Ensure that X_train_seq is of type double. Sometimes, data type mismatches can lead to unexpected behavior.
• Print the class and size of X_train_seq right before calling trainNetwork to ensure it's being interpreted correctly:

disp("Class of X_train_seq: " + class(X_train_seq));
disp("Size of X_train_seq: " + mat2str(size(X_train_seq)));

If these steps don't resolve the issue, consider simplifying the model or data further to identify the root cause. It's often helpful to strip the problem down to its basics and gradually add complexity back in until the issue reappears. This can pinpoint exactly where the misunderstanding or misconfiguration lies.