Data Division of Neural Network training?

I have a 2000000 by 7 matrix where the first 6 volumns are inputs and the last column is the output. So I am training a neural network to fit the 7th column as a function of the first 6 columns:

Input=Input;
Output=Output;
ANN=fitnet([25],'trainbr');
ANN=train(ANN,Input,Output);

My problem is the data division. Since the data is divided into three section: validation, test, and training. I want to make sure that the last 100 rows (index: 1999900 to 2000000) are included in the training set, and the rest of the data (rows) are randomized into the three sections. "dividerand" does what I want but unfortunately, it does not gurantee that the last 100 rows are in the training set. Do you guys have any solutions?

[NAVNEET NAYAN]

Is it possible for you to separate the last 100 rows from the dataset and then do the random allotment using "dividerand" ?

[Ali Almakhmari]

Can you elaborate further on this? I can definitely seperate the last 100 rows but I am not sure how to use dividerand and still make sure that the 100 last rows in the training section.

[NAVNEET NAYAN]

What I am saying is remove the last 100 rows and use "dividerand" on the rest of the data.

[Ali Almakhmari]

ahhh...how will this ensure that the 100 rows are in the training section?

Answer 1

[Shreeya]

Hello Ali Almakhmari

To ensure that the last 100 rows are included in the training set, you can split the data into two arrays, the first set having data from index 0 to 1999900 and the second one should contain data from 1999900 to 2000000. Then, you can use the “dividerand" function to split the first array into training, testing and validation datasets.

After that you can concatenate the training dataset obtained from splitting the first array with the last 100 rows to create the desired input features dataset.

Hope this helps!