Copy 60 most recent values from appended .csv to a new .csv every minute?

Copy 60 most recent values from appended .csv to a new .csv every minute. MATLAB

I'd like to extract the 60 most recent values from a .csv file called MyFile which is appended with 3 new values in the row below the previous set every minute using a timer function:

dlmwrite('MyFile.csv', [MyValue,MyValue2,MyValue3], '-append');

Say I may have 150 rows of values in MyFile after 100 minutes and I want to extract the latest 60 to save to another file called MyFile2 The process happens indefinitely because of an endless timer i.e it accumulates data over time so the row size of the MyFile.csv is increasing by 1 every minute. So, 3 columns of data in each row, new row every minute.(Ignore the timer, i'm just showing here that it works and is used to get the new MyValue's) Timer:

Period = 60; % Update period in seconds

tim = timer('Period', Period, 'ExecutionMode', 'fixedRate',...
    'TimerFcn', 'ThisScript');

start(tim)

stop(tim) 

runtmp = fullfile('MyScriptLocation','MyScript');
run(runtmp);

How can I continually copy over the 60 most recent sets of values from the file and store them in MyFile2?

Is there a size function I could use for my .csv file so that it fetches the size of the .csv as it grows and then pulls the most recent 60 values added to it?

I feel like this would be the easiest way to do it but i'm still unsure.

Rough pseudocode i'm unsure of (don't think its right):

Take size of MyFile
   for k=1:size(MyFile)
          dlmwrite('MyFile2.csv', [MyValue(k),MyValue2(k),MyValue3(k)], '-append');

but obviously need code there that says subtract size from each row number so that the correct values are appended to MyFile2. Hope i've made this clear. If there aren't 60 rows of values in MyFile yet then a text/string message 'N/A' should appear.

Answer 1

[Walter Roberson]

The operating systems that support MATLAB, do not have any functionality that allows the number of lines in a file to be known, short of reading the entire file from beginning to end and counting the number of lines.

[P]

Okay. But what if I just read the entire file then as you suggested, how could I do it? the task is not cpu resource critical in this case. I want to get the average of the latest 60 values in this file, i don't necessarily need to copy them to a new file. I just can't think of a way to make sure MATLAB knows to always take the latest 60 since the file always updates. Thanks.

[Walter Roberson]

filelines = regexp(fileread('MyFile.csv'), '\n', 'split');

Now you need to check filelines{end} to be sure it is a complete line (the likelyhood depends on which OS you are using.) If it is not then delete filelines{end}. Now

mean(str2double(filelines{end-59:end}))

[P]

I get the error:

??? Subscript indices must either be real positive integers or
logicals.

Where is the column specifier in this code? I only want the mean of one column. Thanks

[Walter Roberson]

In your earlier spec you said to put NaN if there was no value available, but you did not indicate how you want to treat those NaN in the mean(). If your intention was to ignore the NaN in the mean, then

lines = filelines(max(1,end-59):end);

to get as many lines as possible up to 60.

col_means = mean(cell2mat(cellfun(@(S) scanf(S, '%f,%f,%f'), lines)));

then col_means(1) or (2) or (3) as appropriate.

[P]

Combining your possible solution I may have come up with an alternative which i'm going to try first. May post in a new thread if I still have problems. Thanks for your help.

[P]

Also. the line defining col_means returns:

??? Undefined function or method 'scanf' for input arguments of
type 'char'.

Can scanf be used here?

[Walter Roberson]

sscanf() instead of scanf()

Answer 2

[Image Analyst]

Can you just read the whole thing with csvread() then extract the last 60 rows?

data = csvread(filename); % Get all the data.
[rows, columns] = size(data); % Find out how many rows.
if rows >= 61
  last60Rows = data(end-59:end,:);
  csvwrite(newFileName, last60Rows);
end

[P]

Thank you!!!!

Using test data this seems to be working. I will need to test it for sure tomorrow when the data I get for my script is available from its external source.