Readtable For Loop for importing large quantities of csv files in to Matlab

112 views (last 30 days)
John Doe
John Doe on 10 Oct 2018
Commented: John Doe on 10 Oct 2018
Hello,
I am currently working on a project which requires me to import sequential data into matlab so that it can be processed and analysed as one data set. There is multiple csv files because the data logger creates a new file every 40mb or so.
I have put together the following code to import the data.
if true
% code
end
clear all
close all
d = uigetdir();
filePattern = fullfile(d, '*.csv');
file = dir(filePattern);
x={};
for k = 1 : numel(file)
baseFileName = file(k).name;
fullFileName = fullfile(d, baseFileName)
x{k} = readtable(fullFileName);
end
There are 4 things I have not done yet.
1. Preallocate memory for x - I don't know how much to preallocate.
2. I am not appending the data in to one matrix. Rather, a cell array combining each set of data. For Example 3 sets of data creates a 1x3 cell array each including a table of the data.
3. Added a status print in cmd window i.e: "read file xyz"
4. Not added datetime format because the first column has cotents like this: "31/03/2017 07:58:31"
Here is an example of the output.
In this case what I actually need is a 64,803 x 191 table (Need to keep variable names - each row is a specific variable).
How do I achieve this?
For any given data set I will always have the same amount of columns but the number of rows may vary in each file.

Accepted Answer

Guillaume
Guillaume on 10 Oct 2018
1. Replace
x = {};
by
x = cell(1, numel(file)); %better name for that variable advised. How about: tables
Note that unless the number of files is significant, this won't have any noticeable impact as it only allocates memory for the cells, not for the tables that go into the cells. This is normal and how cell arrays work.
2. That's fine and is the proper way to do it
3. After the readtable:
fprintf('read file %s\n', fullFileName);
4. Assuming all the tables have the same variable names, to concatenate them all into one table, after the loop:
bigtable = vertcat(x{:});
  3 Comments

Sign in to comment.

More Answers (0)

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!