How to upload 800 csv files that only contains numbers in a cell keeping their names

Hello, I would like to import my 800 csv files by keeping their name so I can identify them afterwards and perform my image processing. I try this script but it takes too much time. Thank you any help is welcome.

fichiersRecherches = '*.csv'; 
[FileName,PathName] = uigetfile(fichiersRecherches,'Sélectionnez les fichiers qui ont pour extention csv', 'MultiSelect', 'on'); 
 
FileName = cellstr(FileName);
 
m = cell(1,length(FileName)); 
 
for i_file = 1:size(m,2)
    m{i_file} = xlsread(fullfile(PathName, FileName{i_file})); 
end

[Rik]

I suspect xlsread has a lot of overhead when reading csv files. Have you tried other functions to read the files?

Also, 800 files is a lot. If you spend half a second on reading each to a variable, you will still need over 6.5 minutes to read them all.

[Leeee]

Thank you Rik for your answer.

No I did not use another function because I wanted to convert csv files to xlsx because I always had a column in 0 more on each file.

Yes it takes a lot of time and is there a way to decrease this time?

[Rik]

The reading can be done with other functions as well. That you want to convert the output to xlsx doesn't matter.

I was pointing out the fundamental nature of your project: if you want to read 800 files you will need an extremely fast process to keep the total time short. Use the profiler to find the bottlenecks in your function.

[Leeee]

Thanks Rik and I do not know which other function to use.

And sorry I do not know what that means:

''Use the profiler to find the bottlenecks in your function.''

[Leeee]

I tried this without converting to xlsx and I have this error message:

pathname = uigetdir();
 
m = cell(2,400);
 
for i = 1:2
    for j = 1:100
        for k = 1:4
 
            filename = sprintf('cycle%d_image%d_%d.csv', j, k, i);
 
            if exist(fullfile(pathname, filename), 'file')~=2
                error('File not found: %s', fullfile(pathname, filename))
            end
 
            m{i,4*(j-1)+k} = csvread(fullfile(pathname, filename));
 
        end
    end
end
/////////////The answer/////////
Error using dlmread (line 147)
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1, field number 3) ==>
;1167,000000;1147,000000;1137,000000;1182,000000;1131,000000;1105,000000;1230,000000;1174,000000;1045,000000;1115,000000;1164,000000;994,000000;1065,000000;942,000000;971,000000;1111,000000;1319,00000...

Error in csvread (line 48)
    m=dlmread(filename, ',', r, c);

Error in bakh2 (line 15)
            m{i,4*(j-1)+k} = csvread(fullfile(pathname, filename));

[Jan]

For "use the profiler":

doc profile

For the message

Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1, field number 3) ==>

There seems to be an unexpected string in row 1 and column 3.

Answer 1

[Jan]

As I said: The number contain commas as decimal separators. Before such a file can be imported, in much be converted. This costs a lot of time.

Maybe this is more efficient to fix the file contents:

function Comma2Dot(FileName)
file  = memmapfile(FileName, 'writable', true);
comma = uint8(',');
point = uint8('.');
file.Data(transpose(file.Data == comma)) = point;
end

Afterwards a simple fscanf(fid, '%g;', [472, Inf]) will import the data efficiently.

By the way, all decimal places are "000000" only. This means that storing only the integer part would be ways better. Storing the data in binary format would even better again. So teh main problem is that a really inefficient file format has been chosen. and you cannot blame the import of MATLAB.

With the original data:

tic;
Comma2Dot('test2.csv');
% Emulatre DLMREAD:
fid = fopen('test2.csv');
C   = textscan(fid, '', -1, 'Delimiter', ';', 'EndOfLine', '\r\n', ...
               'CollectOutput', 1);
fclose(fid);
data = C{1};
toc

% Elapsed time is 0.120229 seconds.

Now try:

% Write data in binary format:
fid = fopen('TestData.bin', 'W');
% Number of dimensions and size
fwrite(fid, [ndims(data), size(data)], 'uint64');
fwrite(fid, data, 'uint16');
fclose(fid);

tic;
fid = fopen('TestData.bin', 'r');
nDimsData = fread(fid, 1, 'uint64');
sizeData  = fread(fid, [1, nDimsData], 'uint64');
data      = fread(fid, sizeData,  'uint16');
fclose(fid);
toc

% Elapsed time is 0.013736 seconds.

The timings might be unfair, because reading data, which have been written to disk directly before, will be taken from the disk cache. But the accelerateion is expected: Compare the file sizes of 2'632 kB for the text file and 442 kB for the binary file.

So the actual optimization is not to improve the Matlab code, but to use a smart format to store the file.

[Rik]

'Answer' by Leeee moved here:

Hi Jan and thank you for your answer!

I did not really understand your script with the Comma2Dot function

[Rik]

[deleted, I didn't read the answer correctly]

[Leeee]

I see that with your script, the response time is just for an element of 800 matrix.

[Jan]

@Leeee: Whose script? Which response time?

The Comma2Dot function replaces commas by dots inside a file.

[Leeee]

Hi Jan I was talking about your script and how to adapt it to my 800 files and have a faster time than strrepInFile

[Leeee]

@Jan I try your method but it does not work

[Jan]

"It does not work" is a lean explanation and does not allow to understand, what the problem is. Please post the details.

I've used the posted methods successfully and the timings show, that it will be much faster, if you use a proper file format instead of a text files with commas and meaingless zeros as decimal places. I've explained thois exhaustively already and do not know, how I can help you now.

[Leeee]

@Jan Sorry it's working now your method I did not fit well to my script.

But if you want to make it easier you just have to use strrep instead of strrepInFile thanks to one of your suggestions on another topic.

Answer 2

[Leeee]

Thanks for your answers I followed your advice but it still takes 2 minutes code and would anyone have a better idea?

And the profile gives her:

pathname = uigetdir();
 
m = cell(2,400);
 
for i = 1:2
    for j = 1:100
        for k = 1:4
 
            filename = sprintf('Cycle%d_Image%d_%d.csv', j, k, i);
 
            if exist(fullfile(pathname, filename), 'file')~=2
                error('File not found: %s', fullfile(pathname, filename))
            end
 
            strrepInFile(fullfile(pathname, filename), ',', '.');
 
            m{i,4*(j-1)+k} = dlmread(fullfile(pathname, ['new_' filename]), ';');
 
            delete(fullfile(pathname, ['new_' filename]))
 
        end
    end
end

[Rik]

Are you using the fastest drive you have? Can you afford to put this on a SATA SSD or a RAM-disk? There is just a physical limit in reading speed that will limit your code. If you want to read a lot of file it is going to take time.

[Jan]

How large are the files? I'm confused by "strrepInFile". Can you post some example data? Perhaps it contains commas as decimal separators?

[Leeee]

Hello Rik,

No I admit that it is not very efficient my computer and I will try on a SSD SATA see what gives it.

Answer 3

[Leeee]

Merci pour vos réponses et une partie attachée de mes données!

https://drive.google.com/file/d/1wKvOSL9tJUyEANB2tO2J3RXPQEc8-xQz/view?usp=sharing

[Jan]

The Google drive is a tedious external service. Use the file attached to my comment instead.

Answer 4

[Leeee]

@Jan I followed one of your methods using strrep in place of strrepInFile and it has greatly reduced the time and I only have the dlmread that takes up to 80% of the script time.