Fastest Way of Opening and Reading .csv Files (Currently using xlsread)
156 views (last 30 days)
Show older comments
Ibro Tutic
on 23 May 2016
Answered: Kristoffer Walker
on 23 Aug 2024 at 19:08
I am currently trying to convert 100,000+ csv files (all the same size, with the same data structuring on the inside) to mat files, and I am running into the issue that it takes an extremely long time, and sometimes Excel stops responding. Are there any other functions that could cut down on the read time of these .csv files?
I read something about trying the COM server that runs Excel, but I am not sure how to implement it. Any thoughts?
0 Comments
Accepted Answer
More Answers (4)
Jeremy Hughes
on 23 Aug 2017
In many cases, you can just use the following pattern to read a large collection of files,
ds = datastore('folder/containing/your/files')
while(hasdata(ds))
t = read(ds)
% do stuff to t.
end
Hope this helps,
Jeremy
0 Comments
TastyPastry
on 23 May 2016
There's a function csvread() which only works on numeric data.
The other way you can do it is to use textscan(). Both of those methods should be faster than xlsread() since xlsread() uses Excel, which is pretty slow.
Kristoffer Walker
on 23 Aug 2024 at 19:08
In my experience, the absolute fastest method is textscan. Here is a benchmark to support my claim using a 3.6 GB CSV file with 4 columns.
Using textscan: 81 seconds
Using readtable: 143 seconds
I tried dlmread and csmread, but they had problems with parsing input. They are not recommended.
Good luck.
Kris
0 Comments
See Also
Categories
Find more on Spreadsheets in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!