MATLAB Answers

How to split huge .csv files to multiple .csv files based on size?

30 views (last 30 days)
Anjan on 20 Nov 2018
Edited: Walter Roberson on 13 Feb 2021
I am trying to split huge .csv files (11 GB) that has both combination of text and numbers into mutiple files based on size (0.5 GB each). I tried using some of the answers in the matlab community but no luck
I hope someone can help!
Anjan on 25 Nov 2018
Thanks Walter, It does work but I had to split many files. The code that user "dbp" recommended in the below comment works great if you want to split files in Matlab.

Sign in to comment.

Accepted Answer

dpb on 21 Nov 2018
fidR=fopen('originalfile.csv','r'); % open the big file to read
NFiles=20; % number of files to create
NPerFile=round(Nrows/NFiles)+1; % rough number records / file
for i=1:NFiles
fidW=fopen(num2str(i,'fileNew%00d.csv'),'w'); % open a file to write
for j=1:NPerFile
fwrite(fidW,fgets(fidR)); % transcribe lines verbatim
fidW=fclose(fidW); % close that one
fidW=fclose(fidW); % close that one
is a poor-man's split tossed off at the console here; the rounding for number of records per file should catch the whole file; the error routine should only occur when the last file runs into feof on the last one if my logic is right.
And, yes, unfortunately, we can't always make others do sensible things about how they collect data we're subsequently given, granted...I recognized that was likely the case, hence the smiley.
Walter Roberson
Walter Roberson on 13 Feb 2021
compose() or the undocumented sprintfc() are good at creating separate outputs.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!