Running out of memory

3 views (last 30 days)
Theo Score
Theo Score on 6 Feb 2017
Commented: Walter Roberson on 8 Feb 2017
Hi All,
I have a couple of data text files which I am extracting data of interest from.
The total text data files size is about 549 MB, which I am writing into a single text file and it comes to about 500 MB since I am disregarding some information in the text files.
When I try to write to read and write the output text file into an excel file;
1. The process consumes all my RAM (8G) and computer freezes
2. I got this error at the end
Error using xlswrite (line 219)
The specified data range is invalid or too large to write to the specified file format. Try writing to an XLSX file and use Excel A1
notation for the range argument, for example, A1:D4.
Error in Data_Overall_Ext_Dir (line 29)
xlswrite (output, data);
Is there any way I can handle this?
With limited number of files, the script works well but as the number of files (data size) increases, I have this issue.
Kindly help.
  7 Comments
Theo Score
Theo Score on 8 Feb 2017
Edited: Theo Score on 8 Feb 2017
Hi Walter,
Thank you for your response. I thought of a work around the problem. I will use Linux sort/ split commands in Linux but I need to get an extra component to my data. This script:
workDir = 'C:\Users\Theo_Score\Desktop\COM_CFDEK';
data = fullfile(workDir, '*.data');
files = dir(data);
fileout = fullfile(workDir, 'Merged.txt');
fout = fopen (fileout, 'w');
for k = 1000:1000:100000
cntfiles = fullfile (workDir, ['dump' num2str(k) '.data']);
fin = fopen (cntfiles);
linecount =0;
while ~feof(fin)
linecount = linecount + 1;
tline = fgetl(fin);
if linecount >= 10 % Number of header lines being skipped
fprintf (fout, '%s\r\n', tline);
end
end
fclose(fin);
end
Was skipping 9 lines of the .data files. I need some help to modify this script so that it reads line 2 (which is time step, say 1000) as shown in the first 9 lines I was previously skipping.
ITEM: TIMESTEP
1000
ITEM: NUMBER OF ATOMS
26
ITEM: BOX BOUNDS ff ff ff
-0.1 0.1
0 0.4
-0.1 0.1
id type x y z ix iy iz vx vy vz fx fy fz omegax omegay omegaz radius
I want to multiply 1000*0.00001 and return the value in the first column and the previous first column becomes the second column.
The first 5 lines from the previous script would be like
1 1 -0.0506691 0.301248 -0.0540098 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
7 1 0.0119942 0.300662 -0.0584242 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
6 1 0.0589997 0.30511 -0.0540171 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
3 1 -0.0512266 0.330591 -0.0441473 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
16 2 -0.0118166 0.320646 -0.046286 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
After the modification, I would want an output like
0.01 1 1 -0.0506691 0.301248 -0.0540098 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 7 1 0.0119942 0.300662 -0.0584242 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 6 1 0.0589997 0.30511 -0.0540171 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 3 1 -0.0512266 0.330591 -0.0441473 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 16 2 -0.0118166 0.320646 -0.046286 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
I would appreciate help on this.
Walter Roberson
Walter Roberson on 8 Feb 2017
I would probably toss it all into perl...

Sign in to comment.

Answers (1)

cr
cr on 6 Feb 2017
If xls format is not critical requirement, you may want to use fileIO operations and produce a csv (or some text delimited) format. Once a file text file is successfully generated it can be converted to xls using MS Excel.
  1 Comment
Theo Score
Theo Score on 7 Feb 2017
Edited: Theo Score on 7 Feb 2017
Hi cr,
I have embedded the script on this thread on a response to Walter.
Essentially, I need to plot graphs with the information I get so I need the data to be in .xlsx. Further comment is also given in that response.
I appreciate your further help.
Best regards.

Sign in to comment.

Tags

No tags entered yet.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!