Load a bunch of ASC files and save the results in a MAT file

Hi,

We are loading a vast of amount of ASC text files containing headers, text and numbers. We are using a loop and the textscan function to retrieve the data and save the final output as MAT file for further post-processing. This is rather time consuming and takes forever. Do you have any advice to speed up the process ?

Thank you,

Answer 1

[Walter Roberson]

If you are willing to assign a fixed maximum length to each text field, and the format of the lines is predictable, then in some cases it can be worth writing a C program that scans the lines and writes out the content in binary format with fixed width fields. Once that is done, in MATLAB you might be able to use memmapfile() . Or, if when writing you write the text fields to a separate file, then you could fread() the binary to pull in the entire numeric array at the same time.

Or see https://sourceforge.net/projects/matio/ for a C library that can write .mat files.

The main reason I do not suggest writing directly .mat from C or similar is that you need to have read in the entire array first before you can send it to be written, and that can be a memory problem if your files are quite big (and requires a bunch of memory management to keep extending the file.) Whereas if you write in binary format and use MATLAB fread(), MATLAB already takes care of memory management for extending the file as the data is read in.