Fastest way to load data.

Hello,

I have an array of size ~320 Mb that I want to store in a file such that the loading afterwards is as fast as possible.

Currently simply doing:

save('filename.mat', 'var');

takes around 2.4 seconds.

What flags (if any) can I give the save() or load() functions that can achieve this?

Currently on saving as .mat file using the '-v6' flag (file size now 470Mb) has allowed me to get the load time down to 0.14 seconds!

Are there things I can leverage to reduce this further?

My SSDs read and write speeds are ~1.4 GB/s

Answer 1

[dpb]

save does compression by default now which takes time as you've discovered.

The absolute fastest albeit somewhat less convenient should be using fwrite, fread as far as pure i/o speed.

fid=fopen('filename.bin','w');
fwrite(fid,var)
fid=fclose(fid);

One test here was almost a tie, however, so likely not sufficiently faster to be worth the effort.

The way to speed it up if you can afford to lose some precision would be to only use single-precision instead of double...half as many bytes to read/write. It won't be a full 2X gain because memory access will be just a tad quicker for 8-byte rather than 4-byte increments probably, but should be observeable speedup. Of course, if you need to preserve full precision this isn't an option.