Find out why mat files differ in size

I'm developing a rather complex class hierachy with a few GB of data embedded in its instances which might get saved to mat files for later analysis.

I refactored a lot to improve memory and CPU footprints (using dependent properties, customized loadobj and saveobj methods etc) and saw that the resulting mat file grows in size (using save() with v7.0 and enabled compression). I screwed it up.

I have some old reference mat files from the former versions that are smaller (~30%). However if I load them using the current class definitions, the resulting objects in RAM are almost exactly (just <1% difference) in size (using the great getArrayFromByteStream function, see Serializing/deserializing Matlab data - Undocumented Matlab). That means I can't infer from the instantiated objects, what grew in size.

Question: How do I find out what really gets saved to the mat file, i.e. which variable/object is much larger compared to the old versions?

I can roll-back to my former version via Git, but that does not really help me to understand, why exactly the mat files got bigger.

Any ideas?

Thanks,

Jan

Answer 1

[Jan Kappen]

Got it fixed.

I've followed a similar approach as @Samay Sagar proposed, but ultimately used getArrayFromByteStream, see Serializing/deserializing Matlab data - Undocumented Matlab. And I checked out the old version of my library in a second MatLab session and compared all properties step by step, skipping Dependent properties via reflection.

Root cause: I've split a data table (class table) into two class objects which should've used dependent properties, and an internal table to store the data. Turned out I forgot to make one block of properties transient/dependent to avoid saving them.

Afterwards, the mat file sizes were basicaly the same - quite interesting that there's no difference if the table is saved or a wrapping class around it - both can get compressed very efficiently, very nice Mathworks!

PS, just found out that mat files can be compared visually too: Compare and Merge MAT-Files - MATLAB & Simulink (mathworks.com) and that it can even "look" into objects, but not arbitrarily nested. But it could also be a good starting point:

Answer 2

[Samay Sagar]

You can utilize the "whos" command for thorough examination of variable sizes within MATLAB objects, facilitating the discernment of any modifications in variable dimensions present in MAT files.

Here is a sample script to identify changes in MAT file:

% Extract variables of interest
oldVariables = whos('-file', 'old_version.mat');
newVariables = whos('-file', 'new_version.mat');

% Compare variable sizes
for i = 1:length(oldVariables)
    oldSize = oldVariables(i).bytes;
    newSize = 0;  % Initialize new size
    
    % Find corresponding variable in new version
    for j = 1:length(newVariables)
        if strcmp(oldVariables(i).name, newVariables(j).name)
            newSize = newVariables(j).bytes;
            break;
        end
    end
    
    if newSize == 0
        fprintf('%s:\n', oldVariables(i).name);
        fprintf('  Variable not found in new version\n\n');
    else
        sizeChange = newSize - oldSize;
        percentageChange = (sizeChange / oldSize) * 100;
        
        fprintf('%s:\n', oldVariables(i).name);
        fprintf('  Old Size: %d bytes\n', oldSize);
        fprintf('  New Size: %d bytes\n', newSize);
        fprintf('  Size Change: %d bytes (%.2f%%)\n\n', sizeChange, percentageChange);
    end
end

Read more about “whos” here:

https://www.mathworks.com/help/releases/R2023b/matlab/ref/whos.html

[Jan Kappen]

Thank you very much for that approach. Unfortunately, it looks like that does not work with handle class objects. Plus, I just had one variable in that mat file, a big class object that capsules all the data.

I've followed a similar approach but ultimately used getArrayFromByteStream, see Serializing/deserializing Matlab data - Undocumented Matlab. And I checked out the old version of my library in a second MatLab session and compared all properties step by step, skipping Dependent properties via reflection.

Root cause: I've split a data table (class table) into two class objects which should've used dependent properties, and an internal table to store the data. Turned out I forgot to make one block of properties transient/dependent to avoid saving them.

Afterwards, the mat file sizes were basicaly the same - quite interesting that there's no difference if the table is saved or a wrapping class around it - both can get compressed very efficiently, very nice Mathworks!

PS, just found out that mat files can be compared visually too: Compare and Merge MAT-Files - MATLAB & Simulink (mathworks.com) and that it can even "look" into objects, but not arbitrarily nested. But it could also be a good starting point: