rng('default') unreliable

I have some lengthy code (too long to post here) which executes some sampling based computations. At the beginning of the code, I set

rng('default')

to ensure reproducibility of results for debugging purposes (afaik, this means a Mersenne twister with seed 0). This is the only place in the code where rng is invoked. During program execution, some quantities are sampled using normrnd, which are subsequently used as inputs for the computations. The thing is, I've noticed on multiple occasions now that running the exact same code several times does NOT always yield the exact same results.

I always have one instance of Matlab running in GUI mode. Alternatively, I can also run the code in batch mode in the background, which I do most of the time. Although I change absolutely nothing in the code, the sample values occasionally (not always) differ from run to run. I can't figure out what's going on, but since my input does not change, it must have something to do with the rng. Has someone experienced the same problem? Could it be some kind of crossfire between GUI and batch mode? Or does rng('default') involve time or some other "random" variable?

Answer 1

[Jan]

rng('default') sets the random number generator to the initial state. This is a reliable procedure.

If your code replies different results for the random numbers, it must contain another source of randomness. If this effect occurs only sometimes, it does not seem to be a problem related to the value of the current time. This would be something like:

% Not likely to be the problem:
itime = now;
while now - itime < 5
    a = rand; 
end

Such dependencies on the time should be observable in all runs.

Another problem can be using different inputs relying ob the path "D:" : This is the last folder used on the disk D and can be influenced by external software. Brrr, such evil.

I assume, you have to check the code and search for the source of the problem. While the RNG is known not to create magical output, you either process different inputs (without knowing it) or there is another source of entropy.

[Jan]

[MOVED from sectio for answers:] broken_arrow wrote:

Hi Jan,

thanks for your answer. I'd really like to figure this one out. Here are my thoughts:

1. Looking at a multitude of results, it seems time can be ruled out as a factor (as you said, rng('default') should not involve time)
2. I do invoke some inputs from path "F:\some_folder\..." (which I dead certain never changed during my tests). My code is also stored there. "F:" is my last local drive (I also have a "D:" drive). Could that be part of the problem (i. e. have an influence on rng)?
3. Could it be that running a job in batch mode makes a difference (different rng behaviour)?
4. What other sources of entropy could there possibly be? Resetting rng and then drawing the same random numbers over and over should leave no room for entropy (at least not user induced). Or are there commands other than rng which introduce a random component?

I am certain the problem is not input related because I do nothing else than running a job, looking at the results, then running the exact same job again and looking at the results again. I could just accept it as randomness (the sample values always lie in the specified range), but unexpected behaviour is a red light for me, so I'd really like to know the cause.

[Jan]

@broken_arrow: Distinguish "D-Drive", "D:" and "D:\". The first is a drive letter, the last ist the path to the root folder on the disk D and "D:" is the last used folder on drive D, which is very volatile and therefore it should be avoided strictly. If you code contains a line like: cd('F:') you cannot be sure, where you get, while cd('F:\') is stable.

Relying on the current folder is a bad programming pattern, because a callback of a timer or GUI can change the current folder. So use absolute paths for all files in general.

But it is unlikely that your code creates a wrong output if the current folder is changed unexpectedly. A crash is more likely.

The most frequent cause of entropy is using orphaned pointers or uninitialized variables in C-Mex functions. Do you call C-mex functions?

In Matlab R2009a, the sum() command was implemented using multithreading above the limit of 88999 elements. Then the partial results were added in the temporal order. This could cause different results between repeated calls, see https://www.mathworks.com/support/bugreports/532399 . Maybe you are using another function, which has an equivalent bug.

It would be useful to narrow down the code, which causes the difference.

[Steven Lord]

Let's check a couple things.

First run this command then run your code and make sure that MATLAB only stops once, immediately before the rng('default') call in your code. If it stops more than once, look at how the code is calling rng. If it calls rng('shuffle') that would be your time dependence.

dbstop in rng

Once MATLAB stops in the rng call you can use the Continue button in the Editor toolstrip or the following command to continue execution.

dbcont

Second, when you say you run your code in batch mode do you mean you launch another session of MATLAB using the -batch startup option or do you use the batch function from Parallel Computing Toolbox? If the latter see this documentation page.

[broken_arrow]

Ok, here is what I got:

1. I use only absolute paths, (i. e. no "D:" or something)
2. I use no C-mex functions.
3. I do not use the sum function
4. Unfortunately I can't really narrow it down further because I can't pick the code apart.
5. I tried "dbstop in rng" (in desktop mode). rng is called only once (as described)
6. It appears to me the problem is related to batch execution. Running in desktop mode yields constant results so far. I use the -batch startup function with the following code (cmd /k is to keep command window open after execution):

cmd /k ^

matlab -batch -wait ""cd('F:\mydir')"; "myfunction(arg1,arg2,arg3)""

[broken_arrow]

I think I have the culprit: It seems to be the fieldnames function. In my code, I create a struct whose field names are then captured via fieldnames. Depending on some other parameters, each field is then assigned a vector of random numbers via the normrnd function using a for loop. The thing is, fieldnames does not always capture the field names in the same order (at least in batch mode, this behaviour is constantly observed). If the order changes, the fields are passed to the normrnd function in a different sequence, which of course has an influence on the returned values. Guess that's all there is to it. Maybe the fieldnames function should be modified to return the field names in alphabetical (or some other) order.

[Stephen23]

" Maybe the fieldnames function should be modified to return the field names in alphabetical (or some other) order."

Structure fields actually have an order (by default the order they were created in), therefore FIELDNAMES should return them in exactly that order. This is critical when using things like STRUCT2CELL and CELL2STRUCT together with the fieldnames.

You can always reorder the fields yourself:

https://www.mathworks.com/help/matlab/ref/orderfields.html

[Steven Lord]

Or you could sort the output of fieldnames before deciding which field gets which random vector if you don't need or want to reorder the struct array itself.

[broken_arrow]

That is one possibilitiy. It still seems strange that dynamic properties apparently are not created in the specified order (even in GUI mode) and in batch mode the order even appears to be random (c. f. below). Maybe that should be changed (just my opinion).

Answer 2

[broken_arrow]

Ok, it seems we have to go one step further. I think it's actually a problem with dynamicprops objects. Let me give a code example. Take this function file:

function fieldnametest

myobj = dynpropobj;

myobj.afield = "these";
myobj.bfield = "are";
myobj.cfield = "exemplary";
myobj.dfield = "attributes";

myfields = fieldnames(myobj);
disp(myfields)

end

with an additional classdef file:

classdef dynpropobj < dynamicprops
    
    methods
        function obj = dynpropobj % constructor
            obj.addprop('afield');
            obj.addprop('bfield');
            obj.addprop('cfield');
            obj.addprop('dfield');
        end
    end
end

If i run fieldnametest in GUI mode, the fields are (as observed so far) always created in the same sequence. But if I create a batch file

cmd /k ^

matlab -batch -wait ""cd('F:\testfolder')"; "fieldnametest""

(where cmd /k ^ is to keep cmd open after execution and 'F:\testfolder' is the directory where the aforementioned files are located) and run it, the field name order seems to get shuffled randomly (at least that is what I observe on my computer).