Why do I get different results with parfor and for loops?

Hi,

I am trying to run a specific code 15 times, each time for a new set of data so my iterations are not sequential. I use SPM toolbox and am trying to speed up the calculations using matlabpool and parfor. The results of parfor loop and for-loop are different even when I just load one dataset instead of 15 datasets i.e. parfor i=1:1. If I close matlabpool then the results are the same. My code looks something like the following. I have not included all the details of defining my structure. This code is exactly the same for for-loop.

I define some parameters in structure variable DCM. The name of the dataset that each parfor iteration should load is saved in DCM.xY.Dfile. Then, DCM is sent to spm_dcm_erp. Inside this function, the dataset whose name is saved in DCM.xY.Dfile is loaded and some parameters are calculated until the model converges which may take from 1 to 64 iterations. Then, some parameters will be added to DCM and at the end of spm_dcm_erp (and inside the function not in parfor loop) the updated DCM will be saved on the disk with the name defined in DCM.name. It is this saved DCM that I check for the two models (with for and parfor) and is different in the 2 conditions. * I even ran parfor for just one of my datasets (instead of i=1:15 I put i=1) and the result is still different from the one from for-loop. * The funny thing is when I run for-loop the model converges at iteration 62 but when I run parfor it converges at iteration 11. * I tried saving the workspace inside my spm_dcm_erp and the parameters of the two conditions (for and parfor) are exactly the same before the for-loop of spm_dcm_erp function starts to calculate the parameters of the model so I made sure that parfor does not change my variable in any way before sending it to the spm_dcm_erp function.

    clc
    clear all
    DCM=[];
    parfor i=1
       spm('defaults','EEG');
       DCM=[];
       % paths to data, etc.
       %-------------------------------------------------------
       Pbase     = pwd;        % directory with your data,
       Pdata     = fullfile(Pbase, '.'); % data directory in Pbase
       Panalysis = fullfile(Pbase, '.'); % analysis directory in Pbase
       % the data (mismatch negativity ERP SPM file from SPM-webpages)
       % -------------------------------------------------------------
       DCM.xY.Dfile = ['mansour_trials_' num2str(i)];
       DCM.name = ['DCM_mansour_trials_' num2str(i) '_10'];
       DCM      = spm_dcm_erp(DCM);
    end

[Jan]

It is only a wild guess, but whenever I see "pwd" in a multi-threaded context, I'm alarmed, that the current directory could change unexpectedly. So let me ask, if you are really sure, that the FOR and PARFOR methods operate on the same data?

I dare to ask such a trivial question, because the situation you explain sounds magic. And the experiences in the Matlab forums show, that the level of magic is in general reciprocally proportional to the complexity level of the underlying problem.

[Pegah Hosseini]

Dear Jan,

Although I just have one file with this name on my whole PC, I again checked the data loaded in my function and it is the same for the two conditions.

[Kevin Claytor]

1. Pegah could move the Pbase command outside of the loop, since it is only set once.
2. Try using a different value than the parfor index to access the data ie; 'i = 15; parfor j = 1 ....' vs 'i = 15; for j = 1 ...'
3. You mention that you update and save the DCM file - could this be the problem? Ie. you run it once with the for loop - it takes 64 iterations to converge, then saves the modified data. Parfor operates on this modified data and only takes 11 iterations to converge?

[Jan]

Fine, or what a pitty - depending on the point of view.

Btw, clear all is a brute waste of time. There is simply no reason to remove all loaded functions from the memory. Sometimes there might be resons for a clear variables, but I'm not convinced that this helps to write clean code. But this does not concern your problem.

[Pegah Hosseini]

Kevin,

1. Although, I get your point for not needing it to happen in all iterations, for the problem that I have right now, it does not have any effect. I have previously tried defining my DCM outside parfor loop and then inside parfor, I just called spm_dcm_erp function and the result was still the same.

2. I will try that and let you know of the results which I don't think will have any difference as I have already checked the elements of t he loaded data and know that the data that is being loaded in the two conditions is the same.

3. No, because the data is not changed. Just the DCM variable is changed which is then saved with a name other than the name of the data. And I load the data in the loop not DCM.

Jan,

Yes I know and I think it is a pitty right now.

[Pegah Hosseini]

I was in contact with SPM coders for a while and it seems that there might be a problem in the interaction between the two toolboxes (PCT and SPM). I will let you know if this is the case.

[Pegah Hosseini]

Although I don't think you are interested in this trend anymore, just so it has a final answer, I am updating it. There really was a problem in the interaction between PCT and SPM and it was solved in the new version. At least that is what I was told by the group. By the time the problem was solved, I had already moved on from SPM so I did not go back to check if it worked for me or it still had any problems.