Data structures and spmd

My code stores composite variables within data structures with a variable number of structure fields i.e.

Pool_Size = 3;
parpool(Pool_Size);
n=2;
for Cntr = 1:n
    Comp_Variable = Composite();
    for Pool_Cntr = 1:Pool_Size
        Comp_Variable{Pool_Cntr} = rand(2,2);
    end
    Data_Struct(Cntr).Comp_Variable = Comp_Variable;
end

I need to pass these variables into functions within spmd i.e. of the form (pseudo code):

spmd
    for Cntr = 1:n
        Result = My_Function(Data_Struct(Cntr).Comp_Variable)
    end
end

Obviously I can't directly index the structure fields due to transparency issues within spmd i.e.

spmd, Data_Struct(1).Comp_Variable, end

which will result in an invalid composite. The standard work around I have seen for other similar cases with structs or cell arrays is to specify new variables before entering the spmd data i.e.

Data1 = Data_Struct(1).Comp_Variable;
Data2 = Data_Struct(2).Comp_Variable;
spmd
    Data1
    Data2
end

where Data1, Data2 can now be passed into a function within the spmd without issue. However, if I have variable number of structure fields I can't think of good way to do this without resorting to the dreaded eval function to both define the new variable names and pass these to the function within spmd (in the later case the calls to eval would be done inside another function, again to adhere to the transparency rules for spmd). Does anybody have a better approach to accessing the composite vbariables within the spmd without having to define each one explicitely before the spmd block?

Any help would be greatly appreciated.

Answer 1

[Raag]

Hi D W,

As per my understanding the issue you're facing stems from MATLAB's transparency rules within 'spmd', which prevents direct indexing of nested Composite fields. Defining separate variables manually isn't scalable, and using eval is not ideal.

As a workaround, you can nest Composite objects inside another Composite to manage your data effectively. Here's a brief example:

% Create a single Composite to hold all data
AllData_Comp = Composite();
for w = 1:Pool_Size
    AllData_Comp{w} = {Data_Struct(1).Comp_Variable{w}, Data_Struct(2).Comp_Variable{w}};
end

The explanation of how it works is that by nesting, each worker has access to all necessary data without manual variable creation and ‘AllData_Comp’ is recognized within ‘spmd’ which avoids transparency errors.

For a better understanding of the above solution, refer to the following MATLAB documentations:

1. Composite Objects: https://www.mathworks.com/help/parallel-computing/composite.html

2. spmd: https://www.mathworks.com/help/parallel-computing/spmd.html

3. parpool: https://www.mathworks.com/help/parallel-computing/parpool.html