Create table in loop

Hello,

How can I create a table so that it grows with each iteration of a loop? I am trying to display all of the results from the loop but only manage to output the last.

Thank you for your time and help in advance!

idx = nchoosek(1:size(M), 3);  %matrix of index trios
    for k = 1:size(idx,1) 
        ix = idx(k ,:); % current idx trio
        Lx = M(ix,1); 
        Ly = M(ix,2);  
        gap = M(ix,3); 
        L = sqrt((Lx.^2 + Ly.^2)/2); 
        lsq = lsqnonlin(@(phi) errFun(phi, L, gap), [1.5; 18; 1]);
        a = lsq(1);
    end
    table(ix(:)', Lx', Ly', a)

[Adam Danz]

The first line produces an error. One of the following might be what you're looking for

• idx = nchoosek(1:numel(M), 3); where numel(M) >=3
• idx = nchoosek(1:size(M,1), 3); where size(M,1) >=3
• idx = nchoosek(1:size(M,2), 3); where size(M,2) >=3

It also appears that ix, Lx, and Ly are vectors with more than 1 element but a only has 1 element. Is that correct?

[Valeria Villegas-Medina]

Ah, thank you! Yes, M is an nx3 matrix and I'm tring to get all possible combinations of groups of 3 for the first column (n). For example, n = 33 so I'd like 33 choose 3 = 5456 iterations of the for loop. The final table I'd like to create would be 5456 x 10. Thanks for your prompt reply!

[Adam Danz]

The first demo in my answer should be what you're looking for.

Please concider accepting the answer (blue accept button) unless you have further questions on this topic.

[Valeria Villegas-Medina]

I've still got bugs but I think that's from a different function that is called earlier. Thank you for such a thorough response!

[Adam Danz]

If the errors pertain to your implementation of any of the solutions, I can probably help out.

[Valeria Villegas-Medina]

Wonderful, thanks! Here's my implementation. I think the trouble is with the initial values in the lsqnonlin function however, didn't have the problem before trying to implement the table solution so I'm wondering where the actual problem lies.

function results = lsqFun(M)
    
    idx = nchoosek(1:size(M,1), 3);  %matrix of index trios
    Lx = nan(size(M,1), 3);
    Ly = nan(size(M,1), 3);
    a = nan(size(M,1),1);
    for k = 1:size(idx,1) 
        ix = idx(k ,:); % current idx trio, row vector
        Lx(k,:) = M(ix,1);  
        Ly(k,:) = M(ix,2);
        gap(k,:) = M(ix,3);  
        L = sqrt((Lx.^2 + Ly.^2)/2); 
      
        lsq = lsqnonlin(@(coeff) errFun(coeff, L, gap), [1; 100; 5]);
        a(:) = lsq(1);
    end
        % Creating table
        table(Lx,Ly,a,'VariableNamesd', {'Lx', 'Ly', 'Prediction'}); 
end

function fErr = errFun(coeff, xdata, ydata) %phi is matrix holding parameter values i.e. what will be used in the onjective fn
    
    %parameters
    a = coeff(1);
    b = coeff(2);
    c = coeff(3);
    
    % calculate prediction from model
    yPredict = a + b*exp(-xdata.*c);
    fErr = ydata - yPredict;
   
end

[Adam Danz]

Are you getting an error message?

What is M and how I can recreate the problem?

[Valeria Villegas-Medina]

Yes, I get the following errors:

Error using snls (line 47)
Objective function is returning undefined values at initial point. lsqnonlin cannot continue.

Error in lsqncommon (line 164)
            snls(funfcn,xC,lb,ub,flags.verbosity,options,defaultopt,initVals.F,initVals.J,caller, ...

Error in lsqnonlin (line 253)
   lsqncommon(funfcn,xCurrent,lb,ub,options,defaultopt,allDefaultOpts,caller,...

Error in idxMat (line 14)
        lsq = lsqnonlin(@(coeff) errFun(coeff, L, gap), [10; 2; .5]);

M is a Nx3 matrix with x and y values for columns 1 and 2 and the response variable in column 3.

[Adam Danz]

Have you looked at the variable values and do they make sense?

I suggest putting a break point on the line that contains lsqnonlin and run your code in debug mode so you can see what the input variables look like on each iteration [instructions].

For example, on the first iteration, the loop variables Lx, Ly, gap, and L, are all NaNs except for the first row. Why would that be a good input to lsqnonlin?

Did you mean to call lsqnonline after the loop instead of within the loop?

[Valeria Villegas-Medina]

I changed Lx, Ly, and a to be initialized with ones() instead but still get the error. Lsqnonlin needs to be called in the look I think becaue it needs to run for every iteration. Thanks for the debugging tutorial!

[Adam Danz]

At a conceptual level, what's the reason behind fitting a nonlinear curve to matrices full of 1s (or 0s)? Anyway, the error is unrelated to the solutions in my answer so if you continue to have questions with lsqnonlin, I think the first step it to think through it at a conceptual level to figure out what you're supposed to be fitting.

Answer 1

[Adam Danz]

1. Store the values within the loop and build the table later

Unless you're already working with table variables, it probably easier to store the values in each iteration and then build the table later. More importantly, this method is much faster and efficient than the other methods below (see last section on speed comparison).

In these demos, A and B are row vectors and C is a scalar.

rng('default') % for reproducibility
% Pre-allocate loop vars
A = nan(5,4); 
B = nan(5,3); 
C = nan(5,1); 
% Create vars within loop
for i = 1:5
    A(i,:) = rand(1,4); 
    B(i,:) = rand(1,3); 
    C(i) = rand(1); 
end
% Build table
T = table(A,B,C,'VariableNames',{'BG','RO','GR'});

Result:

T =
  5×3 table
                        BG                                       RO                      GR   
    ___________________________________________    _______________________________    ________
    0.81472     0.90579     0.12699     0.91338    0.63236     0.09754      0.2785     0.54688
    0.95751     0.96489     0.15761     0.97059    0.95717     0.48538     0.80028     0.14189
    0.42176     0.91574     0.79221     0.95949    0.65574    0.035712     0.84913     0.93399
    0.67874     0.75774     0.74313     0.39223    0.65548     0.17119     0.70605    0.031833
    0.27692    0.046171    0.097132     0.82346    0.69483      0.3171     0.95022    0.034446

2. Build table within a loop - with preallocation

If you'd rather create the table within the loop and you know the size of the table and variable ahead of time, preallocate the table before the loop and then fill in each row and column using table indexing. Incidentally, preallocation does not save time (comparing this demo and the next one).

rng('default') % for reproducibility
T = table(zeros(5,4),zeros(5,3),zeros(5,1),'VariableNames', {'BG','RO','GR'});
for i = 1:5
    T.BG(i,:) = rand(1,4); 
    T.RO(i,:) = rand(1,3); 
    T.GR(i) = rand(1); 
end

Result:

T =
  5×3 table
                        BG                                       RO                      GR   
    ___________________________________________    _______________________________    ________
    0.81472     0.90579     0.12699     0.91338    0.63236     0.09754      0.2785     0.54688
    0.95751     0.96489     0.15761     0.97059    0.95717     0.48538     0.80028     0.14189
    0.42176     0.91574     0.79221     0.95949    0.65574    0.035712     0.84913     0.93399
    0.67874     0.75774     0.74313     0.39223    0.65548     0.17119     0.70605    0.031833
    0.27692    0.046171    0.097132     0.82346    0.69483      0.3171     0.95022    0.034446

3. Build table within a loop - without preallocation

If you don't know the size of the table and variables ahead of time, you can build the table within the loop by vertical concatenation. The sizes within each variable must still be consistent, though.

rng('default') % for reproducibility
T = table(); 
tempTable = table(); 
for i = 1:5
    tempTable.BG = rand(1,4); 
    tempTable.RO = rand(1,3); 
    tempTable.GR = rand(1); 
    T = [T;tempTable];
end

Result:

T =
  5×3 table
                        BG                                       RO                      GR   
    ___________________________________________    _______________________________    ________
    0.81472     0.90579     0.12699     0.91338    0.63236     0.09754      0.2785     0.54688
    0.95751     0.96489     0.15761     0.97059    0.95717     0.48538     0.80028     0.14189
    0.42176     0.91574     0.79221     0.95949    0.65574    0.035712     0.84913     0.93399
    0.67874     0.75774     0.74313     0.39223    0.65548     0.17119     0.70605    0.031833
    0.27692    0.046171    0.097132     0.82346    0.69483      0.3171     0.95022    0.034446

Efficiency and speed

The fastest and most efficient method in this answer is the first one, by far!

Each method was timed 500 times using tic/toc and the median durations were compared.

• 1 was 174 times faster than 2
• 1 was 142 times faster than 3
• 3 was 1.15 times faster than 2 (I checked this twice, #3 was faster than #2 by factors of 1.1 and 1.2)

I haven't looked into my table preallocation is slower than without it but others have reported this as well (example).