Indexing large dataset in for loop and renaming

I'm trying to index a large data set by unique Identifier, rename as that identifier, and save file in a loop. I've done this before. It has worked before. I can't for the life of me get it to work now on this one certain file format and I don't understand why!!!!! Well... I kinda do but again it has worked in the past.

nn = ('Chptnk');
tgtt.Lat = Lat;
tgtt.Lon = Lon;
tgtt.vid = vid;
tgtt.T = T;
vesid = unique(tgtt.vid); % find ids
for i = 1:length(vesid)
    vesids {i} = vesid(i);
    ind = find(vid == vesids{i});
    eval(sprintf('%s_%d(:,1) = tgtt.vid(ind);', nn, vesids{i}))
    eval(sprintf('%s_%d(:,2) = tgtt.T(ind);', nn, vesids{i}))
    eval(sprintf('%s_%d(:,3) = tgtt.Lat(ind);', nn, vesids{i}))
    eval(sprintf('%s_%d(:,4) = tgtt.Lon(ind);', nn, vesids{i}))
    eval(sprintf('x = %s_%d;', nn, vesids{i}))
    dlmwrite(sprintf('%s_%d.csv', nn, vesids{i}), x)
end

This code works for my other formats. The catch is vid is pure numeric.

This other file format the vid is a mix of letters and numbers... example: g565d62d, a55d666g, 66f584fv2, add5544, HY55842.... etc

I've done this code with mixed numbers and letters before just fine(I just changed the %d to %s). but just won't work now for this current format.

I changed up the ind..

   ind = find(ismember(tgtt.vid,vesids{i}));

Got it to index but now it tells me :

Error using sprintf
Function is not defined for 'cell' inputs.

I tried converting to double as a hail mary but get nans. I tried converting to table and altering code to index table in loop but can't get it to index. But then I was probably not writing that code right either.

in the past I've gotten this code to work on water quality data by do this at the start of code:

vars    = {'NH4','DO','SAL','NO23','PO4','DSI','CHLA','WTEMP'};

It's a mix of numbers and letters too! worked fine! This new data has over 300 unique names! I tried to replicate vars by writing this:

vesids {i} = vesid(i);

looks the same to me?!

and so here is where I'm stuck......

nn = ('Chptnk');
tgtt.Lat = Lat;
tgtt.Lon = Lon;
tgtt.vid = vid;
tgtt.T = T;
vesid = unique(tgtt.vid); % find ids
for i = 1:length(vesid)
    vesids {i} = vesid(i);
    ind = find(ismember(tgtt.vid, vesids{i}));
    eval(sprintf('%s_%s(:,1) = tgtt.vid(ind);', nn, vesids{i}))
    eval(sprintf('%s_%s(:,2) = tgtt.T(ind);', nn, vesids{i}))
    eval(sprintf('%s_%s(:,3) = tgtt.Lat(ind);', nn, vesids{i}))
    eval(sprintf('%s_%s(:,4) = tgtt.Lon(ind);', nn, vesids{i}))
    eval(sprintf('x = %s_%s;', nn, vesids{i}))
    dlmwrite(sprintf('%s_%s.csv', nn, vesids{i}), x)
end

[Jan]

Bad. What a bad idea to create the variables dynamically by eval. See http://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval .

[Stephen23]

@Jenn: what do you need the structure for? It seems to be totally superfluous in this code that you have shown us: why not just access the variables Lat, etc, directly, rather than making copies of all of your data?

Your code is quite inefficient with memory: you end up with four copies of the data in four separate variables (original variable, structure tgtt, magically named variable, and temporarily in matrix x), but there seems to be no advantage (that I can see) to replicating your data like this. It is recommended to keep data together as much as possible (because this makes processing it simpler), and that the number of copies of data should be minimized (to reduce memory consumption).

Also find is less efficient than logical indexing.

[Jenn]

the tgtt was from earlier code versions that just never got taken out. I know better than to do that now but I was getting desperate and diving into really old code.

[Jenn]

and thanks for the link! wish I had come across it earlier. The eval method is how I was taught(by an old boss). I've been trying to get away from the old ways and have for the most part. I have a bad habit of going back when I get stuck.

[Jan]

It would not be polite to visit your old boss and hit him.

Whenever you get stuck and try to eval, visit the forum and ask the community. We love to show you better ways. Sometimes this demands for a restructuring of the code, but it is always worth to do so.

[Jenn]

I promise to never use eval again. or hit my old boss... ;)

Answer 1

[Stephen23]

This is a classic example of how eval makes code complex and buggy, and then makes it hard to debug.

When beginners (and self-taught bosses) decide to use eval they force themselves into writing slow, complex, buggy code, and from that point on they are struck with it because there is no going back:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

When I check the code all you are actually doing is looping over the unique ID values, picking the corresponding data from the variables Lat, Lon, vid and T, concatenating this data together, and then saving it to a file. And that description in words is basically all that is required in MATLAB too! Here is much simpler and more efficient code, that does not create multiple copies of the data, and does not rely on slow and buggy eval:

pfx = 'Chptnk';
idu = unique(vid);
for k = 1:numel(idu)
    tmp = idu{k};
    idx = strcmp(tmp,vid);
    fnm = sprintf('%s_%s.csv', pfx, tmp);
    dlmwrite(fnm, [vid(idx),T(idx),Lat(idx),Lon(idx)])
end

[Jenn]

Thanks for the Link! I feel silly now. I should of known better. however I'm still having the problem of sprintf and 'cell' inputs.

Code works (and much nicer btw) for the files that Eval worked on but still gives me the 'cell' error for the new 'Mixed numbers and letters' file. I'm researching answer now. Hopefully I can figure it out now that I'm away from EVAL.

[Jan]

+1: Thanks, Stephen, for saving Jenn out of the eval hell.

[Stephen23]

@Jenn: I edited my answer to assume that vid is a cell array of char vectors. In particular you will need this line at the start of the loop:

tmp = idu{k}; % note the curly braces!

PS: It would be entirely appropriate to visit your old boss and tell them to learn how to use MATLAB. You could show then how!

[Jenn]

Thank you for showing me the way! those curly braces get me every time. as does spelling, capitalization, and formatting. While I'm very thankful to old boss for getting me started in Matlab, would love to visit and tell him to stop teaching people to use eval! Gonna bookmark that link. My coding gets better every day thanks to you guys in Matlab answers.