Combining 2 Matrixes to Run in Large Data Set

I have already ran a successful code that starts and stops at a specific time from a very large data set, with each comma separating the time by day. (i.e. the day 1 data runs from 48009 seconds to 66799 seconds, day 2 from 52467 s to 61999, etc.) Shown below:

utcStarts = [48009, 52467, 54128, 54591, 45001, 55587, 56714];

utcEnds = [66799, 61999, 59999, 57999, 45002, 74999, 61499];

I am attempting to run the same code but with two time periods during each day, like this below:

utcStarts = [48009, 52467, 54128, 54591, 45001, 55587, 56714]; utcEnds = [66799, 61999, 59999, 57999, 45002, 74999, 61499];

utcStarts = [730001, 72001, 74001, 80001, 45003, 82001, 68001]; utcEnds = [77659, 80271, 80894, 86181, 74428, 87252, 78116];

i.e. where day 1 will run from 48009 s to 66799 s and again from 73001 to 77659 s, day 2 from 52467 s to 61999 s and again from 72001 s to 80271 s.

Now that I am trying to run the same code for two time periods, I am getting an error which says: "The value assigned to the variable 'utcStarts' and 'utcEnds' might be unused."

What function can I use to combine the two matrixes and run the code for the two different time periods for each day?

Thank you.

[goyanks1989]

This is the entire code if that provides any more help..

clear
clc
% first we load the file of interest
% contains 496-12 columns ( 484)

% directory where the data is
main_dir = '/Users/alexisekim/Desktop/Terry_Stuff/Luo_Research/Cloud_Data/';

seadays = [20130812, 20130821, 20130823, 20130902, 20130904, 20130911, 20130918];

% start and end utc time (seconds) for area of interest for each day in
% order

% NonConvective Hours
utcStarts = [48009, 52467, 54128, 54591, 45001, 55587, 56714];
utcEnds = [66799, 61999, 59999, 57999, 45002, 74999, 61499];

%NonConvective Hours
utcStarts = [730001, 72001, 74001, 80001, 45003, 82001, 68001];
utcEnds = [77659, 80271, 80894, 86181, 74428, 87252, 78116];

%preloading a really large array to store data
bigDATA = nan(16, 100000);
bigGPS = bigDATA;

bigDATAstart = 1;
bigGPSstart = 1;

for jj = 1:7 % each day...

      if jj== 5 % not interested in this day anymore
          continue 
      end

      disp(jj)
      filename = [main_dir, 'SEAC4RS-mrg01-dc8_merge_', num2str(seadays(jj)), '_R3.ict'];

      delimiterIn = ',';
      headerlinesIn = 526;

      data = importdata(filename,delimiterIn, headerlinesIn);

      utcTime = data.data(:,1);
      zGPS = data.data(:,8);

      %%the value of ii will be the corresponding column number of the var of interest
      %aerosol ext (dry)... OA... SO4... NO3... NH4... aerosol volume... etc

      % making the values less than 0 be NaN
      zGPS(zGPS<0) = NaN;
      goodPoints = find((utcTime >= utcStarts(jj)) &( utcTime <utcEnds(jj)));
      meaning_of_var = {'Aerosol Extinction (dry)', 'OA(ug/m^3)', 'SO4 (ug/m^3)', 'NO3 (ug/m^3) ', 'NH4 (ug/m^3)', ...
          'Aerosol Volume', 'CO', 'Isoprene (ppbv)', 'RH', 'Black Carbon Mass (nm)', ...
          'SO2', 'CO2', 'CH2O (ppbv)', 'O3', 'lat', 'lon', 'Sample Volume (L)', 'Particle Conc. (#/L)', 'Total Count(#)', ...
          'Water Soluble Organic Carbon (ugC/m^3)'};

      %these are the column numbers for the corresponding variables above

      ii_array = [198 139 81 80 84 238 63 353 43 159 92 62 66 73 6 7 238 239 246 156];

      for ii = 1:length(ii_array)

          a = data.data(:,ii_array(ii));
          a (a<0) = NaN;

          bigDATA(ii, bigDATAstart:(bigDATAstart - 1 +length(goodPoints))) = a(goodPoints);
          bigGPS(ii, bigGPSstart:(bigGPSstart - 1 +length(goodPoints))) = zGPS(goodPoints);

          % this section will plot the figures for the individual days 

          %         figure
          %
          %
          %         boxplot(a(goodPoints), ceil(zGPS(goodPoints)), 'orientation', 'horizontal')
          %         ylabel('Altitude (km)')
          %         xlabel(meaning_of_var(ii))
          %         title(meaning_of_var(ii))
          %

      end %ii

      bigDATAstart = bigDATAstart + length(goodPoints);
      bigGPSstart = bigGPSstart + length(goodPoints);

end %%jj

% adjusting the final size of the really large array to what it actually is
finalDATA = bigDATA(:,1:bigDATAstart-1);
finalGPS = bigGPS(:, 1:bigGPSstart-1);

%%--------------------------------------------------------------------
% This section plots cumulatively all of the data collected for all the
% days sampled. 
%%--------------------------------------------------------------------

% kk = is the 'meaning of var' in increasing order
for kk =  1:length(ii_array)

      %these variables were skipped and not plotted (RH, CO2, Lat, Lon)
      if( (kk ==9)|| (kk==12) || (kk==15) || (kk==16) )
          continue
      end

      boxplot(finalDATA(kk,:), ceil(finalGPS(kk,:)), 'orientation', 'horizontal')

      %adjustments that to remove noisey plots
      if kk == 1
          axis ([0 300 0 14])
      elseif kk == 2
          axis ([0 25 0 14])
      elseif kk == 4
          axis([0 1 0 14])
      elseif kk == 7
          axis ([0 250 0 14])
      elseif kk == 8
          axis([0 5 0 14])
      end

      ylabel('Altitude (km)')
      xlabel(meaning_of_var(kk))
      title(meaning_of_var(kk))

      %file name will be uniquely named based off kk which has a meaning 
      %(see meaning of var for the meaning of each)
      fileName = ['sample_ALL_',num2str(kk),'.jpeg'];
      print ('-djpeg99', fileName)
      close
  %     

end %%kk

Answer 1

[John BG]

Does it make sense to

1.- assign clearly differentiated names to both time windows of interest. Instead of calling them both utcStart and utcEnd, let's call

utcStarts1 = [48009, 52467, 54128, 54591, 45001, 55587, 56714]; 
utcEnds1 = [66799, 61999, 59999, 57999, 45002, 74999, 61499];

and

utcStarts2 = [730001, 72001, 74001, 80001, 45003, 82001, 68001]; 
utcEnds2 = [77659, 80271, 80894, 86181, 74428, 87252, 78116];

2.- now replace

 goodPoints = find((utcTime >= utcStarts(jj)) &( utcTime <utcEnds(jj)));

with

goodPoints1=[utcStarts1(jj):1:utcEnds1(jj)]
goodPoints2=[utcStart2(jj):1:utcEnds2(jj)]
goodPoints=[goodPoints1 goodPoints2]

now goodPoints contains both time windows of interest.

What you may want to do next, is to add some code to make sure that when both windows overlapping, or 2nd window earlier than the 1st one, some correction is applied so that goodPoints contains all the increasing time points of interest.

Hope it helps

regards

John

[e-mail]

[goyanks1989]

John, thank you. I am very new to this as you can probably tell. When I make the above changes, I now get an error saying "Index exceeds matrix dimensions" at line 74

bigDATA(ii, bigDATAstart:(bigDATAstart - 1 +length(goodPoints))) = a(goodPoints);

What should I do to the code to fix this error? Thanks again for your help.

[John BG]

only comment: the error message means that matrix dimensions do not match. Either you have more good points that a positions has, or once you have selected a(goodPoints) you are trying to put too many or too few of these points into bigDATA(ii, ..) In previous similar problems, the best thing to solve this error is to

1.- find out in advance the maximum possible value that arrays you will use in equations like line 74 may reach

2.- either start with an all zeros array with max size, that you gradually fill up, or create additional variables that just take the fields that will match a receiving matrix because it has exactly same dimensions, all dimensions should match on both sides of the =.

Answer 2

[Chad Greene]

You define utcStarts and utcEnds twice, and the second time overwrites the first time. That's why Matlab says utcStarts goes unused. I have not tried to parse your entire script so depending on how your code is set up, it may be as easy as defining an A and a B start and end time, then concatenate the A and B times:

% NonConvective Hours (A run)
utcStartsA = [48009, 52467, 54128, 54591, 45001, 55587, 56714];
utcEndsA = [66799, 61999, 59999, 57999, 45002, 74999, 61499];

%NonConvective Hours (B run) 
utcStartsB = [730001, 72001, 74001, 80001, 45003, 82001, 68001];
utcEndsB = [77659, 80271, 80894, 86181, 74428, 87252, 78116];

% Concatenate: 
utcStarts = [utcStartsA utcStartsB]; 
utcEnds = [utcEndsA utcEndsB];