Delete only consecutive repeated string entries from a dataset in matlab

hi!

I am relatively new to matlab. I have a dataset having three columns, time, pitch and notation. for eg

 time   pitch notation
 2.5725 329.63 G
 2.5800 329.63 G
 2.5900 311.13 M
 2.5900 311.13 M
 2.6000 570.40 P

I want to remove duplicates occurring consecutively in the file such that the order remains the same. so the output will be:

 time   pitch notation
 2.5725 329.63 G

 2.5900 311.13 M

 2.6000 570.40 P

I am currently using matlab 7.9.0 so the first of unique command id not supported. Can anyone tell me how to go about it further.

[EDITED, table formatted, Jan]

[Jan]

Please follow the "? Help" link to learn how to format messages in the forum.

The description of the input is not clear. If you use valid Matlab syntax to define the value, it gets clear, if "2.5725" is a string or a double, if the dataset is a cell matrix or a struct, which contains the data as column vectors. Should blank lines appear in the output, or do you want these lines to be deleted? Do only the 2nd and 3rd column matter for the recognition of equal rows? Are you taling about neighboring rows only or does the order of rows not matter?

[Cedric]

Is the 'notation' field/column always one character long?

[avantika]

the notations are two letter words like SL, RM, GM,MM,PM,DM,nM ...ETC...

[avantika]

hi!

I am relatively new to matlab. I have a dataset having three columns, time, pitch and notation. for eg

 time   pitch notation
 2.5725 329.63 GM
 2.5800 329.63 GM
 2.5900 311.13 MM
 2.5900 311.13 MM
 2.6000 570.40 PM
 2.6725 329.63 GM
 2.6800 329.63 GM
 2.7900 311.13 MM
 2.8900 311.13 MM
 2.9000 570.40 PM
 2.9500 570.40 PM
 3.1000 570.40 PM
I want to remove repeated enteries occurring consecutively in the pitch and notation column in the file but the order of the dataset should remain the same. so the output will be:

 time   pitch notation
 2.5725 329.63 GM

 2.5900 311.13 MM

 2.6000 570.40 PM

 2.6725 329.63 GM

 2.7900 311.13 MM

 2.9000 570.40 PM

[Jan]

I still do not understand the empty lines and the type of the input is not explained here. Is this a text file, a cell containing strings, or or is pitch a field of a struct, which contains a double vector?

What should happen for:

 2.5725 329.63 GM
 2.5800 329.63 GM
 2.5725 329.63 GM
 2.5800 329.63 GM

So does "consecutively" mean the position in the array or should a line vanish, if there is any equal set of values before, even if other lines appear in between?

I assume that the solution is very easy, if you define the wanted procedure and the class of the input exactly.

Answer 1

[Andrei Bobrov]

    C  ={  'time'   'pitch' 'notation'
    2.5725 329.63 'GM'
    2.5800 329.63 'GM'
    2.5900 311.13 'MM'
    2.6000 570.40 'PM'
    2.6725 329.63 'GM'
    2.6800 329.63 'GM'
    2.7900 311.13 'MM'
    2.8900 311.13 'MM'
    2.9000 570.40 'PM'
    2.9500 570.40 'PM'
    3.1000 570.40 'PM'};
    d = cell2dataset(C); % your dataset - array

 [~,~,ii] = unique(d.notation);
 out = d([true;diff(ii)~=0],:);

ADD without dataset array

C  ={  'time'   'pitch' 'notation'
    2.5725 329.63 'GM'
    2.5800 329.63 'GM'
    2.5900 311.13 'MM'
    2.6000 570.40 'PM'
    2.6725 329.63 'GM'
    2.6800 329.63 'GM'
    2.7900 311.13 'MM'
    2.8900 311.13 'MM'
    2.9000 570.40 'PM'
    2.9500 570.40 'PM'
    3.1000 570.40 'PM'};

[ii,ii,ii] = unique(C(2:end,3));
out = C([true(2,1);diff(ii)~=0],:);

ADD 2

scale = {'GM';'PM'};
[~,ii] = ismember(C(2:end,3),scale);
i1 = ii > 0;
C1 = C(i1,:);
out = C1([true(2,1);diff(ii(i1))~=0],:);

[avantika]

Hi!

The cell2dataset command is not functioning with matlab v 2009b and even matlab version 2011a. It gives the following error: ??? Undefined function or method 'cell2dataset' for input arguments of type 'cell'.

[Andrei Bobrov]

see block ADD in my answer without dateset array

[avantika]

hi! thanks it worked...but the only the first repeated entry is not removed by this code.

[avantika]

Hi! I need to do one more thing in the 'out' dataset . I want to select specific rows based on a criteria for column 3 . For instance from the dataset 'C' only those rows are selected that have GM and PM. I tried using the following commands but i am getting and error...............

%select the scale of raga

A = out

scale = ['GM';'PM']

B = scale

% select only those values of pitch that are in the specified scale.

A2inB1 = ismember(A(:,3),B(:,1));

S = A( A2inB1 , :);

however i get the following error message:

??? Error using ==> dataset.subsref at 82 Dataset array subscripts must be two-dimensional.

Error in ==> ismember at 74 tf(i) = any(a(i)==s(:)); % ANY returns logical.

[Andrei Bobrov]

see block ADD2

[avantika]

HI

I get an error when i use the commands given by you for ismember:

scale = {'NL';'NM';'NU';'RL';'RM';'RU';'GL';'GM';'GU';'mL';'mM';'mU';'DL';'DM';'DU';'SL';'SM';'SU';'PL';'PM';'PU';};

>> [~,ii] = ismember(C(2:end,3),scale);

i1 = ii > 0;

C1 = C(i1,:);

out2 = C1([true(2,1);diff(ii(i1))~=0],:);

??? Error using ==> dataset.subsref at 82 Dataset array subscripts must be two-dimensional.

Error in ==> ismember at 78 found = find(a(i)==s(:)); % FIND returns indices for LOC.

Answer 2

[Simon]

Hi!

I don't understand why the second entry is removed. Is a "duplicate" defined if all three columns match or only the second and third?

Does the unique command in 7.9 support rows? Like

b = unique(A, 'rows')

Does your data set consist of single lines for each entry? You could try to put each data set as a string in a cell array and use unique of the cell array.

[avantika]

hello sir,

thanks for replying. Yes the dataset consists of single lines of each entry. The first column of dataset is time stamp, second column is pitch values and the third column is of notation for corresponding pitch values.When I use the following unique command to delete the duplicate pitch values it firsts sorts the data in ascending order and deletes the repeating rows

[~,idx]=unique(ds3(:,2));

withoutduplicates=ds3(idx,:)

However i dont want the data the data to be sorted first , i want the command to read it line by line and keep deleting the duplicating lines in accordance with replicated pitch values.

For this matlab does have 'first' command with unique but that is not supported with matlab vs 2009 as it shows the following error: ??? Error using ==> getvarindices at 25 Unrecognized variable name 'first'.

Error in ==> dataset.unique at 34 vars = getvarindices(a,vars,false);.

Is there any solution for this , can i delete rows without sorting in matlab 2009.

[Simon]

Hi!

Try it with a cell array of strings, as proposed.

The unique command has additional return values that contain the relation between the sorted output and the input. Check the documentation for more information.

[avantika]

hi!

i checked the class of dataset before the conversion to cell array of strings .

class(ds3.pitch)

ans =

double

class(ds3.time)

ans =

double

class(ds3.notation)

ans =

cell

I am not able to convert it to cell array of strings.