problem to concatenate table due to "cell and non cell" error

Hello,

I have a script that reads 2 TXT files to two tables (one is significantly larger than the other). then I add both table the columb 'detector' to the 4th column (both tables structure is identicle, and supposed to be the same types of variable to all columns). for some reason I'm getting this error : "Cannot concatenate the table variable 'detector' because it is a cell in one table and a non-cell in another."

the script works fine with smaller files.

is matlab chaneging the type of the variable due to the size?

this is the script:

Ge_table = readtable(hpge_file);  %
Ge_table.Properties.VariableNames = {'time_stamp', 'energy','time_adjusted'};

scint_table = readtable(pmt_file); %
scint_table.Properties.VariableNames = {'time_stamp', 'energy','time_adjusted'};

Ge_table(:,4) = {1}; 
Ge_table.Properties.VariableNames{4} = 'detector';
scint_table(:,4) = {0};      
scint_table.Properties.VariableNames{4} = 'detector';
coince_table = vertcat(Ge_table, scint_table);

Thanks!

[Walter Roberson]

Change

Ge_table(:,4) = {1}; 
Ge_table.Properties.VariableNames{4} = 'detector';

to

Ge_table.detector = ones(height(Ge_table),1);

[Adam Danz]

What's preventing the first version from working, though?

T = array2table(rand(5,4)); 
T(:,4) = {1}

[Walter Roberson]

I hypothesize that one of the tables already had four columns.

[Adam Danz]

Hmmm.... but the variableNames being assigned only contain 3 values which would throw an error.

Perhaps we aren't seeing the full code.

Answer 1

[Adam Danz]

One possibility is that the cell / non-cell mismatch is being caused by a different column in the table.

Here are two ways you can quickly see if there's any mismatches in variable types between two columns.

1) look at the first few rows of each table.

head(Ge_table)
head(scint_table)

If you're unsure what to look for, leave a comment that contains the content of those two outputs.

2) Look at the variable classes for each column, for each table. Do they match?

GeTblClasses = varfun(@class,Ge_table,'OutputFormat','cell'); 
scintTblClasses = varfun(@class,scint_table,'OutputFormat','cell'); 

[sani]

Hi, thanks a lot for your help!

they didn't match, but I'm not sure I understand why...

this is the output:

time_stamp    energy    time_adjusted    detector
    __________    ______    _____________    ________

    6.4543e+05     1896      6.4543e+05         1    
    6.5561e+05     1000      6.5561e+05         1    
    1.9021e+06      781      1.9021e+06         1    
    2.4798e+06      787      2.4798e+06         1    
    2.4892e+06      968      2.4892e+06         1    
    2.5901e+06     8777      2.5901e+06         1    
    3.7164e+06    17891      3.7164e+06         1    
    4.6626e+06      505      4.6626e+06         1    
    
ans =

  8×4 table

    time_stamp    energy    time_adjusted    detector
    __________    ______    _____________    ________

          6440    11426            6440        [0]   
         16148     5700           16148        [0]   
         43172     7680           43172        [0]   
         54932     3438           54932        [0]   
         68828    12397           68828        [0]   
         70160     6929           70160        [0]   
    1.1972e+05     6800      1.1972e+05        [0]   
    1.3287e+05     9433      1.3287e+05        [0]   

In my code I determined Ge_table. detector and scint_table.detector exactly the same way, that the only difference is that the first is a column of 1 and the other is a column of 0.

any idea how to fix it?

thanks!

[Adam Danz]

I'm stumped. Could you attach the two files?

What release of Matlab are you using?

[sani]

Im using 2018b, the files are 25 Gb so it will be a bit problematic to attach here

[sani]

I could attached smaller files, but I don't have this issue with smaller files

[Adam Danz]

So the file size is the problem? That's puzzling.

If you could find the minimum file size that reproduces the problem I could poke around. Just for make sure, you did provide us with the full code that produces this problem, right?

[Adam Danz]

Could you share the first few rows of the tables as they appear immediately after they are read in?

[Walter Roberson]

I could attached smaller files, but I don't have this issue with smaller files

I would tend to suspect that the large file has a field that is not the datatype it should be. It would be worth running a bit of verification code testing the file format

[sani]

I ran similar files of about 6 gb and didn't ran into this problem, and I belive uploading files in that size can be problematic. my table have 1065561261x3 rows and columns and I cannot preview it, nor open the original text file (or that I don't know how, if you know a way I'd love to share the data)

BUT- the original data is the 2 left column I showed previously (I'll post it here again), what I put here is not the entire code, the entire code is (the bold line is where the error appears):

Ge_table = readtable(hpge_file);  %
Ge_table.Properties.VariableNames = {'time_stamp', 'energy','time_adjusted'};

scint_table = readtable(pmt_file); %
scint_table.Properties.VariableNames = {'time_stamp', 'energy','time_adjusted'};

Ge_table(Ge_table.energy < 10, :) = [];      
% Ge_table(Ge_table.energy > 16384, :) = [];
% Ge_table(:,4) = [];
Ge_table(:,4) = {1}; 
Ge_table.Properties.VariableNames{4} = 'detector';

scint_table(scint_table.energy < 10, :) = [];     
% scint_table(scint_table.energy > 16384, :) = [];
% scint_table(:,4) = [];
scint_table.Properties.VariableNames{4} = 'detector';
scint_table(:,4) = {'0'};       
% scint_table.Properties.VariableNames{4} = 'detector';
coince_table = vertcat(Ge_table, scint_table);          %MARKED
coince_table = sortrows(coince_table, 1);
coin_delta = diff(coince_table.time_stamp);

the 4th column is added by me- it isn't part of the original data, the purpose of this column is to flat the original data source,for post analysis.

the problam apears to be in the largest file between the two, so my suspition are the same as walter.

Walter, can you please explain a bit more about the verification code?

isn't there maybe a way to inforce that the column type will be int or float?

thanks a lot!

[sani]

just ignore the 2 right columns on each table

time_stamp energy time_adjusted detector

__________ ______ _____________ ________

6.4543e+05 1896 6.4543e+05 1

6.5561e+05 1000 6.5561e+05 1

1.9021e+06 781 1.9021e+06 1

2.4798e+06 787 2.4798e+06 1

2.4892e+06 968 2.4892e+06 1

2.5901e+06 8777 2.5901e+06 1

3.7164e+06 17891 3.7164e+06 1

4.6626e+06 505 4.6626e+06 1

ans =

8×4 table

time_stamp energy time_adjusted detector

__________ ______ _____________ ________

6440 11426 6440 [0]

16148 5700 16148 [0]

43172 7680 43172 [0]

54932 3438 54932 [0]

68828 12397 68828 [0]

70160 6929 70160 [0]

1.1972e+05 6800 1.1972e+05 [0]

1.3287e+05 9433 1.3287e+05 [0]

[Walter Roberson]

maxerr = 10;
fid = fopen(FILENAME, 'rt');
linenum = 0;
errcount = 0;
while ~feof(fid)
    thisline = fgetl(fid);
    if ~ischar(thisline); break; end
    linenum = linenum + 1;
    fields = sscanf(thisline, '%s');
    nf = length(fields);
    if nf ~= 3
        fprintf('Line #%d has %d fields\n', linenum, nf);
        errcount = errcount + 1;
        if errcount >= maxerr
            fprintf('giving up at %d errors\n', maxerr);
        end
    end
end
fclose(fid);

[sani]

it is running now (it seems to take couple of hours at least)

what should I excpect? (I'm new to matlab- didn't figure it out myself, sorry for that)

[Walter Roberson]

If every line has 3 fields, then it will just eventually stop running.

Otherwise, it will report up to the first 10 rows that have a different number of fields. You can adjust the number it reports with maxerr. I put in a limit so that it didn't start dumping out tens of thousands of rows (especially if my code did not match the file!)

[sani]

thanks for the explenation!

I don't think this will help in my case, since I'm able to load the files to tables, and even to add the 3rd column.

for some reason that I don't understand it gives different types for the same operation for each data set.

my suspition is that in order to utikize the memory better it save the 4th column in the larger file as char. from what that I recall, char takes 8 bits and float (thats how it define the 4th column in the other data set) takes 32 bits.

I wondering if there is a way to inforce it to save it as float? or, maybe I can define another X by one table (where X is the number of line in the data set table) and then join them?

[sani]

SOLVED!

so, what I did was, instead of adding a new column to the table, I created a new table of zeros in the length of my table and than concatenate them. now it works:)

Answer 2

[Gabor]

I would add to check the missmatching table variables by comparing them by for eg.:

class(Table1.Column_name(1))

class(Table2.Column_name(1))

[Adam Danz]

That's what step 2 does in my answer except it checks the class of all columns in the table rather than just the first column.