Reorganization of experimental data
Show older comments
Hello everybody,
I have a problem with the reorganization process of my experimental data.
Attached there is an example of a typical data set
0 0 -93.964 -0.00976171
1 0.001 -93.964 -0.00976627
2 0.002 -93.965 -0.00991459
3 0.003 -93.965 -0.00997991
4 0.004 -93.963 -0.0100311
5 0.005 -93.962 -0.0101206
6 0.006 -93.964 -0.0101955
7 0.007 -93.961 -0.0098519
8 0.008 -93.958 -0.00946246
9 0.009 -93.945 -0.00970383
10 0.01 -93.921 -0.0105877
11 0.011 -93.884 -0.0125595
12 0.012 -93.829 -0.015714
13 0.013 -93.755 -0.0194494
Results:
Sample:
2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg
Curve Name:
]61[&2021_07_29_cool_variab_heat_fix_59603
Curve Values:
Index t Ts Value
[s] [°C] [mW]
0 0 -93.205 -0.000370132
1 0.01 -93.365 -0.00237034
2 0.02 -93.578 -0.00514436
3 0.03 -93.713 -0.00678095
4 0.04 -93.800 -0.00771421
5 0.05 -93.857 -0.00846012
6 0.06 -93.893 -0.00883113
7 0.07 -93.917 -0.00909422
8 0.08 -93.932 -0.00928991
9 0.09 -93.944 -0.00943325
10 0.1 -93.949 -0.00949218
11 0.11 -93.954 -0.00950072
12 0.12 -93.957 -0.00963767
13 0.13 -93.958 -0.0096308
14 0.14 -93.960 -0.00962155
15 0.15 -93.961 -0.00964608
Results:
Sample:
2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg
________________________________________________________________________________
Not signed STARe SW 16.10
I want to remove all the written texts and empty lines and I want to reorganize the blocks of data in a way to invert their order.
Do you have any suggestion?
1 Comment
Star Strider
on 30 Jul 2021
Attach a ‘typical data set’ as a .txt or .mat file.
Accepted Answer
Star Strider
on 30 Jul 2021
Edited: Star Strider
on 1 Aug 2021
The data file you attached does not appear to bear a strong resemblance to the data file you quoted.
I chose to use textscan here because I have experience using it with files with interrupted formats, however this file appears to have no interruptions:
fidi = fopen('typical_data_set.txt','rt');
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
fclose(fidi);
M = cell2mat(C);
Here ‘M’ is a (44006x4) double array.
A file with interruptions would simply require a while loop to loop through it, saving the individual sections. For an example, see: How to skip lines of Data in the middle of a text file . I believe that file was similar to the one you quoted.
EDIT — (1 Aug 2021 at 00:15)
Removed the generic code example. Nothing else changed.
.
14 Comments
Daniele Sonaglioni
on 31 Jul 2021
Star Strider
on 31 Jul 2021
I did not look at the entire file, however I did not detect any blocks. It just read as one entire matrix, with contiguous values of the first variable (that appears to be a counter of some sort).
Did you attach a different file this time?
.
Daniele Sonaglioni
on 31 Jul 2021
Star Strider
on 31 Jul 2021
I can only detect one block of data using:
fidi = fopen('typical_data_set.txt','rt');
k1 = 1;
while ~feof(fidi)
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
M = cell2mat(C);
if isempty(M) % Empty Matrix Indicates End-Of-File
fprintf(1,'Reading finished, %d Sections\n',k1-1)
break
end
D{k1,:} = M;
fseek(fidi, 0, 0);
k1 = k1 + 1
end
fclose(fidi);
Out = cell2mat(D);
This code would detect a line of text that did not match the numeric format descriptors, stop, and re-start with the next block. (It then optionally combines the blocks into a single matrix if desired.) I have tested it on different files over the several years since I wrote it, and it consistently works. There is only one block of data in the posted file, and only one header line.
I have not gone through the file in detail by hand (it has 44006 rows), so I have no idea where the ‘two lines of text’ are. I would appreciate your displaying them and the rows and row numbers where they occur. I cannot detect their presence.
.
Daniele Sonaglioni
on 31 Jul 2021
Star Strider
on 31 Jul 2021
This turned out to be something of an adventure!
My code works, however it needs to be modified a bit for your file:
fidi = fopen('typical_data_set_2ndversion.txt','rt');
k1 = 1;
while ~feof(fidi)
readline = fgetl(fidi);
if strcmp(readline,'Results:')
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',9, 'CollectOutput',1);
else
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',0, 'CollectOutput',1);
end
M = cell2mat(C);
fprintf(1,'\tSection %2d: (%4d x %d)\n', k1, size(M))
if isempty(M) % Empty Matrix Indicates End-Of-File
fprintf(1,'Reading finished, %d Sections\n',k1-1)
break
end
D{k1,:} = M;
fseek(fidi, 0, 0);
k1 = k1 + 1;
end
fclose(fidi);
Dflip = flip(D); % Reverse Section Order
% Out = cell2mat(D); % Optional Concatenation
The end result is 56 segments of varying row length, all with 4 columns. (The section information is not retained, only the numeric information.) I did not go through all 56 results, however the few I sampled appear to be correct.
.
Daniele Sonaglioni
on 1 Aug 2021
Star Strider
on 1 Aug 2021
Thank you.
If you check, that is exactly what my code does. The individual block contents aree not reordered in any way. Only their order in the ‘D’ cell array are changed, so that the last block becomes the first (for example ‘D{1}’ becomes ‘Dflip{56}’, ‘D{56}’ becomes ‘Dflip{1}’) and so for the rest. I checked that as well as selected individual blocks before I posted my code.
Adding these two lines after the ‘Dflip’ assignment:
Check1 = [D{1}(1:5,:); NaN(1,4); Dflip{end}(1:5,:)]
Check2 = [D{end}(1:5,:); NaN(1,4); Dflip{1}(1:5,:)]
produces:
Check1 =
1.0000 0.0001 25.1940 0.0006
2.0000 0.0002 25.1910 0.0006
3.0000 0.0003 25.1900 0.0005
4.0000 0.0004 25.1910 0.0004
5.0000 0.0005 25.1820 0.0004
NaN NaN NaN NaN
1.0000 0.0001 25.1940 0.0006
2.0000 0.0002 25.1910 0.0006
3.0000 0.0003 25.1900 0.0005
4.0000 0.0004 25.1910 0.0004
5.0000 0.0005 25.1820 0.0004
Check2 =
0 0 -93.9600 -0.0104
1.0000 0.0100 -93.9590 -0.0103
2.0000 0.0200 -93.9580 -0.0104
3.0000 0.0300 -93.9590 -0.0103
4.0000 0.0400 -93.9590 -0.0103
NaN NaN NaN NaN
0 0 -93.9600 -0.0104
1.0000 0.0100 -93.9590 -0.0103
2.0000 0.0200 -93.9580 -0.0104
3.0000 0.0300 -93.9590 -0.0103
4.0000 0.0400 -93.9590 -0.0103
The NaN lines are simply used to separate the matrix samples in order to make it easier to read.
Note that ‘D{end}’ is ‘D{56}’ in this file, and similarly for the ‘Dflip’ elements.
So, the order of each block is flipped as requested, not the contents. This additional bit of code (not necessary for the rest of the code) demonstrates that.
.
Daniele Sonaglioni
on 2 Aug 2021
Star Strider
on 2 Aug 2021
As always, my pleasure!
(I am a physician. That is what we do!)
.
Daniele Sonaglioni
on 2 Aug 2021
Star Strider
on 2 Aug 2021
I am an M.D. Board Certified in Internal medicine, and am a fellowship-trained endocrinologist, although not B/C. I subsequently earned a M.S. in Biomedical Engineering, and have done research in diabetes and most notably neurophysiology that had no direct connection with endocrinology, and instrumentation. My areas of interest are system identification, parameter estimation, instrumentation, and signal processing. I am now retired.
My first exposure to computers was a short undergraduate course in FORTRAN. I earned my M.S. after doing my fellowship and becoming fascinated with homeostatic regulatory mechanisms, specifically with respect to endocrinology, however those signals are so difficult (and expensive) to gather that most of my research was in other areas, where that was not a problem.
And yours?
.
Daniele Sonaglioni
on 2 Aug 2021
Star Strider
on 2 Aug 2021
Interesting!
I wish you the best in your research!
More Answers (1)
Daniele Sonaglioni
on 30 Jul 2021
0 votes
Categories
Find more on Biomedical Imaging in Help Center and File Exchange
on 30 Jul 2021
on 2 Aug 2021
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
