Open and read unique spectrum file .spc

I have some old wave spectra data I need to process, but I'm having trouble getting MatLab to read the files.

%% I - Load files of wave data

SpecFiles = dir("RawData\*.spc");
numfiles = length(SpecFiles);
SpecData = cell(1,numfiles);


for k = 1:numfiles
    SpecData{k} = readtable(SpecFiles(k).name);
    %SpecData{k} = dlmread(SpecFiles(k).name,' ',5,0);
    %SpecData{k} = readSPC(SpecFiles(k));
    %SpecData{k} = tgspcread(SpecFiles(k));
end


%data = fileDatastore("RawData\",@tgspcread,@fcn);
%test = tgspcread(SJCSPC195401010000.spc)

Below is an example of the file format. The first 4 rows are unimportant to me, the 5th row begins the data that I need out of them with the fith row being the time stamp and other information and the 6th through the 28th row being the spectral info.

23    35
122   762   100.0
2177571.00       408574.12   255.0  1
0.0390  0.0429  0.0472  0.0520  0.0572  0.0630  0.0693  0.0763  0.0840  0.0924  0.1017  0.1120  0.1233  0.1357  0.1493  0.1643  0.1809  0.1991  0.2191  0.2412  0.2655  0.2922  0.3216
1954010100   82   52  -0.08   1.00    0.0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0   0.001   0.004   0.010   0.022   0.043   0.076   0.111   0.133   0.144   0.141   0.124   0.097   0.065   0.038   0.021   0.014   0.009   0.004   0.001  0  0  0  0  0
0   0.001   0.002   0.003   0.005   0.008   0.014   0.020   0.023   0.025   0.030   0.033   0.035   0.037   0.040   0.050   0.073   0.105   0.140   0.183   0.225   0.246   0.227   0.176   0.116   0.070   0.048   0.029   0.013   0.005   0.001  0  0  0  0
0   0.001   0.003   0.003   0.005   0.007   0.013   0.018   0.021   0.024   0.027   0.031   0.034   0.038   0.046   0.061   0.087   0.120   0.154   0.200   0.256   0.309   0.342   0.339   0.296   0.240   0.196   0.142   0.085   0.044   0.016   0.003  0  0  0
0   0.001   0.002   0.003   0.004   0.005   0.009   0.012   0.013   0.011   0.008   0.005   0.004   0.007   0.015   0.029   0.059   0.103   0.160   0.246   0.357   0.460   0.523   0.538   0.507   0.454   0.403   0.325   0.227   0.136   0.058   0.012   0.001  0  0
0  0  0  0  0  0  0  0  0   0.001   0.003   0.008   0.014   0.020   0.027   0.042   0.081   0.144   0.218   0.314   0.425   0.533   0.621   0.679   0.701   0.687   0.652   0.578   0.459   0.315   0.160   0.046   0.007  0  0
0  0  0  0  0   0.001   0.002   0.003   0.006   0.010   0.014   0.018   0.020   0.023   0.029   0.039   0.059   0.091   0.131   0.191   0.273   0.363   0.453   0.541   0.619   0.673   0.695   0.690   0.635   0.511   0.319   0.124   0.026   0.001  0
0  0  0  0  0   0.000   0.001   0.003   0.006   0.010   0.017   0.023   0.030   0.039   0.050   0.065   0.093   0.134   0.178   0.224   0.270   0.318   0.374   0.438   0.504   0.559   0.594   0.616   0.611   0.555   0.420   0.221   0.070   0.006  0
0  0  0  0  0  0  0  0  0  0   0.002   0.007   0.014   0.028   0.051   0.077   0.103   0.129   0.154   0.179   0.204   0.231   0.264   0.304   0.345   0.383   0.412   0.436   0.446   0.436   0.387   0.268   0.123   0.021  0
0  0  0  0  0  0  0  0  0  0   0.004   0.013   0.023   0.041   0.068   0.096   0.117   0.132   0.146   0.157   0.166   0.175   0.191   0.213   0.235   0.255   0.271   0.286   0.298   0.305   0.302   0.265   0.173   0.052  0
0  0  0  0  0  0  0   0.004   0.011   0.020   0.026   0.031   0.036   0.046   0.063   0.080   0.091   0.095   0.100   0.108   0.120   0.132   0.145   0.157   0.169   0.180   0.191   0.201   0.209   0.211   0.213   0.206   0.179   0.095  0
0  0  0  0  0  0  0   0.002   0.006   0.009   0.017   0.028   0.040   0.048   0.052   0.056   0.062   0.069   0.077   0.085   0.092   0.100   0.108   0.115   0.122   0.128   0.135   0.141   0.144   0.143   0.141   0.138   0.133   0.105  0
0  0  0  0   0.005   0.015   0.025   0.030   0.031   0.031   0.033   0.036   0.040   0.041   0.042   0.042   0.044   0.047   0.050   0.056   0.063   0.070   0.077   0.084   0.090   0.094   0.096   0.100   0.100   0.100   0.099   0.097   0.092   0.085  0
0   0.001   0.002   0.004   0.007   0.012   0.016   0.019   0.021   0.023   0.025   0.028   0.031   0.033   0.033   0.033   0.035   0.038   0.042   0.045   0.049   0.052   0.056   0.058   0.061   0.062   0.063   0.064   0.064   0.063   0.062   0.061   0.060   0.059  0
1954010100   82   56  -0.08   1.00    0.0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

[Stephen23]

Please upload a sample data file by clicking the paperclip button.

[Peter Bohn]

Attached is a complete sample file. It is important to note, that I need to process approximately 3000 files, so it will not be feasible to change each extiension to a .txt.

[Walter Roberson]

You need to use special parameters to read the data correctly.

format long g
T = readmatrix('TestFile.txt', 'headerlines', 4, 'Delimiter', '  ');
disp(T(1:50,1:6))

                1954010100                        82                        52                     -0.08                         1                       NaN
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                     0.001                     0.002                     0.003                     0.005
                       NaN                         0                     0.001                     0.003                     0.003                     0.005
                       NaN                         0                     0.001                     0.002                     0.003                     0.004
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                     0.005
                       NaN                         0                     0.001                     0.002                     0.004                     0.007
                1954010100                        82                        56                     -0.08                         1                       NaN
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                     0.003                     0.004                     0.006
                       NaN                         0                         0                     0.003                     0.004                     0.005
                       NaN                         0                         0                     0.002                     0.003                     0.004
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                         0
                       NaN                         0                         0                         0                         0                     0.005
                       NaN                         0                     0.001                     0.002                     0.004                     0.007
                1954010100                        82                        60                     -0.08                         1                       NaN
                       NaN                         0                         0                         0                         0                         0

[Stephen23]

"it will not be feasible to change each extiension to a .txt."

Do not change the file extension when importing using READTABLE or READMATRIX.

Instead specify the FileType:

https://www.mathworks.com/help/matlab/ref/readmatrix.html#mw_a8b267d8-f53e-4732-9840-0b986d711d79

[dpb]

"it will not be feasible to change each extiension to a .txt."

Of course it is a trivial exercise to do that with either an OS command or in Matlab with movefile, but I didn't suggest to change the working file names, only that the Answers forum won't accept anything excepting one of a very small set of file extensions.

[Peter Bohn]

Sorry, I guess I misunderstood. It seemed the the file type was possibly the issue.

[dpb]

If you use one of the fancy new(ish) i/o functions like readmatrix or readcell, etc., they also have a very short list of builtin file extensions they recognize as text files automagically, so it is a problem unless you tell them explicitly as @Walter Roberson noted, I believe. I figure it would have been better to lmit the list to auto the various Excel and similar formats and then try anything else as text first rather than just failing to accept the file name--but, they didn't ask my opinion. :)

Answer 1

[dpb]

L=[0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0];
numel(L)

ans = 35

:The first 4 rows are unimportant to me, ..."

Well, you may think it unimportant, but that first row

% 23    35
% 122   762   100.0
% 2177571.00       408574.12   255.0  1
% 0.0390  0.0429  0.0472  0.0520  0.0572  0.0630  0.0693  0.0763  0.0840  0.0924  0.1017  0.1120  0.1233  0.1357  0.1493  0.1643  0.1809  0.1991  0.2191  0.2412  0.2655  0.2922  0.3216
% 1954010100   82   52  -0.08   1.00    0.0

is vital because it tells you how may row and columns of data are there to be read...

I pasted your text above into a file and the following seems to work although I haven't vetted it throroughly.

@Stephen23 has noted often in the past that replica and/or sample files don't count, and that may be true here, too, so as he suggests, attaching a copy of a real file would be most useful.

NOTA BENE: You'll have to change the extension to .txt, .dat or something else that the Answers forum 'bot will accept to upload it....

fid=fopen('spectrum.txt','r');
r=fscanf(fid,'%d',1)

r = 23

c=fscanf(fid,'%d',1)

c = 35

for i=1:4, l=fgetl(fid); end
fmt=[repmat('%f ',1,6)];
tdata=fscanf(fid,fmt,6);
fprintf(fmt,tdata), fprintf('%s',newline)

1954010100.000000 82.000000 52.000000 -0.080000 1.000000 0.000000 

fmt1=[repmat('%f ',1,c) newline];
fmt2=repmat(fmt1,1,r);
s=fscanf(fid,fmt2,r*c);
whos s

  Name        Size            Bytes  Class     Attributes

  s         805x1              6440  double              

plot(s)

ans = 805

Repeat until run out of data...

[Peter Bohn]

You're right @dpb, the first line is important, but it only tells you how many rows and columns in each spectral measurement. The second line is how many points positions measurements were taken (in a grid layout from the wave modeling software). The really important line is the 5th and every 24 lines after, which gives the location (in this grid) of where the measurements were taken. That will later be used to pull certain pieces of data relevant to the study.

Thanks for your responce by the way!

[dpb]

As noted, "rinse and repeat"...

fid=fopen('spectrum.txt','r');
r=fscanf(fid,'%d',1)
c=fscanf(fid,'%d',1)
fmt=[repmat('%f ',1,6)];
fmt1=[repmat('%f ',1,c) newline];
fmt2=repmat(fmt1,1,r);
for i=1:4, l=fgetl(fid); end
tdata=fscanf(fid,fmt,6);
s=fscanf(fid,fmt2,r*c);
while ~feof(fid)
  tdata=[tdata;fscanf(fid,fmt,6)];
  s=[s;fscanf(fid,fmt2,r*c)];
end
fid=fclose(fid);

Answer 2

[Image Analyst]

If youi're dealing with Spectroscopy, you really need to get GSTools library. It has functions to read SPC file(s), and other useful spectroscopy functions.

GSTools version 0.4.2

A set of matlab functions to read, write and deal with .spc spectra

using MATLAB ...®

By Kris De Gussem

Research assistant of the Research Foundation Flanders

Ghent University

Department of analytical chemistry

Proeftuinstraat 86

B-9000 Ghent

web: http://www.AnalChem.UGent.be/Raman/

e-mail: Kris.DeGussem @ gmail.com

%function GSimportspec
%
%GSimportspec imports all spc-files in a folder. All files are collected in
%a single structure array containing all spectra with all spectral information.
%
%Syntax:
%    [spectra, path] = GSimportspec (path, compatibility);
%    [spectra, path] = GSimportspec (path);

This is the package I used to read files that our spectroscopists gave me (they recommended it to me). Then you don't have to do all the quirky MATLAB gymnastics to read a file format MATLAB is not familiar with natively.

See my attached SPC file plotter.