How can I fopen/fwrite into memory, or convert my fread double array

Hi, I am using a parfor loop to read images of jpeg compressed sequences (NorPix .seq).

Each worker fopen's the sequence and then fseek's to a previously specified 'readStart' and fread's the 'imageBufferSize'.

fid = fopen(fileName,'r','b');  
fseek(fid,readStart,'bof');
JpegSEQ = fread(fid,imageBufferSize,'uint8','ieee-le');

Afterwards my working solution is to fwrite this 'JpegSEQ' variabe to a temporary jpg file.

tempName = [parallel_ID '_tmp.jpg'];
tempFile = fopen(tempName,'w');
fwrite(tempFile,JpegSEQ);
fclose(tempFile);
I = imread(tempName);

The JpegSEQ/imageBufferSize is variable due to compression (see below). However, fwrite will always generate the correct image dimensions of 420x2048 (WxH). Unfortunately, writing 8 temporary jpg to my HDD simultaneously will result in errors reading frames. If I do

pause(0.01);

after reading the image I can avoid all reading/writing errors, but with less performance.

Therefore, I would need a way to 'fwrite into memory' since this should be faster. Otherwise, I could just convert my JpegSEQ double array to a image of correct size, but reshape would not work due to the different JpegSEQ/imageBufferSize sizes.

readStart  imageBufferSize
1028  115458
116494  116032
232534  115383
347925  115535
463468  116119
579595  115892
695495  115766
811269  115810

Does someone know a solution?

[OCDER]

Why can't you use imwrite and imread?

https://www.mathworks.com/help/matlab/ref/imwrite.html https://www.mathworks.com/help/matlab/ref/imread.html

[Walter Roberson]

imread does not handle seq files as far as I recall.

[Paul Siefert]

I'm already using imread. You mean imwrite instead of fwrite? I will try that, but I thought the problem is the write/read speed of my HDD. Or do you think it is fwrite(tempFile,JpegSEQ) or fclose(tempFile) that is too slow to successfully imread the files?

Answer 1

[OCDER]

Do you have SSD or HDD on your computer? SSD is much faster and perhaps this is the easy solution. Otherwise, you'd have to make your own uint8 matrix to jpg file converter, as so far, matlab only provides file-to-jpg converters such as rjpg8c.mexw64 located in the private folder of imagesci toolbox folder.

Instead of making temporary files in parallel, use serial processing to convert all your .Seq file into a folder containing many .jpg files once. Then use imread to load them all into a single matrix in memory, so you can do parallel processing.

 1) Extract all .jpg files from a .seq file using your fread/fwrite method
 2) Load all .jpg files into a single HxWx3xFrame matrix using imread
 3) Use parallel processing to do what you have to do using parfor, etc.

Parallel processing is not great for reading/writing tasks to hard drive.

[Paul Siefert]

Thanks for your suggestions! I was avoiding my local SSD because I heard that they should not be used for many writing processes.

My SEQ have 80 - 130 GB, usually over a million allocated frames. Your suggestion to write all first would probably work, but needs at least this amount of disk space. I doubt that all jpg files could be loaded into a single matrix like this, although they are grayscale - I could somehow split the batch however.

Do you have any further information regarding rjpg8c.mexw64? I cannot find much in the documentation or google, mostly error messages. If I simply try to use it I get

K = rjpg8c(JpegSEQ)
Undefined function 'rjpg8c' for input arguments of
type 'double'.

Same as for uint8. I don't think I know enough to create my own uint8 matrix to jpg file converter.

I will stick to your suggestion if there is no solution without a big 'HDD memory' requirement. I would love to avoid my HDD and use my RAM.

[Walter Roberson]

If you want to build a stream of contents of a jpeg file and then read it in as if it were a jpeg file on disk, then the known approach for that is to use some java stream I/O objects.

[Paul Siefert]

Thank you Walter, I have not worked with Java in Matlab before, could you tell me where I can find futher information implementing java streams in matlab scripts?

I just discovered a simple workaround using two temporary jpg files

if mod(Frame,2)
    tempName = [parallel_ID '_tmp1.jpg'];
else
    tempName = [parallel_ID '_tmp2.jpg'];
end

suggesting that it is not my HDD write speed, but the access to the individual files.

[OCDER]

Can you show the full parfor loop, or code you're using right now?

[Walter Roberson]

Sorry, I do not seem to be able to concentrate well enough at the moment to research through the various Java possibilities.

[Paul Siefert]

got your point Walter ;)

[Paul Siefert]

Workaround of creating several temp files as shown in my 7 Aug 2018 at 14:38 comment works. I furthermore create a new temp file if fid is < 0. Thanks for your advice.