Read binary data as three UBYTE (8-bit Unsigned Byte) in MATLAB and use bit shift operators to get two 12 bit streams

I need to read the data as UBYTE (8-bit Unsigned Byte) in MATLAB and then used bit shift operators to get two 12-bit streams with 3600000 samples. I have to do that by a command that: the first two 12 bit values are contained in the first 3 UBYTEs, 8 bits from the first byte with the first 4 bits from the second byte, then the first 4 bits from byte 2 with all 8 bits from byte 3. Then the process repeats with the next 3 bytes (byte 4-6 etc.) and so on. The related commands are as follows.

How can I apply this command in MATLAB?

[Walter Roberson]

Odd. If the stream were 123456 hex then the result would be 0312 and 0645. Not common.

However this does not tell us which is the Most Significant Byte. It looks likely to me that the 03 and 06 would be intended as the MSB. But x64 architecture uses Little Endian order where the first memory byte is the least significant, so you need to be careful about value. The bit operations are not defined in terms of numeric results

[Eliza]

I read two 12-bits as 'ubit12=>uint16' and the values are 0 to 4095. It uses all 12 bits but I have to read data as three 8-bit UBYT and use shift arithmetic to get 12-bits and the values range sould be approximately between 200 to 1000.

Sould I consider big or little-endian ordering to read bits properly?

[Walter Roberson]

You do not need to read three bytes and do shift arithmetic. I show how you can read ubit4=>ubit16 and then put together the nibbles.

Answer 1

[Steven Lord]

Use the bit-wise operations. You're also going to need to convert between data types.

x = int8(5)
y = int16(x)

You may need or want to read the "Largest and Smallest Values for Integer Classes" section on this documentation page before you start writing your code.

[Steven Lord]

Read through the documentation pages for the functions listed on the bit-wise operations page to which I linked. It will tell you what each of those functions do and also show you examples of how to use those functions. With that information you should be able to implement the equivalent of your line of code in the original message.

[Eliza]

Thank you for your guidance.

Answer 2

[Walter Roberson]

I believe that you should be able to implement the needed transformation by using fread() with 'ubit12=>uint16' and then using swapbytes() https://www.mathworks.com/help/matlab/ref/swapbytes.html

[Walter Roberson]

According to the question as originally posted, which clearer about the bit order, we are looking for 0312 0645. Can we get that?

testnibbles = [0x01 0x02 0x03 0x04 0x05 0x06];
testbytes = testnibbles(1:2:end) * 16 + testnibbles(2:2:end);
dec2hex(testbytes, 2)

ans = 3x2 char array
    '12'
    '34'
    '56'

filename = tempname;
cleanMe = onCleanup(@() delete(filename));

[fid, msg] = fopen(filename, 'w+');
if fid < 0; error('failed to open file "%s" because "%s"', filename, msg); end

writecount = fwrite(fid, testbytes, 'ubit8');

frewind(fid);
test4LE = fread(fid, [1 inf], 'ubit4=>uint8', 'ieee-le')

test4LE = 1×6

2 1 4 3 6 5

●

dec2hex(test4LE, 2)

ans = 6x2 char array
    '02'
    '01'
    '04'
    '03'
    '06'
    '05'

Uh-oh, if we read 4 bits at a time in the (default) ieee-le mode, the order of elements is not what we expect

frewind(fid);
test4BE = fread(fid, [1 inf], 'ubit4=>uint8', 'ieee-be')

test4BE = 1×6

1 2 3 4 5 6

dec2hex(test4BE, 2)

ans = 6x2 char array
    '01'
    '02'
    '03'
    '04'
    '05'
    '06'

The order looks good if we read 4 bits at a time with ieee-be

frewind(fid)
test8LE = fread(fid, [1 inf], '*ubit8', 'ieee-le')

test8LE = 1×3

18 52 86

dec2hex(test8LE, 2)

ans = 3x2 char array
    '12'
    '34'
    '56'

frewind(fid)
test8BE = fread(fid, [1 inf], '*ubit8', 'ieee-be')

test8BE = 1×3

18 52 86

dec2hex(test8BE, 2)

ans = 3x2 char array
    '12'
    '34'
    '56'

but reading 8 bits at a time is fine no matter whether we use ieee-le or ieee-be

frewind(fid)
test12LE = fread(fid, [1 inf], 'ubit12=>ubit16', 'ieee-le')

test12LE = 1×2

1042 1379

dec2hex(test12LE, 4)

ans = 2x4 char array
    '0412'
    '0563'

Not so good -- ubit12 with ieee-le has read the LSB of the third byte instead of the MSB of the third byte

frewind(fid)
test12BE = fread(fid, [1 inf], 'ubit12=>ubit16', 'ieee-be')

test12BE = 1×2

291 1110

dec2hex(test12BE, 4)

ans = 2x4 char array
    '0123'
    '0456'

ubit12 with ieee-be has read the MSB of the third byte as we would hope. But it did not put the nibbles at the place in the byte stream that we were hoping for

test12LEs = swapbytes(test12LE)

test12LEs = 1×2

4612 25349

dec2hex(test12LEs, 4)

ans = 2x4 char array
    '1204'
    '6305'

Not what we want because ubit12 ieee-le did not read the bits we wanted

test12BEs = swapbytes(test12BE)

test12BEs = 1×2

8961 22020

dec2hex(test12BEs, 4)

ans = 2x4 char array
    '2301'
    '5604'

Not what we were hoping for either, as the nibbles were not packed at the place in the byte stream that we were hoping for. We are hoping for 0312 according to the original post

right_order = uint16(uint16(test4BE(3:3:end)) * 256 + uint16(test4BE(1:3:end)) * 16 + uint16(test4BE(2:3:end)))

right_order = 1×2

786 1605

dec2hex(right_order, 4)

ans = 2x4 char array
    '0312'
    '0645'

Putting that all together:

frewind(fid);
BE4 = fread(fid, [1 inf], 'ubit4=>ubit16', 'ieee-be');
right_order = BE4(3:3:end) * 256 + BE4(1:3:end) * 16 + BE4(2:3:end)

right_order = 1×2

786 1605

dec2hex(right_order, 4)

ans = 2x4 char array
    '0312'
    '0645'

fclose(fid);

Thus, for the sections that you need 12 bits at a time, fread() them using ubit4 => ubit16 and ieee-be, and then do simple arithmetic calculations according to the positions you want the nibbles to show up in in the result. The right_order can obviously be changed if you need a slightly different order of nibbles, just by changing which group is multiplied by 256 and which group is multiplied by 16.

[Eliza]

Thank you a lot for your time and guidance. That works. I had to consider Big-endian instead of Little-endian ordering and also change the sample size in fread() to get the values properly.

Sorry, about your last command 'right_order = BE4(3:3:end) * 256 + BE4(1:3:end) * 16 + BE4(2:3:end)', how did you apply special bits from different bytes based on the original question (i.e. 8 bits from the first byte with the first 4 bits from the second byte, then the first 4 bits from byte 2 with all 8 bits from byte 3)?

For example, for the first part (8 bits from the first byte), we have to consider the first byte multiplied by 2^8= 256, but how you used 'BE4(3:3:end) * 256'?

[Walter Roberson]

You originally posted C type code for the value reconstruction. For reasons unknown to me, you removed that part. But before you removed it I had already posted showing the nibble order reconstruction implied by the code, 0312 0645. That does not involve multiplying the first byte by 256: it involves multiplying the first nibble of the second byte by 256.

If you multiply the first byte by 256 then the result is going to exceed 1000 unless the first byte is no more than 3, which would require that the first nibble would always have to be 0 in the data. Also since you would have used both of the first two nibbles when you multiplied the byte by 256, the second byte would have to have a value constructed only of the third nibble, and you would only be able to reach near 1000 if the third nibble was multiplied by 16, leading to a reconstruction order of 0x1230 and the data would have had to had nibble 1 be 0 and nibble 2 be no more 3 in value, leading to 0x0230 nibble reconstruction, so one of 00X0, 01X0, 02X0, 03X0 where X is the third nibble... less than 64 valid data combinations.

I think you should reconsider your statement that the first byte needs to be multiplied by 256. Your C code multiplied one the nibbles by 256, not one of the bytes.

[Eliza]

I deleted the C type code because I thought it was a bit different from what should I do in MATLAB based on the original statements in my question.

Thank you for your time and perfect tips. Bless you!

[Walter Roberson]

As you seem to think that you need a version with arithmetic operations, here is one. It consists of reading the data as uint8, breaking the bytes into nibbles, and doing simple calculations with nibbles.

You will notice that it produces the same result as using ubit4=>uint16. So you can either read the data with ubit4 and do a very small bit of arithmetic on the nibbles, or you can read the data as ubit8 and do more arithmetic on the bytes

testnibbles = [0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0a 0x0b 0x0c];
testbytes = testnibbles(1:2:end) * 16 + testnibbles(2:2:end);
dec2hex(testbytes, 2)

ans = 6x2 char array
    '12'
    '34'
    '56'
    '78'
    '9A'
    'BC'

filename = tempname;
cleanMe = onCleanup(@() delete(filename));
[fid, msg] = fopen(filename, 'w+');
if fid < 0; error('failed to open file "%s" because "%s"', filename, msg); end
writecount = fwrite(fid, testbytes, 'ubit8');
frewind(fid);

%reading logic begins here
test8BE = fread(fid, [1 inf], 'ubit8=>uint16', 'ieee-be')

test8BE = 1×6

18 52 86 120 154 188

dec2hex(test8BE, 2)

ans = 6x2 char array
    '12'
    '34'
    '56'
    '78'
    '9A'
    'BC'

nibble1 = bitand(test8BE, uint16(240)) / 16;
nibble2 = bitand(test8BE, uint16(15));

BE4 = reshape([nibble1; nibble2], 1, [])

BE4 = 1×12

1 2 3 4 5 6 7 8 9 10 11 12

right_order = BE4(3:3:end) * 256 + BE4(1:3:end) * 16 + BE4(2:3:end)

right_order = 1×4

786 1605 2424 3243

●

dec2hex(right_order, 4)

ans = 4x4 char array
    '0312'
    '0645'
    '0978'
    '0CAB'

fclose(fid);

[Eliza]

Thank you so much for your perfect answer.