Products > Programming

[SOLVED] 24bits/96kHz conversion of s32le to f32le in a stdout to stdin pipe?

(1/11) > >>

RoGeorge:
How to pipe 24bits samples from the sound card, as float 32 into another program?
No resampling please, only data format conversion.

There are 3 different formats to deal with:
- the source (ALSA driver of an audio card) produces 24bits at each sample
- the 24 bits are packed as S32_LE by the recording program, arecord, and can pipe to stdout
- the playing program, baudline (an audio spectrum analyzer), doesn't know s32le, but can read float 32 from stdin

The goal is to get all the 24bits into 'baudline', without lowering the resolution at 16bits and without resampling.

Tried so far:
1. - piping through sox, very cumbersome syntax and slows down the frame rate in 'baudline'
2. - convert the format with python:
        - read S32_LE from stdin
        - convert S32_LE to f32le
        - write f32le to stdout

The Python program is an example from stackoverflow, which works, except 'baudline' displays rubbish, something is not OK with the format conversion (inside the process_data(buffer) there are a few failed attempts to convert, all commented out now):


--- Code: ---#!/usr/bin/env python3

########
# g.py #
########

# run python3 with stin stdout stderr unbuffered (-u)
# arecord -D hw:1,4 -B 10000 -r 96000 -f S32_LE -c 2 | python3 -u ./le32_to_le32f.py | aplay
#
# arecord -D hw:1,2 -B 10000 -r 96000 -f S32_LE -c 2 | \
#   python3 -u ./le32s_to_le32f.py | \
#   ./baudline -stdin -record -channels 2 -format le32f

import signal
import sys

from struct import pack, unpack

def process_data(buffer):
    # convert s32le into f32le
    ## buffer = pack('<f', float((int.from_bytes(buffer, "little")/256)))
    ##pack('<f', float((int.from_bytes(buffer, "little")/256)))
    ##    pack('<f', float(int.from_bytes(buffer, "little", signed=True))/float(2**23) )
    ##    print(buffer)
    ##    print(int.from_bytes(buffer, "little", signed=True))
    ##    print(int.from_bytes(buffer, "little", signed=True)/256)
    ##    print(float(int.from_bytes(buffer, "little", signed=True)))
    ##    print()

    sys.stdout.buffer.write(buffer)
    sys.stdout.buffer.flush()


# def read_stdin_stream(handler, chunk_size=1024):
def read_stdin_stream(handler, chunk_size=8192):
    with sys.stdin as f:
        while True:
            buffer = f.buffer.read(chunk_size)
            if buffer == b'':
                break
            handler(buffer)


def signal_handler(sig, frame):
    sys.stdout.buffer.flush()
    sys.exit(0)


def main():
    signal.signal(signal.SIGINT, signal_handler)

    # notice the `chunk_size` of 1 for this particular example
    # read_stdin_stream(process_data, chunk_size=1)
    read_stdin_stream(process_data, chunk_size=4)


if __name__ == "__main__":
    main()

--- End code ---


Tried to read the docs from https://docs.python.org/3.8/library/struct.html#struct-examples and the first example there doesn't work:  :-//

--- Code: ---$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from struct import *
>>> pack('hhl', 1, 2, 3)
b'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'
>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: unpack requires a buffer of 16 bytes
>>> calcsize('hhl')
16
>>>

--- End code ---


Since not even the example from the manual works, thought I might ask instead of keep poking blindly at it.
To show the alignement of the 24bits inside the S32_LE, this is what a hex dump for noise only looks like (the small +/- noise signal around zero is useful here to see the layout of the 24bits into 32 and the endianness):


--- Code: ---$ arecord -v -D hw:1,2 -B 10000 -r 128000 -f S32_LE -c 2 | hd
...
0019b0f0  00 60 00 00 00 a0 ff ff  00 c0 01 00 00 e0 fd ff  |.`..............|
0019b100  00 a0 fd ff 00 90 05 00  00 10 03 00 00 60 fa ff  |.............`..|
0019b110  00 40 ff ff 00 30 03 00  00 e0 fe ff 00 e0 00 00  |.@...0..........|
0019b120  00 90 04 00 00 40 ff ff  00 d0 fd ff 00 60 02 00  |.....@.......`..|
0019b130  00 a0 fe ff 00 d0 fb ff  00 70 04 00 00 10 02 00  |.........p......|
0019b140  00 10 f9 ff 00 a0 02 00  00 20 06 00 00 a0 fb ff  |......... ......|
0019b150  00 90 fe ff 00 30 04 00  00 70 fd ff 00 c0 ff ff  |.....0...p......|
0019b160  00 10 03 00 00 f0 fa ff  00 60 fe ff 00 d0 04 00  |.........`......|

--- End code ---

If there is a 4 bytes variable called "buffer", how to convert "buffer" from S32_LE into f32le, preferably using struct from Python?

golden_labels:
Note: float32 has a 23-bit mantissa, so you will inevitably lose 1 bit at the maximum level.

What about using ffmpeg?
--- Code: ---ffmpeg -f s32le -i pipe: -c:a pcm_f32le -f f32le pipe:
--- End code ---

The first -f sets the input format, -i pipe: instructs ffmpeg to read from stdin, -c:a sets the output codec to PCM (float32 LE) and the second -f the output file format to raw float32 LE.

DiTBho:
so s32le means  "PCM signed 32-bit little-endian"

magic:
Good enough for x86/ARM Linux.


--- Code: ---#include <stdio.h>

int main() {
int i;
while (fread(&i, 4, 1, stdin)) {
float f = i;
fwrite(&f, 4, 1, stdout);
}
}
--- End code ---

adeuring:

--- Quote from: RoGeorge on August 07, 2022, 07:27:37 am ---Tried to read the docs from https://docs.python.org/3.8/library/struct.html#struct-examples and the first example there doesn't work:  :-//

--- Code: ---$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from struct import *
>>> pack('hhl', 1, 2, 3)
b'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'
>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: unpack requires a buffer of 16 bytes
>>> calcsize('hhl')
16
>>>

--- End code ---

--- End quote ---

Welcome to the messy world of C types ;)! I guess that the documentation was written on a 32 bit platform: h represents a C short integer, which has quite reliably 16 bits; l represents a C long integer, which has 32 bits in the example from the documentation but 64 bits on your machine...

If you run the example with a slight modification, it works:


--- Code: --->>> from struct import *
>>> pack('hhl', 1, 2, 3)
b'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'
>>> unpack('hhl', b'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00')
(1, 2, 3)
>>> unpack('hhl', pack('hhl', 1, 2, 3))
(1, 2, 3)
>>>

--- End code ---

(Just use your mouse to copy the result of the pack() call as an argument into the unpack() call.)

The table under "Format Characters" in the docs (https://docs.python.org/3.8/library/struct.html#format-characters) mentions i as another 32bit integer type which seems to indeed have 32 bits on today's 64bit platforms:

--- Code: --->>> pack('i', 2)
b'\x02\x00\x00\x00'

--- End code ---

So, try pack('hhi', 1, 2, 3) instead.


--- Quote ---If there is a 4 bytes variable called "buffer", how to convert "buffer" from S32_LE into f32le, preferably using struct from Python?

--- End quote ---

I am not absolutely sure about the meaning of "S32_LE" and "f32le". Assuming that "S32_LE" means a 32 bit integer in "little endian" order and "f32le" means a 32 bit float number in "little endian" order, this might work
(See https://docs.python.org/3.8/library/struct.html#byte-order-size-and-alignment):


--- Code: --->>> # Create 32 bit "example input".
>>> pack('<i', 123123)
b'\xf3\xe0\x01\x00'
>>> # unpack the 4 byte string, convert to "Python internal float" and pack as 32bit float
>>> pack('<f', float(unpack('<i', b'\xf3\xe0\x01\x00')[0]))
b'\x80y\xf0G'

--- End code ---

Final remarks:
1. @magic showed a nice example of a much more efficient C solution for your problem. Using Python introduces a huge amount of overhead. (Numpy has, IIRC, also some mechanisms to load arrays from/into byte sequences or buffers - that might be another option. Not as fast as a pure C implementation but for sure faster than my example if used for buffers with, let's say, 512 4 byte numbers or more.
2. There is a reason why building a C program from source starts quite often with running a script called "configure" or "autoconf". One important purpose is to figure out which size a given C type has for the current platform and compiler. The source code of these propgrams does not use the standard C types like int, short or long, but int8, int16, int32, which are defined in special C header files that were generated by configure or autoconf.

Navigation

[0] Message Index

[#] Next page

There was an error while thanking
Thanking...
Go to full version