Electronics > Projects, Designs, and Technical Stuff
Software to store, aggregate, analyse and display ~1G points of streamed data
Twoflower:
My first idea was why not use a lossless(!) audioformat and use existing software to process it. I think even octave should be able to read audiofiles files, or not? The good thing is, they are binary files. So reasonably small. But you can't add additional data to the samples (e.g. timecode).
But then the simplest way would be to use a CSV to record the data. Depending on the requirements do a on the fly compression if the system has limited storage capacity (e.g. on linux you might be able to pipe the file trough gz or similar). Or use some binary to text coding from the beginning (e.g. Base64). If it turns out the data is in a format not reach your required performance, re-format the data to match the computation requirements. As others pointed out actual computers are fast, especially if you can store the data on a SSD.
You should think about the effort you spend on that. For a one-time test you might life with a low performing solution and be done with it. If you re-do that multiple times spending on the performance might start t be useful. It is kind of finding the point of good enough.
bson:
The classic is rrdtool.
Karel:
EDF uses 16-bit integers, so the range is from -32768 to +32767.
When you do the conversion from CSV to EDF, you can also select to convert to BDF
(which is EDF but with 24-bit integers). This increases the range 256 times.
Probably that's what you need.
Navigation
[0] Message Index
[*] Previous page
Go to full version