If it only outputs data when the data changes, then you can make the obvious inference of the values between successive outputs. Simply fill those values into your time series.
Expanding data within the record of truth by interpolation would definitely fall to the category of "NEVER EVER" in data science.
This will however be done in the presentation layer. In much the same way it will in a digital scope when interpolation is enabled.
Grafana supports many different ways to interpolate from none through linear to various standard curves.
The data selection and aggregation is very expansive, much more so than the Siglent scope I have. That's without touching the exotic plots either. Things like grouping by non-negative difference/derivative/integral. Some of this I hope to use for classifying electrical consumption into "buckets" by "size of delta" and aiming towards classifying individual devices.
If it only outputs data when the mood takes it, then that is equivalent to "missing or sparse samples". I believe there are standard DSP techniques to deal with that, but I am not familiar with them.
Consider a magnetic door sensor. It takes about 3 milliseconds for it to emit the event that the contact position has changed. So, if you want an accurate representation in your DSP model, you would need a sample period of 1.5ms to accurate record the transitions? Over a 24 hour period that is a lot of data. Over a 10 year period you are taking about £1000s monthly storage costs.
If do use sparse time-series recording of events then, when the door opens and closes only 3 times a day, we only store 6 records. Without losing ANY precision or accuracy. In fact the sample period is 1x10^-9 seconds. If someone manages to open and close it 10 times in one minute, it will be recorded.
So the questions I am asking are surrounding how to "quantise", select, aggregate, group, box, window the data to give the best visualisation and produce the most "useful" graph.
That entirely depends on the nature of the data and what you will use to which the derived information for.
Yes. Now you are getting it. Take the next step and you will be there. This system can record any data, from anything, at any frequency, periodicity in true and accurate temporal form, without having to store (or even process) a contiguous stream of data points. It doesn't even require any configuration. You just throw data at it whenever you want.
How quickly can a 1 minute rolling average of counts per minute change? That's open to debate, but as it's sampled once a second, then that is the maximum frequency available. Thus it "could" only tell me information about every 2 minutes and below?
A moving average is a form of low-pass filter.
Consult statistics textbooks and signal processing textbooks
What is a "count per minute" then averaged with a low pass filter? It's already a time compound metric which is then averaged again over time.
The ideal, although heavy on resources, would be to process each individual "count" event. That would be the highest accuracy of recording. As an ionisation event is one of the most random things possible (in context) it's time domain resolution is infinite to the planck constant. However, nano-second would do just fine.
This would work fine for background of ~20CPM, but if I put it onto an Uranium glazed plate it will destroy my database server and my network with 10s of thousands of events per second.
I could write better firmware for the device to emit "counts" bucketed to the second for higher resolution, but I have what I have for now. I am stuck with the rolling average and the 1 minute "bucket" size and reporting interval.
What would I be missing? At a long reach, if my counts where constantly coming in small groups like C.C.C.C................CC.C.C.C..... the CPM would not tell me.
If you were to use DSP techniques on this data, as I think you are suggesting, you would have to pick a rather large bucket size, aka period. Then count and group the ionisation events into a single integer each period. In an impure, interrupted and mutated form. Given you want to store 10 years of data. That bucket size will need to HUGE is you are recording non-sparse data series. Good luck processing or storing it.
In my setup I could store every single ionisation event accurate to the nano second and still use orders of magnitude less storage and less processing. I can then do any, all and more mathmatical and statistical wizardry on that data to produce the same as the DSP would... and much more.
Anyway. It's likely we are basically saying the same thing, it's done in different ways, in different places. If you store the raw data as and when it occured, you can always run your bucketing and averaging onto a contiguous stream at a given frequency later. If you do it while you are recording the data, you are stuck with your low resolution approximation.
Maybe I am still wrong, but I can't think in terms of frequency when I have non-periodic samples. Or at least the data storage does not require or retain any periodicity, only the individual discrete sample time stamps.
Non-periodic data still has frequency domain representations. Consider "white" noise, "pink" noise, Poisson processes, (time-domain) impulses.
Consider the frequency domain of when I drive my car into the driveway.
You might ask, why would I care about capturing those events with such an accurate timestamp? Well, I don't record these events for pure amusement. There are automations which respond to them. So when one event happens a cascade of other events may, or may not happen. Having highly accurate timestamps on discrete events allows for their execution order to be determined. Cause and effect to be known.