mplot: logging and plotting script for long term measurements

[Muxr]

I shared a plotting and logging script for long term measurements with the folks over on the USA calibration club thread: https://www.eevblog.com/forum/metrology/usa-calibration-club/msg1219008/#msg1219008

Currently it's been tested on Keithley 2015, but should likely work with other meters from the Keithley family, via serial rs232 interface. I will probably add more support over time, but any instrument should work with minor tweaks possibly.

Most likely only this function should need modification: https://github.com/sirmo/muxrplot/blob/master/mplot/logger.py#L94

Here is an example of it in action:


The script was written using Python and python libraries like, scipy, numpy, seaborn and pyserial. It's only tested to work on Unix-y systems, but it should be possible to make it work under Windows.

If you're on Windows probably the easiest is to get a VirtualBox VM of Linux going, VirtualBox allows passing of a USB interface to the guest for instance. The other option, which is the main reason why I decided to write this for Unix is that it should work on Raspberry Pi. I would like to have the ability for instance to connect multiple remote instruments to a R Pi and have it log all the measurements, I can just then login remotely over wifi if I need to fetch the data.

Anyways, the script is hopefully fairly simple and should be easy to modify to your needs.

The project can be found here: https://github.com/sirmo/muxrplot together with the install instructions.

I will be monitoring this thread, so if you need assistance with modifications to make it work for your needs I am willing to help.

[vindoline]

Since this is the "metrology" section, we should probably fix the Y-axis labels so the don't imply pico-volt sensitivity!  (Unless you do have an 11.5 digit meter)

[Muxr]

Since this is the "metrology" section, we should probably fix the Y-axis labels so the don't imply pico-volt sensitivity!  (Unless you do have an 11.5 digit meter)

That's a good point, will work on that one! Cheers. 

[Muxr]

It's been fixed:


It defaults to 7 least significant digits. I also added an option so you can tweak it when running graphs.

Code: [Select]

> mplot -h
mplot
usage: mplot [-h] [-t TITLE] [-y YDIGITS] [infile] [outfile]

positional arguments:
  infile
  outfile

optional arguments:
  -h, --help            show this help message and exit
  -t TITLE, --title TITLE
                        title to be used in the chart
  -y YDIGITS, --ydigits YDIGITS
                        Number of least significant digits in the Y labels

[cellularmitosis]

This is great!  I'll definitely be using this myself!

[cellularmitosis]

Just got a Si7021 breakout board working last night.  I used your script to plot the temperature and humidity.

The temperature is purple at the bottom of the graph (in C), and the humidity is in cyan at the top of the graph (in %).

Interesting to see how variable the humidity is -- I'm pretty sure you're seeing the frequency at which my A/C came on.  I think the low dip at the beginning was when I adjusted the thermostat lower before going to bed (it ran longer and lowered the humidity further), and the drift at the end is maybe when I bumped up the thermostat a couple of degrees, so the humidity was no longer being controlled.

[cellularmitosis]

(might as well post the code I used to log the data.  I had some fun using a binary protocol with CRC between the arduino and PC.)

[Muxr]

Nice!

Yup, one of the main reasons builders don't recommend oversized AC units is to control the humidity better, when they are on for longer periods of time the humidity drops. I actually have an additional dehumidifier in my house and my humidity is usually around 40%, outside humidity is like 80%-90% (Florida).

High humidity does mean you pretty much don't have to worry about ESD though

[cellularmitosis]

You got your first pull request bub

I ran into an error with matplotlib when I tried to plot several days worth of data from an LTC2400 (over 1 million data points):

"OverflowError: Allocated too many blocks"

It turns out you can configure matplotlib to consider the data in chunks when it performs the plot, which avoids this error for large data sets (see https://stackoverflow.com/questions/20330475/matplotlib-overflowerror-allocated-too-many-blocks/23361090#23361090 ).

[Muxr]

Nice find @cellularmitosis! Thank you! Merged.

[cellularmitosis]

Here's a PR to add a rolling-average filter: https://github.com/sirmo/muxrplot/pull/2

[Muxr]

Here's a PR to add a rolling-average filter: https://github.com/sirmo/muxrplot/pull/2

Sweet! Merged!

[Muxr]

In case anyone has a Fluke 8845/8846 I wrote a quick python class to interface with it:

https://gist.github.com/sirmo/8096cc1fef598762494e3d987befbfef

It will open a connection to the given ip/port, set NPLC 60 and blank out the screen (to save from VFD burnout).

[alm]

Have you considered using an existing library like python-ivi for the instrument drivers instead of rolling your own? Or is that interface insufficient for the features you need?

[Muxr]

I suppose I could give it a try (totally forgot about it). They don't seem to list Fluke 884* models on the supported list though.

[alm]

As far as far as I know they do not support the Fluke. But implementing an existing interface may be more productive than rolling your own by better facilitating reuse.

[Muxr]

I can look at writing a PR for it, sure. I just wrote that snippet in 10 minutes really. A Fluke 8846 driver for python-ivi will take much longer than that.

[alm]

True, although an IVI driver for basic functionality should not take that much code (much of it is handled by the scpi.dmm classes). I was mostly thinking the other way around: if you want your logger to interface with other meters (e.g. GPIB, TCP/IP or USB), then using an existing back-end like python-ivi, or sigrok (seems mostly aimed at handhelds) or some other abstraction makes sense to me, rather than inventing your own. Granted, DMM support in python-ivi is currently limited to a few Agilent meters and one Rigol (in Agilent emulation mode), so the short-term payoff is quite limited.

Python-ivi is the direction I plan to go if I ever get around to rewriting my logging software. Currently it is using a heavily-modified version of gpib-devices but that is not actively maintained and heavily tied to GPIB (instruments inherit from Device488). Since IVI is an industry standard, it seems like a logical choice. The other abstractions I looked at were more limited or seemed aimed at different types of equipment.

[Muxr]

@alm, that's a good idea. I will look into this further. I would like to abstract the driver part of it away for sure, will play with ivi over the weekend, and see if I can add Fluke8846 and hp3458a to it (my Keithley 2015 bit the dust  ).

The other idea I've been sort of toying with as far as the metrics side is concerned, is using something like TSDB to store all the metrics and maybe use a front end like Grafana. Like everyone else I suppose I suffer from too many projects so it's hard to commit to a bigger idea. This was just a quick effort to get the plotting going for our little USA calibration club.

[cellularmitosis]

The other idea I've been sort of toying with as far as the metrics side is concerned, is using something like TSDB to store all the metrics and maybe use a front end like Grafana. Like everyone else I suppose I suffer from too many projects so it's hard to commit to a bigger idea. This was just a quick effort to get the plotting going for our little USA calibration club.

Ooooooh, interesting.  http://opentsdb.net/

Edit: Holy cow!  https://misfra.me/2016/04/09/tsdb-list/

[alm]

Getting the 8846 to work should be straight-forward, since its SCPI interface will be quite similar to the Agilent 344xx meters. The 3458A might be more work since it does not implement SCPI, so you would inherit from the base dmm classes instead. The agilent.agilent437B driver might be vaguely similar in that it implements a pre-SCPI protocol (it is a power meter, rather than a DMM, though). The scpi.dmm classes might be another source.

TSDB definitely looks cool if you are talking long-term measurements. My needs so far have been served by CSV files that I import in R for analysis/statistics/plotting. Not sure about a standard front-end like Grafana: they look designed for monitoring for large changes like power outages or spikes in data traffic, rather than looking for 0.000001 ppm changes through statistical analysis and relating them to atmospheric conditions.

[Muxr]

The other idea I've been sort of toying with as far as the metrics side is concerned, is using something like TSDB to store all the metrics and maybe use a front end like Grafana. Like everyone else I suppose I suffer from too many projects so it's hard to commit to a bigger idea. This was just a quick effort to get the plotting going for our little USA calibration club.

Ooooooh, interesting.  http://opentsdb.net/

Edit: Holy cow!  https://misfra.me/2016/04/09/tsdb-list/

Yeah, one of the lighter ones would probably be a better idea. TSDB is awesome but it's way overkill running something like a Hadoop backend for this, unless you're doing high frequency measurements on like 100+ sources (hbase might be lighter, although haven't tried it). We use TSDB/hadoop at work but we deal with petabytes of data.

TSDB definitely looks cool if you are talking long-term measurements. My needs so far have been served by CSV files that I import in R for analysis/statistics/plotting. Not sure about a standard front-end like Grafana: they look designed for monitoring for large changes like power outages or spikes in data traffic, rather than looking for 0.000001 ppm changes through statistical analysis and relating them to atmospheric conditions.

Grafana supports plugins, like they even have a Plotly extension: https://grafana.com/plugins/natel-plotly-panel but I hear you on ours being a special usecase.