Electronics > Projects, Designs, and Technical Stuff
Energy monitoring using Raspberry Pi, looking for advice regarding data storage
<< < (3/5) > >>
NivagSwerdna:

--- Quote from: JeanF on October 18, 2018, 01:02:30 pm ---NivagSwerdna, great! How did you like it?
--- End quote ---
It worked.  They have recently changed to a charging model for their hosted solution but the software is opensource so you can run your own instance if you don't want to pay.  I used the hardware as a backbone for publishing of temperature from many sensors.  Works great
mariush:
You could use one of those FRAM memory chips to "cache"... store a bunch of measurements, they have very high endurance and all that good stuff.

For example, get a 2mbit or 4 mbit 8pin memory chip and store and you have 256-512 KB of "safe" memory.

Here's a suggestion link: https://www.digikey.com/product-detail/en/fujitsu-electronics-america-inc/MB85RS2MTAPH-G-JNE2/865-1285-ND/7604212

Assuming your data is something like 64-128 bytes per measurement and you make a measurement every 5 seconds, then you have 12 measurements per minute and  720 measurements per hour ... at 128 bytes per measurement that's 92160 bytes for one hour or less than 100 KB ... you could log 2h+ on a chip.  or if you use 30m chunks of time, then at 128 bytes per measurement you can squeeze ~ 2.5 hours worth of logging before you have to overwrite stuff.

(optionally) Your measurements are most likely going to be highly compressible, since the values won't fluctuate so much, so once an hour you could use zlib or zip or whatever to quickly compress those ~ 100KB of data (30m or 1h worth of data) into something smaller and dump it to some media ... for example save it to 2018/12/25/20181225_1400.csv.zip or whichever way you like it.
you could plug two 2 usb sticks and copy each file to both usb sticks (for redundancy) and if you want you could also upload this <100 KB to a remote website or to your local network.

You could also use a SD card and since you're only going to write once every 30m or 1h, i guess you won't have to worry so much about endurance, and the SD card will probably also be happier if you write to it in 512 byte or 4 KB chunks (basically store big chunks at a time instead of a few hundred bytes every few seconds)

if you're really paranoid, or if you want it to be fully offline, you could get this compressed data (the data packet with one hour worth of measurements), encode it in base64 or something and then send it to a dot matrix printer and have it printed on those long continuous computer paper (~2400 sheets per box ...  assuming one page per hour, that's 100 days worth of logging)
If needed you can then simply scan back the paper into a computer and use some OCR software to convert back to data.
I was thinking thermal printer since they're cheaper and you don't have to worry about ink but I'm not so sure about long spools of thermal paper and how much you can fit on a "receipt" width style paper 


When you get the data, you can simply extract all the CSV files and you mass import them into a database (mysql for example) or you can join all csv files in a big csv file and import that into excel or libreoffice calc or whatever

I'd stick to CSV files or TSV (tab separated values) since they're humanly readable, easy to compress, safe to use padding to have each record of equal length if you so desire
JeanF:
ljwinkler, thank you for your tips (good job with the fixed USB peripheral name, I plan to use this trick as well), that's a nice project you have here. Concerning data visualization, my needs are pretty basic :) I was indeed planning to dump a CSV file into Excel, because as I mentioned the primary goal is just to check if (and when) the voltage goes under a predefined value, and when that happens I may check the other values, such as instantaneous current, in order to estimate whether the voltage drop is primarily caused by a heavy load in my parents' house or in other houses around.
I don't know have experience in creating nice dashboards. HoracioDos suggested Grafana and it's not the first time I hear about it, I also saw it being used here. But I don't know if it could fulfil your needs about interactivity and not-hardcoded ranges, I haven't tried it. Also this guy used PHP/SQLite and Google Charts to display his curves (in French)

I tried Thingspeak, it is very easy to use indeed. I just created an account, used curl to send test values and that's it. At first glance, the included visualisations seem pretty basic but I'm not too worried as there is a CSV export button available.

However, I'm curious to see what you finally choose to use for visualisation because in version 2 of this project I may need that as well. Currently, my parents have a Geo Minim+ energy "monitor" (just a simple LCD display really) to help them manage their electricity consumption (mainly because there are more than 17 kW of heaters in the house and they have a 12 kVA contract, so obviously they can't all be turned on AND at full throttle at the same time), but this meter is not accurate because it only monitors current; not voltage nor power factor. So the readings are a fair bit off, particularly with large voltage swings. If I manage to get a Raspberry Pi to grab all the real values from the official energy meter, I could as well build another little device to display them instead of the Minim+. I was thinking maybe an old Android phone connected to wifi and displaying a webpage hosted on the Raspberry Pi. Otherwise, an Arduino or another Raspi Zero with a small Oled display could also work. All of that isn't going to happen soon, because my HTML skills are near zero for now.

If anyone has suggestions for that matter, I am interested as well! :)


mariush, thank you very much for this detailed post. I really appreciate your suggestions. I don't have any experience with that kind of external memory chips. I assume they could be useful additions to standalone microcontroller projects as well. In the rest of your post you talk about USB ports, SD cards etc so I assume that the context of your reply is: using a Raspberry Pi to run the show, and not going completely custom with any µC and doing all the firmware myself. Am I correct? But then, what is the advantage of using one of those FRAM chips instead of the onboard RAM, before sending to USB or to a remote server?

Thank you for your idea of printing, I won't go into this level of redundancy but it is a fascinating idea, I've never heard of something like that. I will also see if I can find any data concerning the reliability of USB sticks vs. SD cards. Anyway, nice addition to have 2 of them. I will also keep the idea of data compression if needed.

Again, thank you everyone for your support :)
cheers
mariush:
I was thinking things from a reliability point of view and as means to reduce the number of writes to media which wears out ...

If you don't have a UPS or something that would hold your pi up for a significant amount of time, then whatever you have in your pi's ram would be lost on power failure. That FRAM memory has a very high endurance and the writes complete practically instantly, while your pi may cache writes to the flash media so if you take one measurement every second you may lose a measurement or two.

If there's a power loss, at next reboot your code can simply scan the fram chip and determine the last logged entry and resume storing measurements from that point and once you have enough measurements saved, you can get them and zip them and save them to flash or some removable media or send them through network somewhere and then reset the fram chip (just erase the data)

if you don't care about small power failures or if you can get a tiny UPS to keep your pi running for a few hours (or basically a 12v lead acid battery that's always charging from a mains charger would have enough charge to keep your pi happy for probably even a day) , then yes you could just hold the measurements in the pi's ram.
 
I have to admit i was also thinking how i would be doing this using a cheaper microcontroller with less ram available, and using something like miniz (zlib ,deflate algorithms) to compress data before you send to a SD card... forgetting you actually have the pi.
sokoloff:

--- Quote from: JeanF on October 20, 2018, 09:49:44 am ---Currently, my parents have a Geo Minim+ energy "monitor" (just a simple LCD display really) to help them manage their electricity consumption (mainly because there are more than 17 kW of heaters in the house and they have a 12 kVA contract, so obviously they can't all be turned on AND at full throttle at the same time), but this meter is not accurate because it only monitors current; not voltage nor power factor. So the readings are a fair bit off, particularly with large voltage swings.
--- End quote ---
If the heaters are resistance heaters, you can model them as having a power factor of 1. I would be surprised if your voltage fluctuated more than 10% total swing, so you should be able to get "engineering close" by just monitoring current if that means the difference between successfully completing the project versus not.
Navigation
Message Index
Next page
Previous page
There was an error while thanking
Thanking...

Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod