Electronics > Projects, Designs, and Technical Stuff
Energy monitoring using Raspberry Pi, looking for advice regarding data storage
(1/5) > >>
JeanF:
Hello,

I would be glad to hear your suggestions about data storage with a raspberry pi, either locally or on a server/some sort of cloud  :)

I would like to log data from two sources, both having serial ports, and to store them somewhere with timestamps. I'd like to be able to read the data later on via remote access. Because that's what I have on hand and because the elegance of it is not hugely important (it's a one-off, for hobby purposes), I plan to use a Raspberry Pi in addition to two UART to USB boards, and connect the RasPi to the router in the house.

There is not much to store, only a handful of values (voltage, current, active power, apparent power, measured at one single place for the whole house) every few seconds. This will have to last only for the winter so even if I go overkill and log every second for 4 months, we are talking about a few hundreds of megabytes at most, if I didn't completely screw up my estimation.  ;D

I don't really need fancy displays or web interfaces, I would just need to come back after the winter to analyze data (Excel would be fine). I heard that logging directly to SD card could have reliability issues with the SD card wearing out, and that making the card read-only could help. What do you think ?

I saw some blog posts with guys doing it on their own in PHP, storing data in a MySQL database, but sadly I think I don't have the experience required to dwelve into that... I heard (in no particular order) about EmonCMS, Node Red, InfluxDB, ThingSpeak, Domoticz, Jeedom ; but I don't have any experience with any and I am a little bit confused.

What would be, in your opinion, the easiest way to store the values ? Any advice or any suggestion will be appreciated. Thank you ! 8)

PS : the house may encounter rare power outages. There will be a small UPS for the Raspberry Pi, but not for the router; if I choose to store values on a server over the internet, there must be some sort of buffer to keep the values and to upload them later.
JeanF:
A little bit more background about this project, for those interested ; it is very optional :

At my parents' house, they have electric heating (The french government heavily promoted it in the 70's to justify investments in nuclear power). They live in a small, yet quite touristic place in the seaside. In summer, no problem ; but during the winter, the local substation is struggling to cope with all the people turning on the heaters for the holidays in their vacation homes.

As you all know, P is proportional to V squared so it doesn't take much of a voltage drop to drastically reduce the heating capabilities in the house. Add that to a poor insulation job, and all in all the temperature can quickly become uncomfortable (in addition to very dim lighting...). The national energy provider is required to beef up his wiring or tweak the transformer taps if I can prove that the supplied voltage goes below 230V -10% tolerance = 207 volts. If I call and tell them the problem, they will (after a fair bit of negotiations) send someone to check the voltage at the "Place of Delivery". But they are smart and will come on a warm sunny day when the voltage is perfectly fine ; that way they can get away without doing anything (and they will charge me for the trip).

So the idea is to log the network voltage during the winter and then see if (and when) voltage drops occur. That way I will know if there is a real problem or if my parents are just getting sensitive to the cold  ;D and if I have to negotiate with the power company, I will know when to make them come and check.
georges80:
A simple/cheap serial (uart) logger is something like the Openlog design - lots on ebay/china units etc.

Logs to a uSD card and can handle FAT32 so gigabyte sized files. Pretty cheap, $12 US on prime and in the $5 range for China ebay. Super easy to use. It takes care of creating the filename and writing/logging data. Just edit the config.txt to set baud rate etc. Easy.

https://github.com/sparkfun/OpenLog

Flash doesn't wear out unless you are rewriting a LOT of data to the same page, for a logger this isn't going to be an issue.

cheers,
george.
hamster_nz:
If you have internet access, and you want a low fuss cloud data solution, have a look at ThingSpeak - here is one of my feeds: https://thingspeak.com/channels/354869

All you will need to do on the Pi is have the program monitoring the serial data perform a web request - e.g run a system command to run "curl https://api.thingspeak.com/update?api_key=[your key]&field1=[your value]&field2=[your value]".

Free accounts are limited to one message every 15 seconds. I would suggest that you log "min", "average" and "max" values for each minute. And also add a DHT11 sensor so you can do temperature and humidity too :-)

PM me if you want a hand.

capt bullshot:
Yes, SD cards can be quite unreliable, especially when data gets written often to them.
You could connect an external SSD with an USB to SATA converter (or just an USB HDD) to the Raspberry and use this to store the data.

For the data storage software, take a look at RRDtool: https://oss.oetiker.ch/rrdtool/
Navigation
Message Index
Next page
There was an error while thanking
Thanking...

Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod