Wireless weather station

[ljwinkler]

As TheLAMPA project needs to be put on hold – some parts are on the way, I have decided to start a new one.
This time it will be a solar powered wireless weather station. The concept is:
1) 10-12 transmitters with solar panels + rechargeable batteries. Each transmitter would send: temperature, humidity, light and battery voltage.
2) 1 main station that collects and processes all the data.
3a) add a screen to the main station to present the data + SD card to keep the recordings/draw history plot.
3b) use a cheap tablet just to  present the data (this would save me a lot of time reinventing the wheel as I would have a large screen with proper driver, lots of memory, nice casing, SQLite, etc. right out of the box.
 
Thoughts:
1) I’m thinking of ATmega328P, sleeping constantly and waking up every 10-20 seconds, powering up all the sensors, reading temperature from DS18B20, humidity from DHT11, light amount (LDR), voltage of the battery, sending a package over NRF24L01+, turning off all the peripherals and going back to sleep. This would be powered from a  solar mobile charger that has built in Li-ion or LiPo battery.
2) Main station would never sleep, constantly waiting for incoming packages from sensors. Each sensor would have unique ID so it will be easy to understand where the data is coming from. Additional it would have a pressure sensor (probably BMP180).
3a) Long and painful process to implement all the libraries for SD card, graphical LCD, etc. Also, smaller (and cheaper) screens will not show all the data (10 sensors sending 4 values = 40 numbers to present). That’s why I think solution 3b is better
3b) I would need to add Bluetooth support (probably HC-06) to the main station and then pair it with a tablet/phone. Additional piece of work would be to write a mobile app to present the data but in my opinion it is much easier task than doing it on a microcontroller. Also, the crappiest tablets have at least 256MB of ram, 4-8GB of storage with a screen of at least 1024x600 (most of the time it is 1280x800), clock, nice cover, charging circuit, possibility to install additional apps/libraries when required, SQLite to keep all the historical data. Let’s say that each sensor sends:
- ID (1-15) – 1 byte (unsigned byte)
- Temp -30.0C...+50.0C without decimal point (-300...+500) – 2 bytes (signed char)
- Humidity (00-99%) – 1 byte (unsigned byte)
- Light (v1) (0-1023) – 2 bytes (unsigned char)
- Light (v2) (00-99%) – 1 byte (unsigned byte)
- Voltage without decimal point (500 for 5.00V) – 2 bytes (unsigned char)
It means I would need 8 bytes of storage per record. 10 sensors = 80 bytes. Even with sending the readings every 10 seconds, I would need to store 80b x 6 times a minute = 0.5kb per minute, which gives 720kb per day. So it is 250MB per year. With 4GB of available memory I could store up to 16 years of historical data. 16GB SD card would provide a lifetime storage.
 
I appreciate any comments.

Cheers,
LJ

[stevech]

With a LaCrosse no-wires weather station including rain gauge, anemometer, barometer, indoor/outdoor temperature/humidity, nice LCD display, wireless link to PC, costing me under US$100 at Costco, DIY makes little sense. Can concentrate DIY on what happens to the data once logged to a file on the PC.

https://www.lacrossetechnology.com/2813/

The MSRP is high as above, but for some reason, Costco had a really low price.

[PeterFW]

A ATMega328p will run out of flash memory very quickly.
After adding a hello world for the grapic LCD and data/time logging to a SD card with FAT16 file system all i had left were ~4kb iirc, that leaves no room for annything fancy.
I switched to a ATMega2560, i wont need all the IO but 256kb are much, much better.

[ljwinkler]

@stevech – thanks for the link but this is not exactly what I need (and also, only the wind sensor is solar powered, the termo/hygro takes 2x C batteries). I need to connect 10-20 sensors to monitor parameters in different rooms.

@PeterFW – yeah, I fully agree with memory usage but I’m not going to put everything into a single microcontroller (this was the option 3a but I have disregarded it very quickly). I’m going with 3b for sure. It means the receiver will only collect the data from all the sensors and transmit it over Bluetooth to a tablet (which has a built in RTC, massive LCD, implemented SD card and FAT support and 8/16/32GB storage, rechargeable battery, charger and a nice/thin enclosure right out of the box). The prices start at €100 which will be much cheaper than building my own ATmega2560 + PCB + LCD + SD + enclosure + hours spent on development



Over the weekend I have built a prototype of a sensor/transmitter.  The only job it will have to do is:
-  wake up periodically
-  power up the sensors
-  read: temperature (DS18B20), humidity (DHT11), light (LDR), battery voltage
-  send a package using NRF24L01+
-  power off the sensors
-  go back to sleep

Having this setup on a breadboard I have written a very simple program using Arduino (my goal is to write optimised code in C, but at this stage I just wanted to prove the concept).

After compilation, Transmitter.ino takes 14kB (15k with serial output) of available 32kB flash in ATmega328P. One thing I don’t like in my program is that I have hardcoded the sensor ID – it means that if I wanted 10 sensors I would need to flash 10 different programs.

I think I will add 4 or 5-way dip-switch allowing me to choose one of 15 or 31 addresses (the receiver has ID=0 hence only 15/31 are left for the sensors).
 
I hooked it up to a solar power bank (it has 2600mAh LiPo battery). I have connected the uC directly to the battery (4.1V max. once charged) but I had to add 1N4148 diode before the radio transceiver (it operates at 3.3V). Using a variable PSU I have discovered that everything works fine until 3.1V (minus voltage drop on the diode for the radio), then the DHT11 humidity sensor goes mental (0% humidity) and then at 2.9V the radio stops transmitting (which is good as I’m already getting rubbish readings at this voltage).

The solar bank circuit shuts off the step-up converter if there is nothing plugged in to the 5V USB port (which is good in my case as I’m connecting the uC to the battery directly) but the solar circuit works and charges the battery if enough light is provided.

The funny thing is that it has a built-in LED indicating charging the battery.
In specific conditions (little amount of sunlight) the solar panel was producing enough voltage to trigger the LED on, which was draining the built-in battery...
I don’t have exact figures off the top of my head but it was something like 3mA going from the panel and additional 6mA from the battery (?!?). After cutting off the LED, the current started to flow into the battery ?
It wasn’t of course a problem in a direct sunlight but even then, the LED was consuming more energy than my complete circuit will.
 

The receiver is just another Arduino with NRF24L01+, formatting and forwarding the packet to the serial port. Receiver.ino – 8kB of flash (or 10k with serial output).

Here is an example of what comes out of transmitter:

27: 220C 3323V 40% 299
WRITE OK
 
27      - sensor’s packet counter
220C      - temperature: 22.0C
3323V   - battery voltage: 3.323V
40%      - humidity (0-99%)
299      - light amount (0-1023)
WRITE OK   - NRF24L01+ acknowledgment


And here is how it looks in a serial monitor on the receiver’s side:

260-27: 1  220C  3323V  40%  299

260      - receiver’s packet counter
27      - received sensor’s packet counter
1      - senor ID (extracted from transmission header)
220C      - received temperature: 22.0C
3323V   - received voltage: 3.323V
40%      - received humidity (0-99%)
299      - received light amount (0-1023)
 

The next step I’m thinking of is building a second sensor/transmitter and a better, standalone receiver, probably with some LCD displaying the readings.

   
All comments/support appreciated as usual.
 

Cheers,
LJ

[PeterFW]

One thing I don’t like in my program is that I have hardcoded the sensor ID – it means that if I wanted 10 sensors I would need to flash 10 different programs.

High value voltage divider and cap connected to one ADC pin, at only 10 adresses you can get away with low tolerance parts.
That way you can read it once during boot, store it, and each module has a unique hardware ID without bulky parts or high cost.

Other solution, store the unique ID in the EEPROM and give each module a common broadcast adress, have only one module running, broadcast a "set ID" command or every module gets a pushbutton and will wait for a new ID broadcast to it after the button was pressed for 10 seconds.

Of course you could just programm the EEPROM of each device with the ID and just read it, that way you need to connect it to the programmer to change the ID.

[ljwinkler]

After parking this project for a long time, I have decided finally to give it a go and complete this time.
The latest approach is:
1)      having approximately 20 sensors across the house/outside.
2)      records stored on Raspberry Pi running MySQL.
3)      data presented on a tablet as a web-based dashboard (haven’t decided it yet).
 
1)    I believe 20 sensor nodes should be enough to have them in various places around the house as well as around it. The design supports up to 32 sensors due to having 5 DIP switches allowing selecting 2^5 possible combinations. Each node will measure temperature and humidity using SHT21 sensor, crude light level using LDR and battery voltage (2x AA). Some of the nodes will be equipped with an e-ink display to present the measures.

2)    I’m currently testing RPi 3B+ but I may pick an earlier edition as the current one runs very hot. I may not need such performance just to read serial data, insert it into a MySQL table and run a webserver that will show just few pages in total.

3)    I’m thinking about Apache + PHP currently but if you have any suggestions -> please let me know.


Let me explain the above in details.

1)    I have started developing it using ATmega328P with Arduino bootloader. As this will be battery operated the MCU needs to be the “P” (Picopower) version. And of course, forget about using Arduino development board as there are plenty of power hungry devices (LEDs, USB communication, voltage regulator) that will eat 2 AA batteries in a couple of days.
ATmega328P on a breadboard with internal oscillator while in a deep sleep mode consumes less energy than a battery self-discharge is. Here is a great article of power saving techniques: https://www.gammon.com.au/power

The node will sleep for most of the time, waking up every minute, taking up few measurements, sending it wirelessly, updating the display and going back to sleep).
As the RFM69H radio as well as the WaveShare 1.54” e-ink display were new to me I have decided to test them separately to see if I can talk to them successfully rather than connecting everything together and wondering if for example the SHT21 is not responding because of display’s IRQ pin is assigned incorrectly in the program…
For the radio part, I have used the LowPowerLab’s RFM69 library (https://github.com/LowPowerLab/RFM69)
I have built a simple RFM69 transmitter and receiver to see if I can communicate between them.

Once the communication was confirmed I have added some complexity to the program. I’m planning to use a struct with all the data I need to send:

Code: [Select]

// packet structure
typedef struct {
       uint8_t       nodeId;
       uint32_t      uptime;
       int16_t       temp;
       uint8_t       humid;
       uint16_t      voltage;
       uint16_t      light;
} Payload;
I have also utilised transmission encryption (not that the temperature readings are sensitive data but having this working may be a very useful feature in the future).
The next step was to mock-up a fake sensor node that transmits a ‘real’ packet that I will use in the future but with some random data. The receiver got all the measurements as transmitted so at this point I decided that I have all I need to transfer the data and the RFM69W radio works along with my requirements.
 
Now, let’s try the e-ink.
First, I tried the code provided by the manufacturer (https://www.waveshare.com/wiki/1.54inch_e-Paper_Module) just to see if I can get it to work. Once proved, I have downloaded a great library (https://github.com/ZinggJM/GxEPD) that supports various models of displays. Having this connected in slightly different way and selecting the correct display I got the demo program working as well. The downside of e-inks is that they are very slow to refresh (which means long power draw). The good thing is that the library supports partial refresh, so only a small section, i.e. a digit that changed, can be redrawn – much quicker which means less power needed for the process.
As ATmega328P has very small amount of memory I couldn’t use one of the available font packs for ESP32. I decided to make my own font as I could trim it to my needs and I could make it to fit best on the screen.
As the screen is 200 x 200 px - I need to fit the longest string (i.e. 20.0*) in one line. I plan to put displays only on the nodes around the house and I don’t expect the temperature to fall below 0 (I hope not!) so I omitted negative temperatures/characters to be displayed.
A thing worth keeping in mind is that the partial refresh is 1 byte wide (so it refreshes 8 pixels in a row). This means that having a digit 9 pixels wide will ‘overlay’ the next 7 pixels on refresh. Ideally the digit’s width would need be in 8 bit increments (so 8, 16, 24… bits wide).
I have created an Excel spreadsheet where I designed my own digits. It helped me create the array of 1s and 0s to build a digit.
I use the same font for digits 0-9 for temperature and humidity, a smaller font to represent battery percentage. Then I have created some characters: a degree sign, decimal point, big percentage (humidity), small percentage (battery) and a small battery icon.
I can now read SHT21 and send it over RFM69 and (separately) read SHT21 and display the values on e-ink.

Let’s combine these 2 tasks, shall we?

First bump: once I initialised RFM69 radio, the display stopped responding. When I unplugged the radio from my breadboard the display came back to life. Uhm… pins conflict, SPI maybe? I had to redefine the RFM_IRQ_PIN to a different one (digital 10 in my case) to get the radio and display working together.

Second bump: once I displayed the 3rd digit the program crashed. I was able to display max. 2 digits on the screen, doesn’t matter which ones, it could be tens from humidity and a decimal place from temperature… all worked fine. Adding more the to the screen (battery %, an icon, 3rd digit or a degree symbol) caused the same behaviour… When I enabled detailed information after compilation (in Visual Micro) I noticed that I’m using 97% available RAM and 90% of available flash memory. Not good. I haven’t even implemented power saving functions or properly utilised reading light and supplied voltage level. I probably could save few bytes here and there by optimising the code slightly and/or using smaller fonts but making the numbers less visible from a distance and still balancing on the edge of memory capacity wasn’t good for me. I have decided to go for a bigger micro.

I went for ATmega1284P-PU as:
- P - PicoPower
- PU - DIP package (the main size factor is 2x AA batteries so going into SMD won’t help much)
- 128kB flash (vs. 32kB in ATmega328)
- 16kB RAM (vs. 2kB in ATmega328)
- has an Optiboot bootloader (https://github.com/maniacbug/mighty-1284p) and can be easily programmed by serial connection
 
I had to change fuses slightly as I needed to run it on internal 8 MHz oscillator, have the BOD disabled (saves power consumption and also, the levels aren’t good in this application, the 1.8V is too low for the SHT21 and RFM69 to operate anyway and the next one – 2.7V would be reached after a while of running the node on 2xAA batteries (3V in total)) and change the boot up time to 0ms (from 65ms by default). The fuses I use are:
 
low_fuses = 0xD2
high_fuses = 0xDE
extended_fuses = 0xFF

2)    The latest available Raspberry Pi (3B+, 2018) is slightly faster than the previous one but I may not gain much using it as the project is simple and doesn’t require a massive amount of computation power.
I have installed raspbian-jessie, configured the standard way (SSH, expand size, lower GPU memory, etc.), removed the default pi user for security reasons, installed python (receiving data over USB), installed MySQL and created an empty database.
From this point I can use HeidiSQL to connect to the RPi  from my PC and start the database development comfortably (rather than using a command window ).
I expect to have 20 sensors, a ‘Live’ dashboard that will display the current measurements as well as a 2-day history (or maybe a weekly as well) to compare the values across days. My approach is:
- A python script will find an active USB connection (using USB2Serial board’s VID and/or PID), open the port and will be waiting for any incoming transmission. It will then check if the received string (i.e.  1,234,56,222,1023 -> SensorID (1), temperature (23.4C), humidity (56%), battery voltage (2.22V), light level (0-1023 from analogue pin) has all the values, are in correct ranges, etc. it will issue then an Insert statement to the MySQL database. Each sensor will send 1 message per minute.
- In the database I’m going to create a small table which will store only the latest record (once a new value is received it will replace the existing one) – this will allow me reading all the records without finding the latest record received along with thousands of received records (a sensor may hang or get transmission interrupted, etc. so the latest entry could be hours (or days) old. Also, I’m planning to archive this table periodically so if the sensor won’t start sending details again it may be archived completely and the table won’t store its entry. In the ‘latest record’ table however, the record will always be there.
- The 2nd table will store all the entries from last 2 (or 7 days), this will be used to draw a daily dashboard (comparing yesterday’s day vs. today’s (from midnight up to now).
- Then, the archive table. I’m going to aggregate data from the above table by SensorID, record date and time (rounded to 15 minutes). I will average records split by 15-minute chunks. I don’t need to know what happened in the kitchen 20 years ago at 8:02 in the morning    An average of temperatures between 8:00 and 8:15 will be more than enough. And this will cut the number of records from 1,440 (24 hours * 60 minutes (if sending once per minute)) per sensor to 96 (4 averages per hour * 24 hours). Having 20 sensors, the annual record count will be reduced from 10M to just 700k. I’m not worried about the storage used, I’m going to use external 120GB SSD. The problem is MySQL performance on RPi. I found that it struggles where a table gets to 50M+ records. I tried index on date and time for reporting purposes as well as a table partitioning on year (2018-2100). I have loaded millions of dummy records, it created 10GB file on the SSD and any more complex query was taking noticeable longer.

3)    At the beginning I was thinking about an Android app that would pull data over the network and display the results. Seems having a webpage running full screen is much simpler solution. I’m going to create one main dashboard where all the temperature/humidity values will be displayed (maybe in some sort of a drawing of the house), like a top level summary. I may change the font colour depending on a temperature, have a battery indicator (just 3 colours: GREEN if the battery is >80%, YELLOW if between 80 and 30 and RED if <30%) and maybe transmission indicator (GREEN if the most recent entry for the sensor came within 10 minutes, YELLOW if less than 1 hour (to indicate that there is some interference, transmission loss, etc.), RED if the sensor didn’t send (or rather the receiver didn’t receive) a packet for more than an hour).  Then, additional pages could show more details, graphs, averages, history, etc.

I’m open to suggestions if you have any or you may have better solution to the above.

[schmitt trigger]

I once built a weather station, and taking readings with periods shorter than 10 minutes is an overkill.
Weather does not change that fast.

[glarsson]

I once built a weather station, and taking readings with periods shorter than 10 minutes is an overkill.
Weather does not change that fast.

To register wind gusts you need the wind sensor to sample way more often than every 10 minutes. The transmission from the sensor to the station can be done with longer intervals.

[floobydust]

I think the energy budget is the hardest part, Canadian winters are very cold and you can get easily a month of no sun where most battery chemistry's freeze.
You need enough energy to run for at least a month (cloud) in cold weather, but I'm not sure about your homeland.

Shipping the data out, the RF transmit pulse uses the most energy. The time to get on the network and complete one transaction can have a lot of overhead, like an ESP8266.

If the datalink is temporarily lost or noisy, do you locally store data and later try to catch up, or just let it get lost.
I give it three tries with random waits, then give up for a long time in order to save batteries instead of continually trying to send a packet, and killing the battery.
Many systems have that bug, they try forever to send data to get the ACK.

[ljwinkler]

That's why I have decided to skip the 'solar' part from this project. I found that having SHT21 + ATmega1284P + RFM69W (all in sleep mode) takes 11uA @ 3V. Adding e-ink brings the current to 35uA. For 2xAA this gives 10 years in sleep mode for the e-ink version or 30 years for the version without it. This of course are just rough calculations, don't include self discharge or poorer performance in cold conditions. Just wanted to see what's the possible out of it. I'd be happy if it works for one year on 2xAA sending all the measurements once per minute.
Once I have some units assembled I will run a longer test to see the average current.

Network -> I have selected RFM69W modules as it takes just few ms to send the packet. All you need to do is to let the module know to send the packet on a specific NETWORK_ID to a specific GATEWAY_ID. You can even disable ACK. The modules itself work fine up to -90dBm on the built-in antenna. Got it working without issues up to 30m in the house, through walls, etc. No packets lost. I've added 86mm antenna (1/4 wavelength) just to be double sure.

I try to send it 3 times if no ACK received and then give up, go to sleep.
I'm going to have a timestamp of the latest transmission received (per sensor) so if a sensor has gaps or didn't send a message in let's say 10 minutes, then it will be flagged.

[NorthGuy]

I think the energy budget is the hardest part, Canadian winters are very cold and you can get easily a month of no sun where most battery chemistry's freeze.
You need enough energy to run for at least a month (cloud) in cold weather,

I have a solar-powered weather station - works fine for many years - withstood temperatures from -47 to +38 C. Although Albera is much sunnier than the rest of Canada

[ljwinkler]

How do you store the energy?
I was thinking about LiFePO4 as standard lithium batteries get swollen after a couple of years. Then having a 50cm x 50cm solar on each sensor, charging circuit, etc...

[schmitt trigger]

I once built a weather station, and taking readings with periods shorter than 10 minutes is an overkill.
Weather does not change that fast.

To register wind gusts you need the wind sensor to sample way more often than every 10 minutes. The transmission from the sensor to the station can be done with longer intervals.

Absolutely correct; however I was following the OP's original requirements on post #1:

 Let’s say that each sensor sends:
- ID (1-15) – 1 byte (unsigned byte)
- Temp -30.0C...+50.0C without decimal point (-300...+500) – 2 bytes (signed char)
- Humidity (00-99%) – 1 byte (unsigned byte)
- Light (v1) (0-1023) – 2 bytes (unsigned char)
- Light (v2) (00-99%) – 1 byte (unsigned byte)
- Voltage without decimal point (500 for 5.00V) – 2 bytes (unsigned char)

[ljwinkler]

Hi again,

Now, after few months I got a prototype working, got all the sensors built (ordered PCBs, assembled and programmed each node, printed cases, etc.) and the data is flowing nicely into MySQL sitting on a Raspberry Pi:
- I have a table where I store raw data coming from sensors (sensor ID, temperature, humidity, light level, battery level, packet number, RSSI, timestamp).
- 2nd table shows aggregated data (15 minutes chunks) for last 2 days) - the idea was to use it as a comparison "yesterday vs. today" - 20 sensors x 48 hours x 4 15-min chunks = nearly 4000 records - this is very fast to retrieve from the database. After every midnight the oldest day is moved the the historic table (below)
- 3rd table has a similar structure to the one above but stores historical data (from before yesterday). The table is partitioned by year so each partition will hold about 700k records - it works all right while retrieving data ad-hoc, but it is not good for 'live' refreshes.
- 4th table stores only one reading per sensor with a timestamp when it was received. Any new transmission overwrites the values and updates the timestamp. This table draws my 'Latest entry' view. I couldn't utilise the Live view (last 2 days) as if there is a problem with a sensor for more than 2 days, the data will be moved to historic table and I will have no data at all for the faulty sensor. Having the "latest" table I will see that the last response from i.e. Kitchen was 5 days ago. I will also use it to indicate on the dashboard how recent is the value... (I was thinking about green-amber-red for <5 mins, <30min and >=30 min.

I have a 10" touch screen connected, rotated 90 degree (so it stands vertically) and I made a quick webpage that utilises charts.js and draws the latest readings in a single chart.
But... I have no idea how to progress from now...

In my head I have:
- a single page with Date/Time on top, a sketch of the floor layouts with temperature/humidity/battery/recent reading indicators in each of the rooms (I believe this will 99% on the screen), refreshing the contents every n seconds,
- a page with charts for the last... day? week? for all the rooms,
- a page where I could pick what I want to compare with (i.e. correlation between temperature and light amount in a shed, etc),
- I'd like to avoid using a mouse or a keyboard, so it would need to be a simple user interface, just swipe left/right between screens, maybe few buttons or simple drop down lists to pick a room, date range, etc.

It is a very good case for Tableu, Qlik or any other fancy BI tool that... is resource hungry and won't run on RPi

If you have any suggestions, please let me know.

Cheers,
LJ