Author Topic: what is inside a data logger ?  (Read 3062 times)

0 Members and 1 Guest are viewing this topic.

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
what is inside a data logger ?
« on: March 08, 2018, 10:40:01 pm »
hi

what is the physical layer of a data logger exactly ?

my product should have a reliable an robust data logger . how should i Implement it?

i it a good idea to connect SD card directly and simply to MCU ? or it should be a more complicated circuit?
 

Offline Rick Law

  • Super Contributor
  • ***
  • Posts: 3423
  • Country: us
Re: what is inside a data logger ?
« Reply #1 on: March 09, 2018, 01:52:45 am »
hi

what is the physical layer of a data logger exactly ?

my product should have a reliable an robust data logger . how should i Implement it?

i it a good idea to connect SD card directly and simply to MCU ? or it should be a more complicated circuit?

I think the lack of reply here is because no one knows what you are trying to log, how is the log data given, how much you want to log, and how frequent you want to log...  So on, so on, so on.

The quality of answer depends on the quality of question.  A general question like "should I buy a bike, an electric car, or a motorcycle" without any further info will result in a rather useless answer.

So, if I may suggest, please try to give us more information here.  Most likely, someone here will forgive you a good answer.
 

Offline Brumby

  • Supporter
  • ****
  • Posts: 12288
  • Country: au
Re: what is inside a data logger ?
« Reply #2 on: March 09, 2018, 02:47:50 am »
The quality of answer depends on the quality of question.  A general question like "should I buy a bike, an electric car, or a motorcycle" without any further info will result in a rather useless answer.
.... if any answer is offered.

Yes, more information is needed.

Things we need to know include:
The type of data being logged - Temperature?  Air Pressure?  Hydraulic Pressure?  Voltage - mV or kV?  Fluid level?  GPS location?  Altitude?  Vibration? .... Anything you can measure can be logged.
The environment:  Indoors/outdoors?  Office?  Aircraft?  Weather balloon?  Whale tracker?  Jungle?  Desert?  Army Tank?

All these are some basic examples of the sort of things that need to be stated ... and there will be more questions once these have been established.
 

Offline Chris-IP5

  • Contributor
  • Posts: 19
  • Country: gb
Re: what is inside a data logger ?
« Reply #3 on: March 09, 2018, 04:27:45 am »
I thought about how these devices remember what they logged. There's a Wiki page with lots of snippets of information. It points out the storage could be by means of: backed static random access memory, flash memory or EEPROM..."
And of course remember those strip chart recorders? Many of those must be antiques now.

The Wiki page is https://en.m.wikipedia.org/wiki/Data_logger

OP are you looking for IC part numbers and so on?
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #4 on: March 09, 2018, 08:19:10 am »
yes you are right i should give more information

actually my system is a tariff device. it measures water fluid and log it during day and night and it should work for a long time so it should be durable  and also pass some environmental tests

operation temperature : -40 to 85 Celsius degree
and pass these tests:
Radiated electromagnetic field
Conducted electromagnetic field
Dry Heat test @70ºC
Cold test @ -40ºC
Vibration test
Mecanical shock

The operation  environment:  outdoors  (Jungle ? Desert , farm , ...)

also i should mention that i want a chip or IC  and  implement my own data logger on the main pcb of me system

so what is the best storage element for my system?

please inform me if there is other specifications that i show determine to make you able to help me

« Last Edit: March 09, 2018, 08:25:03 am by alireza7 »
 

Offline Rerouter

  • Super Contributor
  • ***
  • Posts: 4694
  • Country: au
  • Question Everything... Except This Statement
Re: what is inside a data logger ?
« Reply #5 on: March 09, 2018, 08:38:42 am »
You failed giving any kind of indication of how much data are you logging, and how frequently,

If its a tiny amount of data, e.g. 8 bytes every minute, you could go with a NOR flash memory chip, low speed but guarantee no dead cells. that memory would likely outlast the device.

Equally to add to the math is how often you expect to download the devices, and how long you want them to hold the recordings for.

You will also need to plan space for data verification, e.g. checksum, inverted copy of each record, or a combination.
« Last Edit: March 09, 2018, 08:48:23 am by Rerouter »
 

Offline capt bullshot

  • Super Contributor
  • ***
  • Posts: 3033
  • Country: de
    • Mostly useless stuff, but nice to have: wunderkis.de
Re: what is inside a data logger ?
« Reply #6 on: March 09, 2018, 08:40:31 am »
So in this case, my first question is:
What kind of regulation / applicable standards are involved with this project? Metering in general is heavily regulated and depending on the target location (country specific), so at first you'd have to check which standards apply.
Is the data logger part of a metering device, and may the logged data treated differently from the meter data?
Will the logged data be used to turn mass flow into cash flow at some point?
What amount of data is to be logged and kept for what time interval?
What is the interface to read out the logged data?
I guess, power consumption will be of great matter, such stuff usually is operated from a non-replaceable battery for it's lifetime.

In general, you'll have to apply and verify some techniques to ensure data aren't lost and have some level of confidence. That'll work on any storage device of your choice, so usually the choice of the data storage is driven by economical reasons (lowest cost) and data integrity is shifted into development (usually SW and paperwork).1
Safety devices hinder evolution
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #7 on: March 09, 2018, 09:33:31 am »
You failed giving any kind of indication of how much data are you logging, and how frequently,

If its a tiny amount of data, e.g. 8 bytes every minute, you could go with a NOR flash memory chip, low speed but guarantee no dead cells. that memory would likely outlast the device.

Equally to add to the math is how often you expect to download the devices, and how long you want them to hold the recordings for.

You will also need to plan space for data verification, e.g. checksum, inverted copy of each record, or a combination.

 how frequently : i don't need high data bandwidth, it is sufficient to be capable of read or write data every 30 second in the rate of 0.5kB/s

how much data are you logging : it is sufficient to be capable of storage 256-Kbit

can you have a discussion on flash memory vs EEPROM  ? which one is more appropriate for my system?
« Last Edit: March 09, 2018, 09:54:02 am by alireza7 »
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #8 on: March 09, 2018, 09:42:25 am »
So in this case, my first question is:
What kind of regulation / applicable standards are involved with this project? Metering in general is heavily regulated and depending on the target location (country specific), so at first you'd have to check which standards apply.
Is the data logger part of a metering device, and may the logged data treated differently from the meter data?
Will the logged data be used to turn mass flow into cash flow at some point?
What amount of data is to be logged and kept for what time interval?
What is the interface to read out the logged data?
I guess, power consumption will be of great matter, such stuff usually is operated from a non-replaceable battery for it's lifetime.

In general, you'll have to apply and verify some techniques to ensure data aren't lost and have some level of confidence. That'll work on any storage device of your choice, so usually the choice of the data storage is driven by economical reasons (lowest cost) and data integrity is shifted into development (usually SW and paperwork).1

i should pass standards like : ISO 4064-2  , ISO 16399 , ...
the data logger is part of metering device.
data should be kept at least for 20 years
interface between MCU and storage element is not my concern i with implement it.
and yes power consumption is important

 

Offline capt bullshot

  • Super Contributor
  • ***
  • Posts: 3033
  • Country: de
    • Mostly useless stuff, but nice to have: wunderkis.de
Re: what is inside a data logger ?
« Reply #9 on: March 09, 2018, 10:40:24 am »
can you have a discussion on flash memory vs EEPROM  ? which one is more appropriate for my system?

You can get your work done with either one. My personal decision would be the EEPROM.
EEPROMs are available in this size, and they have the great advantage of easiness to use (true random writing possible). They are readily available with way much better data retention specifications than Flash memories.
Flash memory can be written once (some serial SPI ones more than once) per block, then a whole block must be erased at once to overwrite. You'll have to put more effort into your SW to organize the data, and you'll get less data retention and write cycles guaranteed than an EEPROM.
Both should be available in ultra low power variants (ultra low power is very common with EEPROMs, not so with Flash) to fulfil your power consumption requirements.

Safety devices hinder evolution
 

Offline Rerouter

  • Super Contributor
  • ***
  • Posts: 4694
  • Country: au
  • Question Everything... Except This Statement
Re: what is inside a data logger ?
« Reply #10 on: March 09, 2018, 10:45:49 am »
Ok so 15 bytes per second,

And the requirement to store for 20 years, Does this imply there are times where it is not logging?

Just to give clarification, 15 bytes per second, or 0.5KB/30 seconds, is 10 megabit every day, In other words your 256Kb memory would last 25 days, or roughly 1 month,

If you went with the maximum normal non nand flash size of 2Gb then it would last about 210 days, or just shy of a 2/3rd of a year.
I chose NOR flash because of the size you will likely require, most data logger projects end up being limited only be memory technologies, EEPROM and even SRAM the other 2 fairly low power options generally cap out at 64Mb, while NAND's low reliability doesn't meet your specs,

Can you store the data in a more efficient form? or sample less often, 20 years is difficult for your current requirements. which is closer to 10GB, now while an SD card may be able to manage it, I would expect you to struggle to make it both low power, and have a warrented reliability for that long.
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #11 on: March 09, 2018, 11:04:29 am »
Ok so 15 bytes per second,

And the requirement to store for 20 years, Does this imply there are times where it is not logging?

Just to give clarification, 15 bytes per second, or 0.5KB/30 seconds, is 10 megabit every day, In other words your 256Kb memory would last 25 days, or roughly 1 month,

If you went with the maximum normal non nand flash size of 2Gb then it would last about 210 days, or just shy of a 2/3rd of a year.
I chose NOR flash because of the size you will likely require, most data logger projects end up being limited only be memory technologies, EEPROM and even SRAM the other 2 fairly low power options generally cap out at 64Mb, while NAND's low reliability doesn't meet your specs,

Can you store the data in a more efficient form? or sample less often, 20 years is difficult for your current requirements. which is closer to 10GB, now while an SD card may be able to manage it, I would expect you to struggle to make it both low power, and have a warrented reliability for that long.


not exactly!  most of write operations just update previous data.   in other word i need store accumulative water flow . and the other thing is that there is a hand held reader and we can extract data  from tariff device periodically .

20 years is just a confidence margin for data don't lose.

« Last Edit: March 09, 2018, 11:07:43 am by alireza7 »
 

Offline mikerj

  • Super Contributor
  • ***
  • Posts: 3233
  • Country: gb
Re: what is inside a data logger ?
« Reply #12 on: March 09, 2018, 11:31:45 am »

not exactly!  most of write operations just update previous data.   in other word i need store accumulative water flow . and the other thing is that there is a hand held reader and we can extract data  from tariff device periodically .

20 years is just a confidence margin for data don't lose.

If you are simply updating the integrated water flow rather than recording a history, then you could use a small EEPROM device.  These typically have a write endurance of up to 1 million writes (though check the datasheet carefully, there are often some gotchas) and this could be extended greatly by using a wear leveling method e.g. having a circular buffer of structures in the EEPROM holding the accumulated flow, a timestamp and a checksum.  Not only does this provide a large write endurance, it also gives you a little bit of  near term historical data and provides fault resilience; if the last value gets corrupted the logger can start again from the last good value so you don't lose everything.
 

Offline Rerouter

  • Super Contributor
  • ***
  • Posts: 4694
  • Country: au
  • Question Everything... Except This Statement
Re: what is inside a data logger ?
« Reply #13 on: March 09, 2018, 11:47:26 am »
Ok, as an integrator, that greatly reduces the complexity, this could be accomplished in any old eeprom, I would still recommend you store a normal copy of your value, an inverted, and a checksum of some kind for each record, but from there, just attach it as a ring buffer,

You would track in your micros RAM where the highest value is, and overwrite the next instance with your new maximum, should your device loose power, it runs through the eeprom finds the highest value, verifies its checksum and saves the location to RAM and continues on, If it encounters a bad checksum it checks the bit flipped copy, and either recovers the normal record, or finds them both faulty and reads back 1 location to verify that,
 

Online newbrain

  • Super Contributor
  • ***
  • Posts: 1714
  • Country: se
Re: what is inside a data logger ?
« Reply #14 on: March 09, 2018, 01:29:20 pm »
For greater write endurance and data retention, no need of wear levelling, and (possibly) low power consumption with respect to EEPROM, consider also ferroelectric RAM.
http://www.ti.com/lit/ml/slat151/slat151.pdf
https://www.fujitsu.com/us/Images/SPBG_FRAM_Overview_BR.pdf

Power consumption (dis)advantage depends on the memory size and the amount of writing that is done:
http://www.cypress.com/file/46746/download



Nandemo wa shiranai wa yo, shitteru koto dake.
 

Online paulca

  • Super Contributor
  • ***
  • Posts: 3996
  • Country: gb
Re: what is inside a data logger ?
« Reply #15 on: March 09, 2018, 01:40:57 pm »
It's a bunch of little clerical daemons with notepads. 

(Terry Pratchett reference).
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 
The following users thanked this post: newbrain

Online newbrain

  • Super Contributor
  • ***
  • Posts: 1714
  • Country: se
Re: what is inside a data logger ?
« Reply #16 on: March 09, 2018, 01:54:58 pm »
It's a bunch of little clerical daemons with notepads. 

But do they work in Hex?

Edit: reference added. Sorry for the OT. ::)
« Last Edit: March 09, 2018, 02:28:02 pm by newbrain »
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Online paulca

  • Super Contributor
  • ***
  • Posts: 3996
  • Country: gb
Re: what is inside a data logger ?
« Reply #17 on: March 09, 2018, 01:58:20 pm »
It's a bunch of little clerical daemons with notepads. 

But do they work in Hex?

Good question.  If  you ask them nicely they might give you the answer in hex.

As long as it's safe hex, bata boom.
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline ebastler

  • Super Contributor
  • ***
  • Posts: 6202
  • Country: de
Re: what is inside a data logger ?
« Reply #18 on: March 09, 2018, 03:09:47 pm »
This whole discussion is a bit scary. Is this about a real product development? Do these really get started based on so little in terms of clues?!  ???
Please comfort me and say that this is a student assignment with a "mock" use case...
 

Offline Kalvin

  • Super Contributor
  • ***
  • Posts: 2145
  • Country: fi
  • Embedded SW/HW.
Re: what is inside a data logger ?
« Reply #19 on: March 09, 2018, 04:23:02 pm »
Do not forget data integrity check and data recovery due to sudden power loss and intermittent resets etc. real-world events which may reset the device in the middle of data write/update.

Edit: Plan for data loss ie. how much data is permitted be lost at maximum if the memory gets corrupted and how it will be detected and possible recovered so that the user will be able to detect that some data has been lost or corrupted.
« Last Edit: March 09, 2018, 04:25:56 pm by Kalvin »
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #20 on: March 12, 2018, 01:41:01 pm »
the endurance of a EEPROM  is related to each byte or it's about whole device?
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #21 on: March 12, 2018, 03:18:14 pm »
This whole discussion is a bit scary. Is this about a real product development? Do these really get started based on so little in terms of clues?!  ???
Please comfort me and say that this is a student assignment with a "mock" use case...

it's just an start up. we should start at a point any way , even a zero point . you can consider it as a student assignment but finally we want to make it a real product.

i appreciate your concern and demand  wish you give me your advice to help me develop a real product.
« Last Edit: March 12, 2018, 03:47:45 pm by alireza7 »
 

Online paulca

  • Super Contributor
  • ***
  • Posts: 3996
  • Country: gb
Re: what is inside a data logger ?
« Reply #22 on: March 12, 2018, 03:23:54 pm »
i appreciate your concern and demand you give me your advice to help me develop a real product.

Typically "demanding" time or knowledge on a forum will only get your silence at best or a appropriately negative response at worst.
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline alireza7Topic starter

  • Regular Contributor
  • *
  • Posts: 102
  • Country: us
Re: what is inside a data logger ?
« Reply #23 on: March 12, 2018, 03:44:22 pm »
i appreciate your concern and demand you give me your advice to help me develop a real product.

Typically "demanding" time or knowledge on a forum will only get your silence at best or a appropriately negative response at worst.

oh sorry English is not my mother tongue and i just picked the wrong word "demanding"   :( .
i think it is better to say i wish you help me



 

Online paulca

  • Super Contributor
  • ***
  • Posts: 3996
  • Country: gb
Re: what is inside a data logger ?
« Reply #24 on: March 12, 2018, 04:03:57 pm »
i appreciate your concern and demand you give me your advice to help me develop a real product.

Typically "demanding" time or knowledge on a forum will only get your silence at best or a appropriately negative response at worst.

oh sorry English is not my mother tongue and i just picked the wrong word "demanding"   :( .
i think it is better to say i wish you help me

No problem.  One way to say what you mean would be:

"I would be grateful for your help and advice.  :)"
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf