Author Topic: How do we know if software has way too many EEPROM writes?  (Read 10666 times)

0 Members and 1 Guest are viewing this topic.

Offline frozenfrogz

  • Frequent Contributor
  • **
  • Posts: 936
  • Country: de
  • Having fun with Arduino and Raspberry Pi
Re: How do we know if software has way too many EEPROM writes?
« Reply #25 on: January 30, 2018, 11:49:53 am »
I have some of the old source code...there's been so many code re-writes, so many bugs , that i personally dont know where the code is.....my boss knows. So do our software test guys, who also do many other jobs in the company.

The OP mentioned that there are sources somewhere in the stratosphere, but it really sounds like very badly managed. Could as well be, that something is buried somewhere in a cluttered folder structure telling you: Important_final_version_edit2_bugfixes.zip
Besides the imminent problem of not knowing what is going on with a certain product, I would highly suggest to spend some money on audit, process management and a reliable version control system. This does not even have to be overly complicated and it will greatly increase the level of convenience for everyone involved in new projects also.

Also, it sounds like code that a lot of different people "fixed" occasionally.  :scared:
He’s like a trained ape. Without the training.
 
The following users thanked this post: ocset

Offline donotdespisethesnake

  • Super Contributor
  • ***
  • Posts: 1093
  • Country: gb
  • Embedded stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #26 on: January 30, 2018, 12:17:34 pm »
I would highly suggest to spend some money on audit, process management and a reliable version control system. This does not even have to be overly complicated and it will greatly increase the level of convenience for everyone involved in new projects also.

I get the impression it is one of those "one man and his dog" type companies, where employees jealousy guard their little domains and don't cooperate well. Management is either ineffective or unable to change personnel because they have made themselves indispensable. Hiring a cheap replacement and training them up is too disruptive for small companies. Auditing and process management (or any sort of quality approach) are purely theoretical concepts for them. I speak from experience...

The fact that employees are desperately seeking solutions on web forums is a sign that the company is beyond repair.
Bob
"All you said is just a bunch of opinions."
 
The following users thanked this post: Jacon, ocset

Offline abraxa

  • Frequent Contributor
  • **
  • Posts: 377
  • Country: de
  • Sigrok associate
Re: How do we know if software has way too many EEPROM writes?
« Reply #27 on: January 30, 2018, 02:07:45 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage. What if the firmware developer feels scorned and wanted to add something that won't cause a problem now but will break the product much later, when he's long gone? I wouldn't be surprised.
 
The following users thanked this post: ocset

Online wraper

  • Supporter
  • ****
  • Posts: 16860
  • Country: lv
Re: How do we know if software has way too many EEPROM writes?
« Reply #28 on: January 30, 2018, 02:11:48 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage. What if the firmware developer feels scorned and wanted to add something that won't cause a problem now but will break the product much later, when he's long gone? I wouldn't be surprised.
Because treez already wrote so many fairy tales that many people think about all of this as a nothing but a joke. It's like everyone around is sabotaging, including UK with it's mains voltage  :-DD.
 
The following users thanked this post: ocset

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: How do we know if software has way too many EEPROM writes?
« Reply #29 on: January 30, 2018, 02:11:59 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage.
Mainly because the OP hasn't posted anything except really wild claims and speculation. Whatever's going on at their company, it's not a technical issue, but management and personnel problems.
 
The following users thanked this post: ocset

Offline NorthGuy

  • Super Contributor
  • ***
  • Posts: 3146
  • Country: ca
Re: How do we know if software has way too many EEPROM writes?
« Reply #30 on: January 30, 2018, 02:12:26 pm »
Thanks, so each byte of the eeprom can be written 100000 times before that byte dies?...so for a 256 byte eeprom like the pic16f18856, we have 25.6 million eeprom writes before it fails?

If it writes evenly to all bytes then yes. If it writes to only one location, the will wear out sooner. There are different algorithms which distribute write to prolong the life.

The power is specified at 5 mA max during programming (in the table 3-3 of the Programming Specification, parameter IDDP), which IMHO includes EEPROM programming, so 7.5 mA is too much.
 
The following users thanked this post: ocset

Online mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13746
  • Country: gb
    • Mike's Electric Stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #31 on: January 30, 2018, 02:49:52 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage. What if the firmware developer feels scorned and wanted to add something that won't cause a problem now but will break the product much later, when he's long gone? I wouldn't be surprised.
Given the choice between incompetence and conspiricy I'd say the former is the most plausible.
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: ocset

Offline donotdespisethesnake

  • Super Contributor
  • ***
  • Posts: 1093
  • Country: gb
  • Embedded stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #32 on: January 30, 2018, 03:43:06 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage. What if the firmware developer feels scorned and wanted to add something that won't cause a problem now but will break the product much later, when he's long gone? I wouldn't be surprised.

Even if for the sake of argument, we assume that there is deliberate sabotage, the advice would be to get the source code, then cut contact with this developer. Do it  in house or find someone competent (and honest). That is the same advice we have been giving anyway

I did some endurance testing with a Microchip serial EEPROM, the results were interesting, but the technology used may not be the same as on the MCUs. Anyway, our rule of thumb is one write every hour stays within the limit of 100K writes, for a 10 year product lifetime. When I hit about 5 million cycles, errors started to appear, but the failure mode was not how I expected.

The investigation was prompted by a colleague who accidentally triggered an EEPROM write several times a second, and I left the code running over a weekend which killed the part of EEPROM being written to.
Bob
"All you said is just a bunch of opinions."
 
The following users thanked this post: ocset

Offline C

  • Super Contributor
  • ***
  • Posts: 1346
  • Country: us
Re: How do we know if software has way too many EEPROM writes?
« Reply #33 on: January 30, 2018, 05:42:57 pm »

Think about what could be done by firmware to kill device.
If programmer really wanted it to fail, power up is not the best time to hide killing EEPROM. This is stupid programming error area.
Long term too many writes to EEPROM is not playing by rules limits & easy to hide error.

Your 15 minutes writes to EEPROM is to get history. Would not be that hard to hide increasing writes over time, would just have to not update last history value location on every write.

incompetence is easy, just need a value going out of range to cause a failure.



 
The following users thanked this post: ocset

Offline dmills

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: How do we know if software has way too many EEPROM writes?
« Reply #34 on: January 30, 2018, 11:56:24 pm »
Post the source (And the real schematics, not some butcherised thing in spice!), nobody here is going to copy the piece of shit, nobody here needs the pain.

At this point folks here around would I suspect take such a set of design files as a 'spot the cockup' contest!

This is almost certainly just a bug and may well not involve the eeprom at all, I would bet on some floating inputs, but who knows without seeing the real diagrams.

As to a cockup Treez has not yet claimed, I will admit to a tin/gold connector pair choice 'issue' that only became apparent after we had a couple of hundred units in the field (Expensive!). Obvious enough once we got some back, but embarrassing as all hell because I knew about the problem :palm:
This sort of crap is what design reviews are for!

BTW: SVN is a free version control tool, reasonably good and works well for both software and hardware design files, the cool kids are all pushing GIT like mad, but SVN is simpler if you have never used such things and works well, you will quickly wonder how you ever lived without versioning.

Regards, Dan.
 
The following users thanked this post: ocset

Offline MT

  • Super Contributor
  • ***
  • Posts: 1616
  • Country: aq
Re: How do we know if software has way too many EEPROM writes?
« Reply #35 on: January 31, 2018, 06:56:14 pm »
I'm surprised people here are so quick to dismiss the possibility of deliberate sabotage. What if the firmware developer feels scorned and wanted to add something that won't cause a problem now but will break the product much later, when he's long gone? I wouldn't be surprised.
Because treez already wrote so many fairy tales that many people think about all of this as a nothing but a joke. It's like everyone around is sabotaging, including UK with it's mains voltage  :-DD.

Treez actually "thanked" your post! That's quite some achievement either way.. :popcorn:
 
The following users thanked this post: ocset

Online Psi

  • Super Contributor
  • ***
  • Posts: 9944
  • Country: nz
Re: How do we know if software has way too many EEPROM writes?
« Reply #36 on: January 31, 2018, 07:28:30 pm »
I'd laugh so hard if the 15s was some crazy implementation of wear leveling where every bootup it moves all eeprom data by 1 byte so data slowly moves it's location across all ee memory.
Greek letter 'Psi' (not Pounds per Square Inch)
 
The following users thanked this post: ocset

Offline fourtytwo42

  • Super Contributor
  • ***
  • Posts: 1185
  • Country: gb
  • Interested in all things green/ECO NOT political
Re: How do we know if software has way too many EEPROM writes?
« Reply #37 on: January 31, 2018, 07:54:43 pm »
I'd laugh so hard if the 15s was some crazy implementation of wear leveling where every bootup it moves all eeprom data by 1 byte so data slowly moves it's location across all ee memory.
Now that could easely be the case if it's some rubbish C library code garnered from somewhere!
IMOP many dont know how to write software or design hardware anymore, they just cobble together a load of stuff seen on the web, after all they got used to that method getting there degrees and phd's right  >:D
« Last Edit: January 31, 2018, 07:57:32 pm by fourtytwo42 »
 
The following users thanked this post: 3db, ocset

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 21675
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: How do we know if software has way too many EEPROM writes?
« Reply #38 on: January 31, 2018, 08:10:58 pm »
Treez actually "thanked" your post! That's quite some achievement either way.. :popcorn:

If you've been paying attention (I don't mean this perjoratively: just that, others here have been following this slow-motion train wreck for dozens of threads :o ), you'll notice that treez thanks almost every post in reply, even the highly critical ones.

Alas, words -- not even words, the mere press of a button rather -- are rather shallow.  I'll pay attention when "has thanked this post" turns into "has donated $5 in thanks for this post". ;D

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 
The following users thanked this post: mrflibble, MT, ocset

Offline Delta

  • Super Contributor
  • ***
  • Posts: 1221
  • Country: gb
Re: How do we know if software has way too many EEPROM writes?
« Reply #39 on: February 01, 2018, 11:44:52 am »
EEPROM write times increase with mains voltage.  As your superb product is running on the UK's 292 VAC RMS for periods of several minutes, 15s is to be expected.
 
The following users thanked this post: ocset

Offline voltsandjolts

  • Supporter
  • ****
  • Posts: 2298
  • Country: gb
Re: How do we know if software has way too many EEPROM writes?
« Reply #40 on: February 02, 2018, 07:21:22 pm »
I'm just posting here to get thanked by treez. This is the best thread I've seen today.
 
The following users thanked this post: Someone, Ice-Tea, ocset

Offline imidis

  • Frequent Contributor
  • **
  • Posts: 426
  • Country: ca
Re: How do we know if software has way too many EEPROM writes?
« Reply #41 on: February 02, 2018, 08:22:44 pm »
Are we sure it is not a rogue chip that was sabotaged by another company?
Gone for good
 
The following users thanked this post: ocset

Offline dmills

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: How do we know if software has way too many EEPROM writes?
« Reply #42 on: February 02, 2018, 09:56:50 pm »
The CIA/NSA/Illuminati/Lizards diddled a C Compiler somewhere back up the chain and pulled a "Reflections on trusting trust" style attack to break eeproms containing keys in a predictable way and this is just a false positive.....

Or it is just a stupid bug.

Regards, Dan.
 
The following users thanked this post: ocset

Online Ice-Tea

  • Super Contributor
  • ***
  • Posts: 3070
  • Country: be
    • Freelance Hardware Engineer
Re: How do we know if software has way too many EEPROM writes?
« Reply #43 on: February 02, 2018, 11:38:46 pm »
If you've been paying attention (I don't mean this perjoratively: just that, others here have been following this slow-motion train wreck for dozens of threads yeqrs :o ),

There; fixed it for you.
 
The following users thanked this post: ocset

Offline KL27x

  • Super Contributor
  • ***
  • Posts: 4101
  • Country: us
Re: How do we know if software has way too many EEPROM writes?
« Reply #44 on: February 03, 2018, 12:54:47 am »
Treez: Remember 6 months back, when more than half your units were fail, right out of the box?

One of the things we asked was if the working units eventually failed, as well?

And remember when we said that if this were the case, this type of behavior is possibly due to bad EEPROM write, probably due to EEPROM values being written too soon after startup? Rather than a read-modify-write error, which you were trying to find?

And apparently, you got this problem sorted out, because you have working lamps, now.

The coder might have discovered that this was the problem. When you asked about this (perhaps new) behaviour, he did mention something to that effect, right? That's why you are asking about EEPROM writes? And to fix it, perhaps he simply added a startup delay with a very generous margin of safety. This would be perhaps a tiny bit lazy to not get the delay closer to, say, twice w/e the datasheet says is needed. If I were too lazy to look it up, I'd probably throw my dart at ~100-200mS. But this is because I've had the same problem, and I while I don't remember the actual figure for a given PIC, I am pretty confident that 100mS is way more than enough for any of them. (It's also important for anything where EEPROM can brick it that power good and brown out detection are both being utilized). But this 14.8ish seconds is quibbling over nothing. In the bigger picture, 15 seconds of the micro counting sheep before finally doing its thing and going to sleep? On something that runs on mains, and which is switched on/off only a few times a day, and for which the micro max draw is probably not even a tenth of a percent of the total power consumption, 15s is really meaningless.

Additionally, if a bad EEPROM write (or a worn out EEPROM) is going to cause some serious/lasting problem, it might be nice to have a valid parameter check/overwrite in the code, so the lamp at least turns on if there is an error. I don't know if that's reasonable for what this micro needs to do, mind you. And it looks like you are not actually coding, just that you're being a backseat driver to your actual coder. (Maybe not very productive if you don't know how to drive). Unless there's some major communication barrier between you and your coder, you should be able to figure out what he did; and he should make a fix if he is not delivering what you originally specified. This may or may not be the case (it sounds like you're chasing a phantom whatif) . Email, man. Copy and paste the technical stuff, here, if you don't understand it. Relaying second or third hand information (he said this/that) is sketchy. If you can't write it yourself, you might not understand what he actually said.

If you continue to be concerned (for reasons real or imagined), you could perhaps shoot an email to your programmer. I would simply ask him what is the expected lifespan of the EEPROM. If he bothers to track them all down and count them up for you, he might answer in cycles and/or time depending on what the code is doing, based on the min EEPROM writes in the datasheet. I imagine the guy that wrote it might be able to accurately figure out which EEPROM registers are used the most, insert a few lines of code into the save subroutine(s) to increment an internal software counter or an external counter and output pin, and play with the lamp for you. Sending him a fully assembled lamp or 2 (at least with all the important bits that you plan to use in production) and a few bucks might help him out because he probably doesn't have this hardware, anymore. And this shit doesn't do itself. Esp where human interface/input may be involved with dials and whatnot, it's not necessarily possible to make a decent estimate without a working, assembled, finished product.
« Last Edit: February 03, 2018, 04:54:07 am by KL27x »
 
The following users thanked this post: ocset

Offline donotdespisethesnake

  • Super Contributor
  • ***
  • Posts: 1093
  • Country: gb
  • Embedded stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #45 on: February 03, 2018, 09:11:40 am »
I've come to the conclusion that this is all part of an elaborate troll. I wonder if this company that is the victim of all manner of unfortunate and improbable events really exists.

Bob
"All you said is just a bunch of opinions."
 
The following users thanked this post: ocset

Online mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13746
  • Country: gb
    • Mike's Electric Stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #46 on: February 03, 2018, 09:33:03 am »
I've come to the conclusion that this is all part of an elaborate troll. I wonder if this company that is the victim of all manner of unfortunate and improbable events really exists.
IME the situations are entirely plausible - it may well be that treez is firefighting the results of incompetent management above him.
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 
The following users thanked this post: ocset

Offline 3db

  • Frequent Contributor
  • **
  • Posts: 331
  • Country: gb
Re: How do we know if software has way too many EEPROM writes?
« Reply #47 on: February 03, 2018, 12:57:54 pm »
I'm just posting here to get thanked by treez. This is the best thread I've seen today.

Epic  !!  :-DD :-DD
 
The following users thanked this post: ocset

Offline donotdespisethesnake

  • Super Contributor
  • ***
  • Posts: 1093
  • Country: gb
  • Embedded stuff
Re: How do we know if software has way too many EEPROM writes?
« Reply #48 on: February 03, 2018, 01:28:15 pm »
I've come to the conclusion that this is all part of an elaborate troll. I wonder if this company that is the victim of all manner of unfortunate and improbable events really exists.
IME the situations are entirely plausible - it may well be that treez is firefighting the results of incompetent management above him.

Sure, that is one way of looking at it, there are others.
Bob
"All you said is just a bunch of opinions."
 
The following users thanked this post: ocset

Offline janoc

  • Super Contributor
  • ***
  • Posts: 3785
  • Country: de
Re: How do we know if software has way too many EEPROM writes?
« Reply #49 on: February 03, 2018, 03:14:44 pm »
Quote
Another point is that the micro is fed by a linear regulator which feeds off rectified mains, so this heavy writing to EEPROM is overheating the linear regulator that supplies the micro.  :scared:

I think this is both pretty symptomatic of these threads started by treez:

- Wild guesses and fishing expeditions with no evidence or trying to actually verify them

- However, ample evidence of terribly terribly designed piece of crap product. If a voltage regulator overheats only because a micro writing an EEPROM (which is a perfectly normal thing for a micro to do), then what is that regulator going to do when it is 40C outside, with Sun frying the metal enclosure? Which is also something a street lamp should be able to deal with.

And a linear reg for dropping rectified mains (i.e. some 300V) down to 3.3-5V?!?! I do hope there is at least some pre-regulator there ...

Oh and it is always easier to look for intentional sabotage than own incompetence (admitting incompetence hurts and could get one sued). Seriously, dude ... :palm:
« Last Edit: February 03, 2018, 03:17:26 pm by janoc »
 
The following users thanked this post: ocset


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf