How do we know if software has way too many EEPROM writes?

[ocset]

Hello,
Please help explain why our software engineer has programmed our PIC16F18856 microcontroller to write to its EEPROM for 15 seconds continuously when it only needed 5ms to do this?   

As you know, since it should only take 5ms to write to EEPROM (Pg 611 of datasheet says so), this means he has done 2999 EEPROM writes which he did not need to do. As such, we are concerned that every time he is writing a byte to EEPROM he is actually doing it 3000 times, instead of just once. –As you know, after so many EEPROM writes, a microcontroller EEPROM dies. It looks to me that he has been deliberately trying to kill our microcontroller in our lamp, and thus bring the lamp’s life to a premature end.       

The lamp periodically stores ambient light readings into microcontroller EEPROM. (These light readings are obtained through the micro’s ADC after reading a SFH5711 light sensor). The readings are taken several times per day…..once every 15 minutes.

We are actually concerned because we only discovered that he was doing this 15 seconds worth of EEPROM writes when we realised that the microcontroller consumed over 5mA for the first 15 seconds after being switched ON. If we hadn’t have noticed this and challenged the software guy about this, it would never have come to be known what was actually going on.    . After being switched ON, the lamp should just take one light reading and store it to EEPROM.....this should take only 5ms, not 15 seconds.  Another point is that the micro is fed by a linear regulator which feeds off rectified mains, so this heavy writing to EEPROM is overheating the linear regulator that supplies the micro. 

For all we know, the lamp may, at some time in the future , embark on continuous EEPROM writes, which would kill the lamp……he has obviously no control over what he is doing with EEPROM writes…unless of course he is deliberately trying to destroy our company…what do you think? 

PIC16F18856 datasheet
http://ww1.microchip.com/downloads/en/DeviceDoc/40001824B.pdf

[fourtytwo42]

I have to say the ongoing conspiracy theories at your company are highly entertaining 
Are you sure OUR hardware and OUR software engineer are not in fact yourself ?
Did you track down the enemy agent destroying your streetlamps in Budapest or wherever it was ?

[capt bullshot]

Sounds like pure stupidity to me
Yes there are SW guys that work that way out there (doing some stupid mistake by mis-reading the datasheet, or solving a PEBKAC in a "clever" way) . Congrats on finding one 

[rstofer]

If you don't have a formal 'code review' process, it's your company's fault.  Every single line of code should be reviewed by a panel of experts.  Not just another 'contractor'.

There should be a similar hardware review process.  You can't let some kid design a product and just hope it all works out.  There are real EEs who do this kind of thing for a living.

Mistakes happen even with formal code review.  But there are a lot fewer of them!

I'm surprised your company is still in business.

[nctnico]

If you don't have a formal 'code review' process, it's your company's fault.  Every single line of code should be reviewed by a panel of experts.  Not just another 'contractor'.

There should be a similar hardware review process.  You can't let some kid design a product and just hope it all works out.  There are real EEs who do this kind of thing for a living.

Mistakes happen even with formal code review.  But there are a lot fewer of them!

I'm surprised your company is still in business.

I'm surprised he still works there 

[T3sl4co1l]

I'm surprised your company is still in business.

Alas, there's a disturbing amount of free money out there, even (especially?) if you skimp on engineering budget.

Tim

[mariush]

Why don't you post the source code here, maybe removing any proprietary information... if we're bored, maybe we can give you some code review

It's not like someone here would clone your product.

Maybe he's erasing each byte one at a time, reading back to verify it's erased, then writing just a bunch of bytes.

How much data is that light information ?  If you want longer life or a more even write spread, maybe you can keep in that eeprom the last 20-50 or so measurements and loop through those overwriting the oldest one with the newest measurement.

[ocset]

Thanks, may i ask , is it the EEPROM writes which make  the  high current draw of the micro?
Because we understand that   every micro has only a limited possible number of EEPROM writes before EEPROM dies.
We  know that the current draw was way excessive for 15 seconds immediately after switch on...and the software guy has confessed that that is because of EEPROM operations........i presume he must mean writes? (he should mean writeing because the lamp should take a single  light reading shortly after switch-ON and store it in EEPROM).......Presumably you can read EEPROM as much as you like and it means no high current draw of the micro and does not impact the life of the micro?

The ADC reading of the light level, is, i beleive, either an 8 bit byte or 16 bits.......i am sure its nowhere near long enough to take 15 seconds to write to EEPROM.

I think we've been  done over.   
I reckon the software guy, ( who lives in another country) will say its a mistake. But i am not sure how you can make that kind of mistake..without it actually being deliberate.
I understood that my company had gotten another software guy to inspect the code.

Maybe he's erasing each byte one at a time, reading back to verify it's erased, then writing just a bunch of bytes.

Thanks, so you mean the EEPROM may not die quickly, like we thought it might?...in other words, there may be hope for us yet?

[Gyro]

If you don't have a formal 'code review' process, it's your company's fault.  Every single line of code should be reviewed by a panel of experts.  Not just another 'contractor'.

There should be a similar hardware review process.  You can't let some kid design a product and just hope it all works out.  There are real EEs who do this kind of thing for a living.

Mistakes happen even with formal code review.  But there are a lot fewer of them!

I'm surprised your company is still in business.

^^ What he said!

Hello,
Please help explain why our software engineer has programmed our PIC16F18856 microcontroller to write to its EEPROM for 15 seconds continuously when it only needed 5ms to do this?   

What a very strange question to ask - why should we be explaining it - your software engineer should be doing that.

When you say "our software engineer", I assume that you mean the software engineer of the company in China that you are buying your product from.

You need to take FAR greater control of your supplier's development strategy and processes than you currently are if you are to stand any chance of staying in business. You should be closely reviewing the design process, every step of the way.

Do you even have a copy of the source code? If not, then you have no reasonable chance of controlling the project.

[mikeselectricstuff]

I reckon the software guy, ( who lives in another country)

There's your problem.
Can you seriously not find anyone local to do the job?
or at least  just do code (and hardware design) reviews.

What does the source code look like?
If the answer is "we don't have it" then this simply an incompetent product design process.
From what you have posted it seems you don't even know if that's actually what happenned and are jsut taking the word of an idiot. 
 

[ocset]

I have some of the old source code...there's been so many code re-writes, so many bugs , that i personally dont know where the code is.....my boss knows. So do our software test guys, who also do many other jobs in the company.
I usually work on the hardware......thats why i noticed the input cap to the linear regulator that feeds the micro was getting pulled right down to dc negative and realised that high current was being drawn, and that it could only be the micro doing this.

[mikeselectricstuff]

I wouldn't expect writing eeprom to draw appreciably more current than running.
Excess current draw is more likely due to selecting the wrong clock, i/o clash or floating input pins.
What is the actual current draw you're seeing in the first 15 secs, and afterwards ?

[mikeselectricstuff]

With MCU based products you just can't get by with having a seperate "hardware guy" and "Software guy". Both must have a good knowledge of the other's field.
 

[NorthGuy]

EEPROM endurance is listed in the datasheet, usually 100,000 erase/writes.

If the software writes to EEPROM for 15 seconds, it is probably not writing the same byte, but different bytes. So, it's a very little chance you kill the EEPROM during just few starts.

You're telling this amazing stories about your software guy doing lots of stupid mistakes, but you don't want to fire him anyway. I don't believe this.

[frozenfrogz]

Back to the original question: You would either have to take a look at the sources to see what is going on at startup or use an in-circuit debugger (MPLAB ICD or PICkit).
Personally I would not bother with either. Simply ask the programmer - even if you do not get a sufficient answer, that might be your answer (-> code review et al.).

[rstofer]

You really need a formal version control system.  That way changes are tracked.

[donotdespisethesnake]

For all we know, the lamp may, at some time in the future , embark on continuous EEPROM writes, which would kill the lamp……he has obviously no control over what he is doing with EEPROM writes…unless of course he is deliberately trying to destroy our company…what do you think? 

Paranoid much? Cock ups are nearly always incompetence not conspiracy.

Your company obviously does not have sufficient control of your software development, that is a management problem. Sounds like someone there is trying to destroy your own company.

But maybe even that function is outsourced nowadays.

[TJ232]

OMG, another "treez production" story. 

Your saga looks like a heavy run after any possbile mistake that can be done in terms of business development, project management, hardware/software development, testing, etc, you just name it. I have seen a lot in 25 years of engineering but this one is way to much 

If somebody can find any silly mistake that treez can say they didn't done it, I think it should be posted here for posterity.

PS: I still don't believe that all these stories are real, but at least they are very good examples that can help starters in building their own DO_NOT_EVER_DO_IT_IN_THAT_WAY checklist.

[ocset]

If the software writes to EEPROM for 15 seconds, it is probably not writing the same byte, but different bytes. So, it's a very little chance you kill the EEPROM during just few starts.

Thanks, so each byte of the eeprom can be written 100000 times before that byte dies?...so for a 256 byte eeprom like the pic16f18856, we have 25.6 million eeprom writes before it fails?

Can you seriously not find anyone local to do the job?

I think its an EU grant, and we have to pick someone from a different EU country other than our own.

I wouldn't expect writing eeprom to draw appreciably more current than running.
Excess current draw is more likely due to selecting the wrong clock, i/o clash or floating input pins.
What is the actual current draw you're seeing in the first 15 secs, and afterwards ?

Thanks, strangely the datasheet doesnt say how much current is drawn when eeprom is writing.
Sorry i do not understand what is meant by "i/o clash", please could you possibly advise?
( i think it means an input has accidentally been declared as an output and this output is possibly soucing loads of current....this is possible with us because we have dipswitchs to inputs which go direct to ground...so if these were declared as outputs then lots of current would flow to ground from that pin....recently, our software test guy noticed that the micro rail was short-circuit, but only when one of the dipswitchs was switched to ground)

We also wonder whether he is doing repeated ADC readings of the light reading...and maybe thats drawing this excess current?
Strangely the elec specs of the micro dont tell what is the current draw of things like ADC or EEPROM writing/erasing etc.
I must admit i am surprised it can write to its EEPROM with just a 2v5 rail...i thought writing to eeprom took 9v or so?

The normal running current for the 2V5 rail is 2.2mA. Thats the micro and some really low current drawing things like SFH5711, MCP9700 and an opamp buffer with an output resistor of 100k........so the 7.5mA of current draw we see at startup is unquestionably from the micro.

[Nusa]

In any case, if the eeprom fails to write, your software should fail in a manner that gives the light basic function as a default case -- you've still got the real-time sensor reading, after all.

What, you didn't think of those things in the design? Who's fault is that?

[djococaud]

Just a thought :

What if the micro just measures the light one time each second, do it 15 times and then averages it and store the value in the eeprom ?

Thats would be only one eeprom write and 15s of full speed micro activity (doing nothing between measures, wich causes the regulator to overheat).

(Then, the regulator could olso be undersized)...

[mikeselectricstuff]

I wouldn't expect writing eeprom to draw appreciably more current than running.
Excess current draw is more likely due to selecting the wrong clock, i/o clash or floating input pins.
What is the actual current draw you're seeing in the first 15 secs, and afterwards ?

Thanks, strangely the datasheet doesnt say how much current is drawn when eeprom is writing.
Sorry i do not understand what is meant by "i/o clash", please could you possibly advise?
( i think it means an input has accidentally been declared as an output and this output is possibly soucing loads of current....this is possible with us because we have dipswitchs to inputs which go direct to ground...so if these were declared as outputs then lots of current would flow to ground from that pin....recently, our software test guy noticed that the micro rail was short-circuit, but only when one of the dipswitchs was switched to ground)

Yes. Incorrectly setting input pins as outputs.

We also wonder whether he is doing repeated ADC readings of the light reading...and maybe thats drawing this excess current?

No - ADC won't draw that much. You may want to do multiple readings to avarage out values to reduce noise.
 

Strangely the elec specs of the micro dont tell what is the current draw of things like ADC or EEPROM writing/erasing etc.

Which is why I doubt the draw increases significantly when writing, thought may be that it's for a short enough time that they don't think it's worth documenting

I must admit i am surprised it can write to its EEPROM with just a 2v5 rail...i thought writing to eeprom took 9v or so?

It does. There is an on-chip charge pump

The normal running current for the 2V5 rail is 2.2mA. Thats the micro and some really low current drawing things like SFH5711, MCP9700 and an opamp buffer with an output resistor of 100k........so the 7.5mA of current draw we see at startup is unquestionably from the micro.

This is exactly why hardware people need to know enough  software to be able to write some trivial code in a few minutes to fire up the MCU and check current draw in various situations.

Looking at the datasheet, 7.5mA is twice the max current spec at the highest clock, so something odd is occurring - continuous ee write is one possibility, probably too low to be an I/O clash but could be a floating input pin.
 
Do not spend another second on this until you have the source in front of you - it's just a waste of time guessing until you know exactly what is happenning.

[frozenfrogz]

Do not spend another second on this until you have the source in front of you - it's just a waste of time guessing until you know exactly what is happenning.

This. 

[JPortici]

which is exactly what they keep telling him at the microchip forums

[donotdespisethesnake]

I get that an external consultant is used, that is actually quite common. I also get that the contract may be offset by grants or subsidies, I've worked in some of those. But it's sheer insanity to enter a contract where the scope of supply is to only provide binary files, and not *at least* source, more likely also requiring some design documentation, precise instructions on how to build.

The first thing I do when using an external consultant is to get someone in house to build the software and verify that the binary is byte for byte exactly the same as the delivered binary. Otherwise, it's like you have just loaded a gun and pointed it at your head.