Arduino Pro Mini - two copies have different sleep currents

[Peabody]

My Pro-Minis from Taobao finally arrived (shipping PITA currently), date code 2002.  I won't pull a chip off yet to test but will hold onto a couple in case we still need more data.

I'm not good at SMD rework stuff.  So what I usually do is remove the regulator and the power LED's resistor, leaving only the processor to draw current.  But that's also how I use Minis, so it all works out.  Anyway, Kevin sent me a picture of the 328P he received from Digikey, also with a date code of 2002.  Look anything like yours?

[sleemanj]

I'd have to check the  fab code in the morning but think it's the same from memory.  Did he test it?

[Peabody]

He's working on it today, but I don't know when to expect any news.  I think he plans to make a video.

[sleemanj]

Markings on mine are the same except 2002CJE instead of his 2002G1U

[Peabody]

Ok, I just got email from Kevin saying the new chip he got from Digikey sleeps properly - the same as his old 2016 stock.  It would sure be interesting to know how your new chips sleep.  If they sleep properly, then I guess all of mine were counterfeits.  Either that or Microchip produced a string of bad chips that they've since corrected.  I don't really know what to think.  But at least it's possible to replace bad chips with good ones if sleep current matters.

[SilverSolder]

Ok, I just got email from Kevin saying the new chip he got from Digikey sleeps properly - the same as his old 2016 stock.  It would sure be interesting to know how your new chips sleep.  If they sleep properly, then I guess all of mine were counterfeits.  Either that or Microchip produced a string of bad chips that they've since corrected.  I don't really know what to think.  But at least it's possible to replace bad chips with good ones if sleep current matters.

It is also possible that Microchip baked a batch of chips that work fine, but fail the sleep current test, which somehow found their way out into the wild?   In a weird sort of way, these "bad" chips are actually acceptable for use on an Arduino Mini because other devices on that board use more current anyway, so unless you modify the board nobody would ever know...   Using them this way keeps the prices down and keeps usable chips out of the landfill.   Conspiracy theory mode ON! 

[Peabody]

sleemanj, could you tell me what the location code is on your date code 1933 chip that sleeps at 150uA?  It will be KR, TW, TH or PH.  I"m testing the theory that all the bad ones are either KR or TW, but all THs are good.

[DrG]

Ok, I just got email from Kevin saying the new chip he got from Digikey sleeps properly - the same as his old 2016 stock.  It would sure be interesting to know how your new chips sleep.  If they sleep properly, then I guess all of mine were counterfeits.  Either that or Microchip produced a string of bad chips that they've since corrected.  I don't really know what to think.  But at least it's possible to replace bad chips with good ones if sleep current matters.

It is also possible that Microchip baked a batch of chips that work fine, but fail the sleep current test, which somehow found their way out into the wild?   In a weird sort of way, these "bad" chips are actually acceptable for use on an Arduino Mini because other devices on that board use more current anyway, so unless you modify the board nobody would ever know...   Using them this way keeps the prices down and keeps usable chips out of the landfill.   Conspiracy theory mode ON! 

First off, this is all completely speculative at this point. We all seem to realize that but it merits additional mentioning. That being said, I'm sorry, but  what you are saying just does not pass the smell test.

Microchip (or Atmel) puts out specifications. If, for example, the chip does not follow the specification for typical characteristics e.g., 29.1.2   Power-down Supply Current http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-7810-Automotive-Microcontrollers-ATmega328P_Datasheet.pdf BY A LARGE MARGIN, like the differences reported in the thread, it is a bad specification, and would indicate, it seems to me, poor quality control, period.

To think that Microchip (Atmel) would know that some batches of chips are OK except for failing these specifications (whatever they might be), put their marking on it, and sell them, but only to folks who promise to only use them in Pro Minis clones (or some specific clone product only), strikes me as being bizarre. Even more so since a counterfeit chip would explain the situation in a far simpler and more reasonable manner.

Reminds me of years ago, Radio Shack selling quantities of 3055 transistors that were not marked (or had uninterruptible uninterpretable markings) - everyone knew they were "seconds", but they sure did not carry legitimate manufacturer marks as I remember

But, I don't know what the story will turn out to be.

BTW: I still have those 3055s 

[SilverSolder]

Ok, I just got email from Kevin saying the new chip he got from Digikey sleeps properly - the same as his old 2016 stock.  It would sure be interesting to know how your new chips sleep.  If they sleep properly, then I guess all of mine were counterfeits.  Either that or Microchip produced a string of bad chips that they've since corrected.  I don't really know what to think.  But at least it's possible to replace bad chips with good ones if sleep current matters.

It is also possible that Microchip baked a batch of chips that work fine, but fail the sleep current test, which somehow found their way out into the wild?   In a weird sort of way, these "bad" chips are actually acceptable for use on an Arduino Mini because other devices on that board use more current anyway, so unless you modify the board nobody would ever know...   Using them this way keeps the prices down and keeps usable chips out of the landfill.   Conspiracy theory mode ON! 

First off, this is all completely speculative at this point. We all seem to realize that but it merits additional mentioning. That being said, I'm sorry, but  what you are saying just does not pass the smell test.

Microchip (or Atmel) puts out specifications. If, for example, the chip does not follow the specification for typical characteristics e.g., 29.1.2   Power-down Supply Current http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-7810-Automotive-Microcontrollers-ATmega328P_Datasheet.pdf BY A LARGE MARGIN, like the differences reported in the thread, it is a bad specification, and would indicate, it seems to me, poor quality control, period.

To think that Microchip (Atmel) would know that some batches of chips are OK except for failing these specifications (whatever they might be), put their marking on it, and sell them, but only to folks who promise to only use them in Pro Minis clones (or some specific clone product only), strikes me as being bizarre. Even more so since a counterfeit chip would explain the situation in a far simpler and more reasonable manner.

Reminds me of years ago, Radio Shack selling quantities of 3055 transistors that were not marked (or had uninterruptible markings) - everyone knew they were "seconds", but they sure did not carry legitimate manufacturer marks as I remember

But, I don't know what the story will turn out to be.

I agree with you, I don't think Atmel would have sold them as anything other than scrap...   unmarked scrap, even.  -  But someone might pay them reasonably good money for the scrap...   like Radio Shack buying the 2N3055 seconds that couldn't be sold as the "real thing".   Except in this case, they couldn't avoid marking the components so people would know what it is...

The only thing I have against this kind of e-cycling is that they should be up front about it.  I've always wondered how they can sell an Arduino Mini for less than I can buy a processor for...   this kind of thing would be one explanation.

[All of this is purely speculative idle chatter, of course!]

[sleemanj]

sleemanj, could you tell me what the location code is on your date code 1933 chip that sleeps at 150uA?

The 1933 is TW

I pulled the 2002 from one of my Pro Minis and it sleeps correctly under 1uA.

So I think one of

  1. some fabs have an issue which had not been picked up and they were sold as production normal - seems unlikely
  2. some fabs had an issue which was picked up and instead of the batch going into the trash somebody walked it out the back door and it wound up in the chinese market - this seems most feasible to me
  3. counterfeit - I don't think this is likely given I've just bought pro-minis (and nanos too) with seemingly legit chips

[Peabody]

sleemanj, could you tell me what the location code is on your date code 1933 chip that sleeps at 150uA?

The 1933 is TW

I pulled the 2002 from one of my Pro Minis and it sleeps correctly under 1uA.

So I think one of

  1. some fabs have an issue which had not been picked up and they were sold as production normal - seems unlikely
  2. some fabs had an issue which was picked up and instead of the batch going into the trash somebody walked it out the back door and it wound up in the chinese market - this seems most feasible to me
  3. counterfeit - I don't think this is likely given I've just bought pro-minis (and nanos too) with seemingly legit chips

That's great news about the 2002 Pro Mini.  And that's a TH fab.  So at least it's still possible to get Pro Minis that work right so long as you buy from the right place.  Speaking of which, could you post a link to where you bought yours?

I've always thought the counterfeit explanation was unlikely.  The 328P has been around a long time, and I've been unable to find a single confirmed case of a fake.  Of course it could be that the pirates finally got around to it, but I'm still skeptical.

Kevin has posted his video:

https://youtu.be/PlGycKwnsSw

[DrG]

Just got done watching the video. He did a good job. I think he showed, unequivocally, that your "bad" pro mini was indeed "bad". He moved it off of the pro mini board and onto his boards (with his boot loader), after demonstrating the low current sleep, and he can't get your chip to show that low current sleep. In fact, he found pretty much what you found. That's all good and convincing.

He does not, however, show any evidence that the bad chip is either genuine or counterfeit. Regardless of what he (or any of us) suspects, there is no real evidence [I am discounting the labeling difference at this point] of it being genuine or not.

[sleemanj]

Speaking of which, could you post a link to where you bought yours?

This is the supplier I got them from

  https://item.taobao.com/item.htm?id=571250487735

if you're not experienced with buying from Taobao, well, this isn't really the thread to explain it but probably spreenow is the "easiest" agent to buy through.

[sleemanj]

He does not, however, show any evidence that the bad chip is either genuine or counterfeit.

Well, there's really no way to know, short of decapping it (and even then not necessarily conclusive unless you are Microchip).

That said, if somebody here wants to decap and has an appropriate metallurgical microscope to inspect and photograph for the forum I'm more than happy to post a good and bad one to them (of course currently post from NZ to anywhere else in the world might take a while!).

[Psi]

They're probably more likely to be factory rejects than counterfeits.

However it would be cool to remove the case and compare the die to a legit chip

[westfw]

The 328P has been around a long time, and I've been unable to find a single confirmed case of a fake.

There's the LGT8f328.  It's not an exact clone of an AVR, but it has most of the "hard" parts, and I think it would be pretty easy to spin a version that was more compatible (essentially replacing the SWD-like programming interface with the Atmel SPI-based programming.  Especially at the rate that Asia seems to counterfeit themselves.

It might be interesting to poke at one of the misbehaving chips with a debugger, and see if it seems to support the full debugwire protocols (which are more secretive than the rest of the AVR architecture.  Though it has been reverse-engineered.)

[Psi]

I have seen a few AVRs permanently behave strange after having lots of programming issues.   
Like running at odd clock speeds that shouldn't be possible.
I figured it was secret undocumented fuse bits getting corrupt.


But i still think it's more likely there is a difference on the PCB to account for the additional current. Something non-obvious.

Have you tried putting the chip into sleep mode and then measuring the voltage on each IC pin with a scope. Then compare to a chip without the issue. 

[sleemanj]

But i still think it's more likely there is a difference on the PCB to account for the additional current..

That is not the case.  See my tests which confirmed the behaviour, I drop each IC into my socketed test board, no other change (also, nothing connected to any pins except for power, ground).

[Ian.M]

Ok, I just got email from Kevin saying the new chip he got from Digikey sleeps properly - the same as his old 2016 stock.  It would sure be interesting to know how your new chips sleep.  If they sleep properly, then I guess all of mine were counterfeits.  Either that or Microchip produced a string of bad chips that they've since corrected.  I don't really know what to think.  But at least it's possible to replace bad chips with good ones if sleep current matters.

It is also possible that Microchip baked a batch of chips that work fine, but fail the sleep current test, which somehow found their way out into the wild?   In a weird sort of way, these "bad" chips are actually acceptable for use on an Arduino Mini because other devices on that board use more current anyway, so unless you modify the board nobody would ever know...   Using them this way keeps the prices down and keeps usable chips out of the landfill.   Conspiracy theory mode ON! 

First off, this is all completely speculative at this point. We all seem to realize that but it merits additional mentioning. That being said, I'm sorry, but  what you are saying just does not pass the smell test.

Microchip (or Atmel) puts out specifications. If, for example, the chip does not follow the specification for typical characteristics e.g., 29.1.2   Power-down Supply Current http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-7810-Automotive-Microcontrollers-ATmega328P_Datasheet.pdf BY A LARGE MARGIN, like the differences reported in the thread, it is a bad specification, and would indicate, it seems to me, poor quality control, period.

To think that Microchip (Atmel) would know that some batches of chips are OK except for failing these specifications (whatever they might be), put their marking on it, and sell them, but only to folks who promise to only use them in Pro Minis clones (or some specific clone product only), strikes me as being bizarre. Even more so since a counterfeit chip would explain the situation in a far simpler and more reasonable manner.

Reminds me of years ago, Radio Shack selling quantities of 3055 transistors that were not marked (or had uninterruptible markings) - everyone knew they were "seconds", but they sure did not carry legitimate manufacturer marks as I remember

But, I don't know what the story will turn out to be.

I agree with you, I don't think Atmel would have sold them as anything other than scrap...   unmarked scrap, even.  -  But someone might pay them reasonably good money for the scrap...   like Radio Shack buying the 2N3055 seconds that couldn't be sold as the "real thing".   Except in this case, they couldn't avoid marking the components so people would know what it is...

The only thing I have against this kind of e-cycling is that they should be up front about it.  I've always wondered how they can sell an Arduino Mini for less than I can buy a processor for...   this kind of thing would be one explanation.

[All of this is purely speculative idle chatter, of course!]

To add to the speculation, I noted way back at reply#20

A decade or so back, Microchip Technology used to do special far east market versions of some of their PICs that didn't meet the US datasheet specs, for really cost-sensitive applications.   They were supposedly marked with distinct part numbers, but we all know how that one goes when there are no westerners 'in the loop' . . .  Maybe they've started doing similar with AVRs?

Short of decapping and side by side die inspection of a high sleep current chip and a low sleep current genuine chip straight from MicrochipDirect, it looks like we aren't going to get any answers.  If it is a batch problem and Microchip ever do respond with an errata update, odds are it will be months or even years away.

[Psi]

But i still think it's more likely there is a difference on the PCB to account for the additional current..

That is not the case.  See my tests which confirmed the behaviour, I drop each IC into my socketed test board, no other change (also, nothing connected to any pins except for power, ground).

hm..  i'm having some design issues with a product im making that uses a ATMEGA328P, not related to sleep current or the chip, but thread this is prompting me to change the chip to something less popular to avoid potentially getting fakes in the future.

[DrG]

But i still think it's more likely there is a difference on the PCB to account for the additional current..

That is not the case.  See my tests which confirmed the behaviour, I drop each IC into my socketed test board, no other change (also, nothing connected to any pins except for power, ground).

Yes, the tests run in the video as well as your tests have convinced me that the phenomenon is a real one. Those efforts, as well as those of the OP to bring the issue to light in the first place are definitely a contribution and it is an interesting.

Comments after the video also suggest decapping as, potentially, a definitive test.

The other issue that has been mentioned here and in the comments of the video, suggest this idea of factory rejects finding there way onto the boards. I just don't have the experience to know if this really does happen to any extent and while it seems at least plausible, I have not seen any documentation that something like that happened, just anecdotal evidence.

I would hope (with more of a sense that it would be nice if things worked that way, than any evidence) that some other characteristics of the "bad" chips would be found to be different - something like NOT turning off a peripheral, as being responsible.

On the other hand, it seems like there is a specific time niche that these "bad" chips were made, based on some markings - no?

Why would it be happening with only that package of the 328P (at least so far)? Why not the DIP package, for example? Is there a reasonable explanation for that?

[Ian.M]

hm..  i'm having some design issues with a product im making that uses a ATMEGA328P, not related to sleep current or the chip, but thread this is prompting me to change the chip to something less popular to avoid potentially getting fakes in the future.

Order preprogrammed ATMEGA328P chips from MicrochipDirect, with all security lock bits programmed, drop-shipped direct to your assembler.  Include a factory test mode in the preprogrammed firmware and reject any boards that don't pass it.  Include another concealed test mode you don't disclose till you have to in case they are too clever for their own good!

That of course doesn't guarantee your assembler wont substitute grey-market or fake parts, but it raises the bar considerably and prevents simply 'forgetting' to change a reel loaded with questionable chips before runing your boards, because to substitute the chip, they'd have to spend a four figure sum to recover a copy of your test firmware, and eat the overhead of programming it into every chip.

[Peabody]

Yes, the tests run in the video as well as your tests have convinced me that the phenomenon is a real one. Those efforts, as well as those of the OP to bring the issue to light in the first place are definitely a contribution and it is an interesting.

Comments after the video also suggest decapping as, potentially, a definitive test.

I don't want anyone, especially me, to spend any money decapping chips.  But I think it's worth trying to get a response from Microchip.  I'm hoping Kevin Darrah can do that.  He mentioned in email trying to contact an FAE he's talked to before.  Field Applications Engineer?  That's probably not the right guy to answer the question, but maybe he could refer Kevin to someone.  In the end, whether they are counterfeits or a bad run or whatever, I think we're all going to have to be careful about any clone we buy that uses a 328P which (so far) is dated in the last half of 2019 and which has a fab code of TW or KR.  So far nobody has mentioned having a chip with those markings that actually works properly.  And of course we don't know if sleep current is the only issue with those chips.

My original problem was that after two tries with two different sellers, none of the Pro Mini clones I received slept soundly.  But at least we know now that Microchip's current production is good, so in the worst case I could buy genuine chips from Digikey and try to find someone locally to switch them out for me.

I would hope (with more of a sense that it would be nice if things worked that way, than any evidence) that some other characteristics of the "bad" chips would be found to be different - something like NOT turning off a peripheral, as being responsible.

Yes, that's what I've been hoping for.  If there's a software fix - some bit in some register - then no harm, no foul.  I just don't know what peripheral consumes 150uA at 3.3V.  I know it's not ADC or BOD because I can measure those as using the same current on both good and bad chips.  But I don't know for sure about WDT.  In my test sketch I turn it off, but Kevin doesn't bother, I think because the Arduino bootloaders all turn it off by default.  But turning it off in the sketch, or not, has no effect on current, so I've assumed it was already off, as does Kevin.  But thinking about it, it might be worth changing the fuse setting so that WDT is always on.  If that raises the sleep current on a good chip to, say, 150uA, but has no effect on a bad chip, that would be very interesting.  But otherwise, I just don't know what other peripheral could have been left on.

On the other hand, it seems like there is a specific time niche that these "bad" chips were made, based on some markings - no?

Why would it be happening with only that package of the 328P (at least so far)? Why not the DIP package, for example? Is there a reasonable explanation for that?

Well, we may have an example of a bad DIP package.  See Ralph Bacon's video on using the DS3231 to wake up an Arduino.  He puts the 328P-PU to sleep, then manually disconnects power from the DS3231, and ends up with a sleep current of 150uA.  He blames that on the I2C lines, but he has disconnected the power to the module, so I don't see how that could be the explanation.  I've written to him asking for the markings, and asking him to try again with an older chip to see if the current changes, but have not heard back from him, and it's been a while.

[Peabody]

Speaking of which, could you post a link to where you bought yours?

This is the supplier I got them from

  https://item.taobao.com/item.htm?id=571250487735

if you're not experienced with buying from Taobao, well, this isn't really the thread to explain it but probably spreenow is the "easiest" agent to buy through.

I have no experience with Taobao, and suspect it wouldn't be a good source for most hobbyists in the US.  Maybe that seller has an AliExpress or eBay affiliate with the same current stock.

[Peabody]

I think we now have what appears to be a definitive answer.  Here's what I posted this morning on AVRfreaks:

"Kevin Darrah sent the suspect chip to a company called Sensible Micro, which has lots of experience testing for bogus chips.  They decapped both the bad chip and a known good chip and compared them.  Their report is included in Kevin's followup video:



"Their conclusion is that the part is "suspect counterfeit".  It contains the logo "Aries" instead of "Atmel", and there are other differences.  So unless Microchip engaged someone else to make chips for them, it looks like this is indeed a counterfeit.

"The bad news is that this means the entire "clone" market may now be infected with bogus processors, with no practical way to tell the difference.  I don't know whether Kevin plans to refer this to Microchip, but if anyone here has a good contact there, you might refer them to Kevin's videos.  One thing that would be nice to have is a test sketch of some kind that works one way on a good chip, but another way on a bad one.  Microchip might be able to provide something like that.  Removing regulators and LEDs to test for sleep current isn't practical for most hobbyists."


I wish I had better news.  I don't begin to understand how it could be profitable to duplicate all of the engineering for something as complicated, and cheap, as the 328P.  But it seems that has happened, and it could affect any clone Uno, Nano, Mini, or anything else that uses the 328P.