Author Topic: Board repair : HOBART " hood " type dish washer  (Read 3447 times)

0 Members and 1 Guest are viewing this topic.

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Board repair : HOBART " hood " type dish washer
« on: September 16, 2018, 12:18:08 am »
Hi all,

Well it's not as fancy as a 50GHz RF generator or spectrum analyzer, but well it's still electronics, so what the heck ! This forum is about electronics... not just fancy lab equipment, right ?  :P

I am trying to save the life of an old professional " hood " type dishwasher. Can't find any service manual or even bare schematics for it. Did find the user manual though, better than nothing...

See some pics below.

These things cost an absolute fortune compared to consumer dish washers, so it's well worth having a go at it ! 
My friend had to cough like the equivalent of 2500 USD to get it, used, when it was 10 year old ! I dread to think what it cost brand new...
Service tech from the manufacturer, told them the main board was toast, and cost 1800 USD to replace !
So it's worth a shot isn't it ?

It was manufactured in 2005 I believe.

Manufacturer : HOBART , from what I gather, it's a big name worldwide for this type of gear, and is a USA company.

Model : AM900 - 14

The main board has 12 relays,  4 of which are unused (not wired to anything), and the PCB has provision for fitting 8 more relays.
The IC socket for the EPROM / firmware, also has 4 more pins than the EPROM chip has. I also see at least 4 unpopulated IC foot prints, and also an unpopulated 8 way connector foot print. So clearly this board was common to several different models, some of which must be more complex than this one, with a larger EPROM / Firmware, and more relays / things to control.  So I guess it's common to the entire " AM900" range, not just the " - 14 " model I am having to work on here.  So schematics from any AM900 model would probably be relevant / helpful....

Will try to contact the US Head Quarters, begging for the service manual, but I am not holding my breath....

So if any body out there has some info on this type of gear, worked at HOBART as a service Tech or whatever, or just knows some good magical website that carries schematics for this thing.... that would be much, MUCH appreciated indeed !  :-+


Until then, I am trying to do what I can without schematics, guess work and poking around as you do...

Thanks for any help...


« Last Edit: September 16, 2018, 04:35:06 pm by Vince »
 

Online drussell

  • Super Contributor
  • ***
  • Posts: 1188
  • Country: ca
  • Hardcore Geek
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #1 on: September 16, 2018, 01:29:56 am »
What is the failure mode?  Does it not even run at all?
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #2 on: September 16, 2018, 01:47:19 am »
Does the display show something ???,

you have an green smt fuse near the main transformer, check if open ??? 

you have an  small bridge rectifier too, check on the plus sign if you have an dc voltage around the one written on the big "brownish"capacitor


some docs :

http://www.hobart.co.kr/data_file/IO-AM900-v1.6.pdf

http://cearcommercial.co.uk/uploads/hobart-manual-service-manual-am900-amxxt-auxxt-aup-serie.pdf

part list:

http://www.hobart.co.kr/data_file/AM900Dec09.pdf

one who has tons of services center at the end ??
http://users.telenet.be/kraisy/hobart/WWW/Wash/AMX-14-01_Standard_SPC-22338-002-EN.pdf
« Last Edit: September 16, 2018, 02:01:46 am by coromonadalix »
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #3 on: September 16, 2018, 03:42:34 am »
Thanks for all the links !  :)

That service manual sure is helpful ! Apparently the main board was redesigned at some point, and it does not match the one I have at hand. Still, it's still helpful. The front panel looks like it has remained the same, so all the good info about entering the "Service mode" using it, is a gold mine for sure.

As for the symptoms, the owner said that it worked just fine then overnight, out of the blue, it refused to turn on ! The front panel was totally unresponsive and displayed funky/abnormal things, and the dishwasher would not budge at all no matter what key you would press.

So it definitely sounds like it's a problem with the electronics rather than the dishwasher 'hardware" as such.

What I found when I fiddle with it, was pretty much what he said : when you power up the dishwasher, the front panel lights up, but is totally unresponsive, and the dishwasher would not do anything, zero sign of activity what so ever...

The front panel lights up (solid) all the decimal point of the 6 LED displays. No other LEDs are lit.   Looking at the service manual and user manual, this is not at all indicative of an error code, and if error code there was, the front panel should still be "responsive".

So my take is that the front panel board is toast. It has it's own 8 bit micro to handle itself, and communicates to the main board via a 10 wire cable.

Of course, the main board sends commands/orders to the front panel, telling it what to display... however, I don't think that even in the event that, at power up, the front panel might be unable to establish communication with the main board (because, say, it's defective in some way), then I would assume the front panel should still be able to "behave" : display some thing that makes sense to the user, and have some " life " left in it : pressing some buttons should be able to trigger something on the front panel... seem like a reasonable expectation.

If not so, if the front panel is all over the shop simply because it fails to communicate with the motherboard, then... honestly I would consider this as very poor/crap design !  :-[

So my starting point at the moment, is to concentrate on the front panel board rather than the main board.

The front panel does get power from the main board, obviously, since it manages to light the decimal points on the 7 segment displays.. which is about the only thing it does !

On site, I checked with my DMM across the decoupling electrolytic cap on that board, I did see 4,9V, good enough for a 5V digital rail.

Now I have brought the two boards home, so I can start working on them on the bench/lab. HAven't had time to, so far.. wen to see the dishwasher late yesterday.  Will start working on the board rzeal soon. Need to make some space on the bench first, as I am currently working on fixing an old 2215 Tek scope, parts all over the bench obviously !  ;D Will put that scope aside, and start working on this front panel board.

I am not equipped for working on SMD stuff though !  I can barely read the writing on the IC packages, and I need to get super sharp & fine probes for my DMM, otherwise I will just short 3 pins at a time, every time I try to probe anything !  :-\

I am also trying to figure out how the two boards are connected... I don't think it uses the UART, because the interconnecting cable as you can see on the above pictures, is several meters long, and is not even shielded, doesn't even have twisted pairs. Running a RS-232 link in on this long a cable, in such a highly "noisy" environment " (several motor pumps, mains all over the place etc...), is suicide at best.  Maybe if they run it at a massively low speed, like 120 BAUD or something, and add some form of error detection, CRC or whatever... maybe that couuuuuuld work ? Maybe  ? Hmmm.... doesn't seem like good engineering to me anyhow ...

I notice that the micro on the main board (a SAF C505 , modern 8051 derivative), features a CAN bus ! Yeah, that would be a much more logical choice here, that's exactly what the CAN bus was designed for !  So, I assumed that the micro on the front panel would have an integrated CAN controller as well... but it does not ! On the front panel it's a more modern part, a PIC 16F628. It's a bare bone chip : only 2KB of Flash/code money, a few bytes of RAM, and a UART... no CAN controller.   Well... strange. There is a mystery chip on the front panel board (ran the part number in Google with zero relevant result ! :-/ ). So I can only assume that it must be a dedicated CAN controller, and that the PIC MCU is controlling it via its UART. Seems convoluted (and expensive).. why not just get a micro with a built-in CAN controller, just like they did on the main board....   
I need to spend some time with the DMM, to see what pins are connected to what, start tracing things here and there...

The mystery chip (see pics) is a 20 pin IC which bears what looks to me like the Texas Instrument logo.
There are two lines written on it :

First one, next to the TI logo, reads :   51E6TVT
second line reads :   TPI    C6B596

Neither line returns anything relevant in Google....

I understand to use ultra short / "coded" part numbers, on tiny SMD packages... but in the case at hand, it's a bloody 20 pin chip, plenty of space to write the actual part number on it, why put some non-sensical numbers on this thing ?! Drives me nuts...  ???


Anyway, need to clear the work bench, plug these boards and get working... just hoping that the PIC micro is not toast because somehow I very much doubt HOBART would be willing to send me the HEX file to flash a new chip !   :palm:

Not that I have a chip programmer anyway, mind you... yet another thing on my mile long "to buy" list !...




 

Online drussell

  • Super Contributor
  • ***
  • Posts: 1188
  • Country: ca
  • Hardcore Geek
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #4 on: September 16, 2018, 09:41:29 am »
Firstly, check to be sure all power rails are clean DC, rather than just checking voltage with a multimeter.

Simple things like bad filter capacitors commonly cause these kinds of weird problems.
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #6 on: September 16, 2018, 12:13:21 pm »
The chip with the red stripe
Can you give us all the markings on it,  you have pads for an crystal near it, must be an controller of some sort.

You have 3 long pads with solder near it, must be programming pins ....



On the main pcb,  can you give us the markings on the plcc44 chip,  must be the "brain" cpu  since you have an 28pins eeprom next to it.

You could check all the idc 10 connector pins solders going to the display, if the solders are ok ???
And the same on the display pcb

Check the flat cable connections, they mane have some loose crimping in them ??
« Last Edit: September 16, 2018, 12:23:08 pm by coromonadalix »
 

Offline cvanc

  • Frequent Contributor
  • **
  • Posts: 598
  • Country: us
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #7 on: September 16, 2018, 01:35:30 pm »
Investigate the pushbuttons for possible failure ("stuck down").  Replacements are easy to find.  Good luck!
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
Re: Searching for schematics for a HOBART " hood " type dish washer
« Reply #8 on: September 16, 2018, 03:41:01 pm »
I would have said the humidity may / could have caused some problems, but i don't see any corrosion on surface parts,  its a well sealed dishwasher
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #9 on: September 16, 2018, 06:03:05 pm »
Thanks everyone for your contributions !  :)


TPI C6B596C  : hmmm, so might be a  "power" shift register then, will have to confirm that by probing around and following traces, but hey why not. I am not sure why they would need any "power" shift register for, though. There are only LED's to drive on this front panel board, and the MCU is able to drive LEDs directly, says its datasheet. To drive the 7 segment displays, 6 of them, from what I can see they do it this way : a couple "normal" shift registers (74HC595D), 16 lines then, then there are 3 transistors driven directly by the MCU. It all adds up nicely : 8 segments per display (the decimal point...), fits an 8 bit shift register. So one shift register for each of the 3 digits display, then the 3 transistors drive "columns" of two digits : one tranny for the top and bottom left most digits, another one for the top and bottom middle digits, and last tranny for the right most top and bottom digits.

At least this bit is limpid...

Well...I just noticed that the main board also carries two of these TPI C6B596C chips !  Looking at them, it looks like most likely, 99.999% certain, they drive the 12V coils for all the relays. So they indeed would be power shift registers, alright.... makes sense to drive relays on the main board, but what the heck are they doing on the FRONT PANEL board? there is nothing to drive there but a few LED's ! :-/   Maybe they don't drive LEDS, maybe they drive some line on the interface connector that goes to the main board... to do... I don't know what.


Quote
The chip with the red stripe Can you give us all the markings on it, 

Yeah sorry the pic was blurry  there. At this short a distance, the camera did not have enough depth of field to be able to focus on both chips at the same time. Anyway, this chip is the MCU I was mentioning : the PIC 16F628, with just 2KB of Flash and no CAN controller... just a UART. 

Quote
you have pads for an crystal near it, must be an controller of some sort.

Yes indeed it's the MCU, but these pads aren't connected to the oscillator pins of the MCU. I don't know what they are for. Some passives maybe.

Quote
You have 3 long pads with solder near it, must be programming pins ....

Well spotted... these are not for programming, they run straight to the oscillator pins of the MCU !
3 pins in a row.. not a typical foot-print for a Crystal is it. However would be a good match for a cheap 3 pin ceramic resonator, me think.
I guess they used it for prototyping.. and decided to do without it in the end. The MCU actually has an on-chip 4MHz oscillator, according to the datasheet, so it must be using it. That will be easy enough to probe with the scope. That will also let me see if the MCU is actually running or not, an essential test...


Quote
I would have said the humidity may / could have caused some problems, but i don't see any corrosion on surface parts,  its a well sealed dishwasher

Yes, both boards are like brand spanking new. All shiny solder joints and IC pins, not the slightest sign of corrosion anywhere. They used a top quality gold plated socket for the EPROM chip too.  The electrolytic caps  (only 3 of them, two on the main board for the 12V and 5V, and one on the front panel board), don't show any sign of anything. No bulge, zero sign of leaking electrolyte, zero corrosion in their vicinity...
Of course I will have to test their ESR (need to buy an ESR meter first ! LOL ) and ripple with the scope, since electrolytic caps no matter how good, all fail eventually... but my initial gut feeling is that they are probably just fine.


Quote
Investigate the push-buttons for possible failure ("stuck down").  Replacements are easy to find.

Yep , definitely. The most likely to fail being the on/off button of course, since that's the one that gets the most use. Had just this happen on my personal washing machine, which would randomly turn off in the middle of the washing cycle (not fun ! )... I swapped the on/off switch with one of the other switches on the front panel, that I never ever use, and I was back in business in no time !  8)


Quote
On the main pcb,  can you give us the markings on the plcc44 chip,  must be the "brain" cpu  since you have an 28pins eeprom next to it.

That's the MCU I was mentioning earlier. A  " SAF C 505 CA ".   A very basic 8051 derivative, whose sole interest lies in its built-in CAN controller.
Which is why I still believe, for the time being, that they do use a CAN bus to communicate with the front panel. It just makes sense. It's the only bus that would be reliable in such a noisy environment, and over such a long cable.  Plus, now I think of it : it somehow they were fool enough to use the UART, at the very least they would have to use actual RS-232 voltage levels, to have even a remote chance of getting it to work... but I don't see any RS-232 drivers anywhere, not on the front panel, and not on the main board. So they would have to use 5V / digital level straight from the MCU's... which is pure folly and has zero chance of working. 
Plus, if they aren't using CAN.. then why the hell use a CAN enabled MCU on the main board ? I mean, this is the main asset of this chip ! And it they didn't want it, then they could just get the same chip withOUT a CAN controller... because it is available in  CAN -less form.

So CAN it muse be. I do have a book on CAN.. bough it 20+ years ago when I was studying electronics... never had the opportunity to work on a CAN bus so far, this is my chance I guess ! LOL  will have to read that book a gain, needless to say I have forgotten all about it, 20 years down the line...



OH !  just had an idea... I will give the possibility of an RS 232 link a LAST chance (I am trying hard) :  on the front panel there is a little 8 pin IC, that reads "2903", with a few passive around it, and a couple diodes, which seem arranged like two identical/symmetrical sections. 
Let's say it's an LM2903, which is a dual comparator. Maybe this could constitute a " hand-made " RS-232 driver ? But if so, they would need "high" voltages to come from somewhere... maybe they use the 12V from the main board ? Why not, a +/- 12V RS(-232 link would be a common sight, and would have of course much better chances of working than a raw digital 0 / 5V link... 
So that means I should be able to see +12V some on the 10 pin interface connector, coming the man board... easy enough to check ! :-)
But, we still need the main board to send the NEGATIVE 12Volts to the front panel... and the power supply on the main board is limpid (as far as I can see !)

- A linear supply, a transformer with a single 12 winding, full wave-bridge rectifier, and big 2200uF filter cap. That's used to drive the relay coils.
Then off this 12V DC,  I can see a little DC-DC converter chip, which I assume must be stepping the 12V down to 5V, to supply the digital stuff on the board.
So if there is negative 12V somewhere on that main board.... I guess there must a charge pump somewhere.. but I don't see that anywhere (for now at least). I would assume it would use fairly large/quite visible, capacitors next to it.. and I don't see any "big" capacitor anywhere on the board, other than the two filter caps for the 12V and 5V rails... hmmm...


ORRRR..... maybe it's a mix of both my assumptions : maybe this 2903 IS a line driver.. but maybe it's used to drive CAN lines rahter than RS-232 ?! In this case that means that there would be no (hardware) CAN controller on the front panel... meaning that the PIC MCU would have to do it all in SOFTWARE ?!  Never heard of, is it ??? Quite a task at any rate, and it has only 2KB of Flash and 224 bytes of RAM to achieve that ! Seems unrealistic to me... unreasonable at very the least....



Anyway !  Lots of ideas have been thrown here, at least I have lots things to check to get me started, lots of leads, that's good !!! :-)

Will report back with my findings. But first things first : need to buy some sharp/super fine probe tips for my DMM, in order to be able to efficiently and reliably probe all this surface mount stuff !  Otherwise I will just waste time, pull my hair out and short pins no matter how careful I try to be, and possibly doing damage to the board ! I don't want that...
Need to buy some kind of magnifying glass, because my eye sight is not good enough to work comfortably on this small stuff...

Oh BTW : I forgot to mention it, but there are NO components on the underside of either board, not even a single tiny passive component. The underside carries just traces, and lots of gold plated test points for the " bed of nails " in production, as you do.


Thanks again for your help chaps, much appreciated ! Stay tuned !  8)

I renamed the topic into  " board repair " rather than " searching for schematics "... seems more relevant since we are now well into trouble-shooting now !

Don't know what the outcome will be, if I will be able to diagnose it or not... but regardless, I find myself quite enjoying reverse engineering these boards, it's fun to play  with nice industrial boards !  8)
« Last Edit: September 16, 2018, 06:24:49 pm by Vince »
 

Online drussell

  • Super Contributor
  • ***
  • Posts: 1188
  • Country: ca
  • Hardcore Geek
Re: Board repair : HOBART " hood " type dish washer
« Reply #10 on: September 16, 2018, 07:29:46 pm »
Well...I just noticed that the main board also carries two of these TPI C6B596C chips !  Looking at them, it looks like most likely, 99.999% certain, they drive the 12V coils for all the relays. So they indeed would be power shift registers, alright.... makes sense to drive relays on the main board, but what the heck are they doing on the FRONT PANEL board? there is nothing to drive there but a few LED's ! :-/

It is entirely possible that they just use the same chip so there is one less part on the BOM, even though it is on a totally separate board, meaning one less chip to source and stock in the PCB manufacturing process across their various models.
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
Re: Board repair : HOBART " hood " type dish washer
« Reply #11 on: September 17, 2018, 02:06:50 am »
on the main board, near the big capacitor, you have an 6 pin unsoldered connector pads, the chip near it seems to be an rs232 interface not populated ??? you see some capacitor foot print going pin to pins ???  i may be totally wrong

For the display board, the 3 pads must be to supply an clock source to program the chip ???
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #12 on: September 20, 2018, 11:11:09 pm »
OK, spent a few hours probing the main board, main board only, with the front panel board disconnected. I wanted to get a status on the main board alone, first, to see if it were dead like the Hobart service Tech pretended it was...

First with the board powered off, I looked at some PCB traces and followed them, to try to figure out what they were going to. Mostly, I traced the 10 pin interface header that goes to the front panel board, and I also looked at the pins of these " TPI C6B596C " chips, to see if they were indeed those power shift registers, or not.

Then powered up the board, looked at the two power rails for DC level and ripple/noise, then checked for any activity on the 10 pin interface header. Then probed the MCU a bit too, for CAN/UART/reset/clock, the basics then.

So, here are the results, with a few pics down below to illustrate :

TPI C6B596C

Yes, it is indeed that power shift register, so thanks again coromonadalix for finding that datasheet !  :-+
They are used to directly drive the coils of all those relays on the board.

Power Rails

The 12V rail,

Well, I thought it was 12V since it feeds only the relay coils, which are marked as being 12V... however it is actually putting out a healthy 15V !!!  A fair bit more then... however I don't think that would be high enough to do any damage to the relay coils, so I am not worried. Also, I don't think it's uncommon to slightly "overdrive" electro mechanical devices like this. This rail doesn't power anything else on the board but the relays, so it's fine.
Also, I really don't see how a fault in the power supply could lead to having MORE voltage. I mean, there is nothing in there, just a bare bone linear supply : transformer tap, a full wave bridge rectifier, and a big filter cap, as you do. Nothing more. Can't get any simpler than this. If the transformer were faulty, it would short internally... which would result in a lower voltage, not higher. I don't see any mechanism by which a faulty transformer could output more voltage than expected ?!  Now I am no expert so I willing to stand corrected of course !... Oh wait... if the short is in the primary winding not secondary, the turn ratio would  mean a higher output voltage, ahem... OK, so let's not rule out a dodgy transformer just now...

Anyway, having a a slightly high voltage for the relays is definitely not going to cause the symptoms we are having, so I moved on, for now at least.

I do notice however that the electrolytic filter cap (2200uF, the big/tall brown one), is rated at only 16V, only ONE Volt of derating  !  That's scary low !  :palm:

I mean even if the supply was producing 12V not 15V, it would still not be enough derating.. so I feel some planned obsolescence there !... grrr... strange, it's a mega expensive professionnal bit of gear, not a crappy dirt ycheap consumer grade junk... so I am surprised...  ???
If I can get these boards working, might replace that cap with a 25 V one to be on the safe side.

As for ripple, there is zero 100Hz ripple, because well, the load on this rail is NIL, since there is zero activity from the relays...
However there is about 60mVp-p "crap", (80mV worst case, every now and then) which " leaked " from the DC-DC converter nearby, which produces the 5V rail. Again this is hardly a concern I think, as this is only going to power the relay coils.. they don't care about 80 or even 800mV of noise...


The 5V rail,

As I just said, this rail is produced by a DC-DC  converter / switch mode supply, fed by the 12V rail.
DC level is spot on, at 4,98V.   Ripple is the typical saw-tooth one expect from a switch mode supply, with the usual large transient when the FET switches.
The overall peak to peak amplitude is about 20mV including the transient, but only 10mV if we only consider the saw tooth. From experience on Tek scopes, which are obviously much more delicate and accurate than an industrial board.. 20mVp-p is acceptable for the 5V digital supply. So, I am sure it's just fine for this industrial board...

The converter switches at about 40kHz, which is typical of these old converters, so I was not surprised.

Filtering is done by the smaller SMD electrolytic cap you can see next to the big brown one. As for voltage rating, again I am scared : markings are not as clear/explicit as they are on the other cap, but if the "6E" marking on it means "6 VOLTS", then again that's only one volt of head room, ridiculously low. So might replace that one too, later down the line.


Micro-controller

Quote
on the main board, near the big capacitor, you have an 6 pin unsoldered connector pads, the chip near it seems to be an rs232 interface not populated ??? you see some capacitor foot print going pin to pins ???  i may be totally wrong

Not for RS-232 (connector would need to be 9 pin not 6). This one if for the serial programming of the MCU ! There is exactly the same (also unpopulated) foot print on the front panel board, and the pads on this connector go straight to the reset line and programming pins of the MCU.


I first checked that 5V was getting to the MCU... it was. then checked for the Reset line (active low), it was high, so good there too.
Then probed the CAN and UART output of the chip, to see if I would have activity on either of them, so I can finally know what kind of interface it was using... and big surprise : there is NO activity on either of them ! OK, maybe the MCU is not running... so I checked the clock with the scope... hmm yeah, clock is running, approximately 11,04MHz, hard to get an accurate measurement with a scope, but that was close enough to the 11,0592 MHz marking on the crystal, so that was not a show stopper. Still, I couldn't resist the temptation to measure it more accurately with my old Nixie tube frequency counter, which I recently bought.. this board would give me an excuse to actually use it for something useful ! LOL  So, hooked it up, and wow... it measured BANG ON 11,0592 !  :D  Not bad for a 60 year old counter, which doens't even have a TCXO.

So, since the crystal has not drifted by any significant amount, we can also rule out any timing issue which might have cause the main board and front panel to fail at communicating.

So, the MCU is running... but is not using the CAN nor UART, what the ?!



10 pin interface connector

So, since the MCU appeared to use neither CAN nor UART to communicate, I was still very much confused as to how on earth the two boards communicate ! :-/

So, I started to look at all the pins on the interface header, and followed them on the board to see where they would go, hoping it would give me a clue...

It's actually way simpler than I feared : out of 10 pins, 5, half of them, is devoted to the ground. 2 are for the 5V supply. There is no 12V. The 3 remaining pins are signal lines, which I followed.. with some difficulty because they changed PCB side several times using via...  It was a bit of  a pain to follow the traces ...

Still, I managed. So, 2 traces go to digital chips, jelly bean 74HC "glue logic" chips. The last and third trace goes into an analog section of the board, where there is this 8 pin "2903" chip, like there is on the front panel board as well.

Then I powered the board back up, and probed those 3 lines with the scope, at the header. Boy that was most interesting, at last it answered my questions !  :D

So, this is clearly a " home brew " serial link ! There is the typical activity you would expect on a 3 line serial link.

The two lines which go to digital chips, are outputs : clock, and data. The last / third line, which goes to the analog section, sees no activity, zero volt or almost (0.18V). Clearly this the return line from the front panel board, and since it's not connected, of course there is  no activity on there. The analog circuitry is certainly there to "shape" the signal back into a usable digital signal, then fed to the MCU.

Data line :

See screen shot. The MCU is constantly spitting out data frames. The "period" is about 10ms, so 100 frames per second... an order of magnitude faster than what is required to communicate with a front panel... so no worries there, the system must be very responsive !  :-DD

The frames are about 5ms long, separated by an idle time of about the same length of time. Hard to be 100% sure where the idle period stops, and where the data packet starts, but roughly 5ms each, as a first approximation. Looking at the clock line (see below) is a better and much more accurate way of determining idle time and packet length, I think !

The contents of the frame appears to be changing, I could not get a stable "picture" of it on the scope. This does not necessarily mean that it's not sending the same date over and over and over again. Could still be the case, but that means that the message it's sending consists of more than just one data packet.

Zooming on the data "packets"/frames, I see that the "bit time" is about 100us.  So, that means that there are about 50 data bits in one frame, so the equivalent of 6 bytes or so, assuming the protocol is byte oriented, no way to know about that. At any rate, 50 bits gives plenty of room to implement a strong error detection, even correction, algorithm , doesn't it ? So my feeling is that a good chuck of those 50 bits are devoted to error detection and correction, which would "compensate" for the deficiencies of the hardware/physical medium they chose ! :-/  They didn't try to maximize noise immunity on the physical layer, rather, they live with it and handle it on the software/protocol layer... well, I guess they must have their reasons for that (would be curious to know, though !), whatever floats their boat hey !   :P


Voltage levels are logic level : 0 and 5V. So looks like there is no transceiver what so ever.. the board is indeed, shock horror, sending "naked" digital signal in the wild... my god.. and somehow it works !  :o

So there we have it. We now know the details of the link.. no UART, no CAN... but instead a home made serial link, and spitting out logic levels, zero driver to attack this long cable swimming in an ocean of electro-mechanical noise !  Wow..... they dared to do it, impressed by their audacity !


Clock line :

See screenshot.  It's perfectly consistent with what I observed on the data line : the clock runs at 10kHz, 50% duty cycle.  10kHz that's a 100us period... surprise surprise, that's precisely the bit time I measured in the data packets !  :)

Also, this clock signal is "gated", it is sending bursts of pulses. Here again it's consistent : the clock bursts last for 4ms, same length as the data frames. Then the clock is "muted" for 6ms, which corresponds to the idle time on the data line.

So it's all perfectly consistent....


Sooooo... so much for a main board that was supposedly dead, according the Hobart service Tech !  Sure, the board might still have some weird issue, you never know, but the service Tech had no way to study the board in details, in the field, like I just did here on the bench. So his diagnostic I think was wrong.. this board is very much alive, and trying hard to communicate with the front panel board ! There is activity on the serial link at ALL times, never stops !

So.. this again tends to confirm my initial gut feeling that the problem lies in the front panel board rather than the main board  8)


So, next time I get to work on it... I will switch to studying the front panel board in more detail...


See ya  later for the next episode !  ;D

« Last Edit: September 20, 2018, 11:52:53 pm by Vince »
 

Offline coromonadalix

  • Super Contributor
  • ***
  • Posts: 2235
  • Country: ca
Re: Board repair : HOBART " hood " type dish washer
« Reply #13 on: September 21, 2018, 02:48:35 am »
You don't need 9 pins to be an rs232 port, you can use rx tx ground at minimum ... , it would be enough to do communications, you an have other types or rs232 transceivers other than the famous max232 ... you have sipex transceivers ...

If it is the front panel pcb who's gone bad,  will it be hard to find this one too ??


 

Offline james_s

  • Super Contributor
  • ***
  • Posts: 11184
  • Country: us
Re: Board repair : HOBART " hood " type dish washer
« Reply #14 on: September 21, 2018, 04:15:28 am »
Those power rails look horrid! I would bet that if you replace all the electrolytic capacitors it will completely fix the problem. They are by far the most common failure point in gear like that.
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #15 on: September 21, 2018, 11:49:08 am »
You don't need 9 pins to be an rs232 port, you can use rx tx ground at minimum ... , 

Yes of course, unless you are connecting to a modem, then most applications don't use the H/W flow control lines, so TX/RX and ground is all you need.
But regardless of what lines you make use of, the de facto standard for a serial port/RS-232 link, is the good old DB-9 plug.. even on modern stuff. Has been for 35 years... used to be DB-25 before that.. but who remembers that. So in order to fit a DB-9 connector on the PCB, well you need 9 pads/holes of course  ;D

Since this is an industrial board, they are even less likely to break such a "standard", they no need to male their life more difficult than it needs to be...


Quote
If it is the front panel pcb who's gone bad,  will it be hard to find this one too ??

The complete, assemble board you mean ? Finding a used one is out of the question I assume, but it's likely that Hobart would be willing to sell one. Their service Tech looked at that dish washer 2 years ago or so, and said the main board was available.. for 1500 Euros. So I guess 2 years down the line, boards are still available, since this machine is not that old. I mean it was built in 2005, but from what I understand in the service manual, this model was still available in 2011, at which point they redesigned a bit the main board a little bit, and front panel remained the same, from an electrical perspective. Only they re-arrange the buttons and LEDs slightly differently.  So I would say their is a very decent chance that Hobart would still be able to supply a front panel board for this machine...

Hopefully I can fix the front panel ! Everything is replaceable.. even the MCU... however of course Hobart it unlikely to give me the firmware / HEX file to flash a replacement MUC if I end up replacing it !  >:(  So let's hope  that the problem is not the MCU !  :palm:

I will start playing with that front panel today I think.... I feel like it !  ;D

 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #16 on: September 21, 2018, 11:57:23 am »
Those power rails look horrid! I would bet that if you replace all the electrolytic capacitors it will completely fix the problem. They are by far the most common failure point in gear like that.

Yeah we shall see what the final outcome is.

However I don't see how it could cause our symptoms ? The 5V supply is spot on and ripple and noise is within reason, so unlikely to upset the MCU on the front panel. Does't upset the MCU on the main board to start with, it seems. Also, that "horrid" 5V supply gets to the front panel board via a 3 meter long cable that's not shielded and is surrounded by several electrical motors spitting out tons of powerful noise in close vicinity, so even it the 5V was super mega clean when leaving the main board, I very much doubt it would still anywhere near as clean when getting to its destination on the front panel board, what do you think ? So I would assume that the front panel board is engineered to cope with a dirty incoming 5V, because it has no other choice...
That being said, all there is on the front panel board to "clean" the incoming 5V supply, is a little SMD electrolytic cap. I don't see any filtering network to get rid of high-frequency stuff, no LC filter filter like Tek does on their scopes, for example.

At any rate if I see that there is activity on the board but that it still doesn't work, in order to rule out a noisy 5V supply causing perverse/naughty glitches that might upset the MCU, I would just try powering the boards (or the front panel board alone) with my linear/clean lab power supply, and see if that makes any difference...


I wish I had a stock of brand new SMD caps to substitute, at least for test purposes, but I don't...

Anyway, should be reporting soon on that front panel board, I feel like working on it a bit today...  ;D
« Last Edit: September 21, 2018, 12:17:23 pm by Vince »
 

Offline james_s

  • Super Contributor
  • ***
  • Posts: 11184
  • Country: us
Re: Board repair : HOBART " hood " type dish washer
« Reply #17 on: September 21, 2018, 03:51:41 pm »
Oh, I see now that says 0.8mV, I thought the ripple said 0.8V, big difference there.
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #18 on: September 21, 2018, 06:56:05 pm »
Almost !  ;)  Not 0.8mV but 80mV (60mV typical) on the 12V rail, which we don't care too much about as it only powers the relay coils.

The 5V rail has 20mVpp ripple and only 10mV if you disregard the big, brief transient when the FET switches, as you can see from the relevant waveform/picture.


I just worked 30 minutes on the front panel board, this time plugged into the main board... and I think I might already be onto something interesting, looks promising!  :D 

And... and the cable is not 3 meter long.. even worse than this. I actually just measured it, it's twice that, SIX bloody meters long ! For people in the US, that's 20 feet ! 6 meters, not shielded... with bloody 5V digital signals traveling in there, just... wow.........  :o
« Last Edit: September 21, 2018, 07:00:53 pm by Vince »
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 17552
  • Country: nz
  • Taupaki Technologies Ltd. NZ Siglent Distributor
    • Taupaki Technologies Ltd.
Re: Board repair : HOBART " hood " type dish washer
« Reply #19 on: September 21, 2018, 07:18:23 pm »
That mightn't be a problem Vince if the data lines are correctly terminated as 5V logic thresholds give plenty of tolerance for bus noise.
Avid Rabid Hobbyist
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #20 on: September 21, 2018, 11:42:26 pm »
Hi Tautech, nice to see you here, didn't think a dish washer would get much interest to be honest !  ;D

But trust me, I don't care what machine these board are used in ! All I see are a couple nice board and I am having a lot of fun trying to reverse engineering them and trouble-shooting them ! It is just as addictive as crack is for some people.... only it better for your health, gets the neurons working, instead of frying them ! LOL

Sorry...

So, non the lines are not terminated as far as I can see ! All there is, is a 19K resistor IN SERIES with the lines, right by the header. I assume they are there to limit damage to the board / IC inputs, in case of a massive noise surge from some motor or what not. I see that every where.. though usually they are lower values, like in the hundreds of ohms, not kilo ohms.


Anyway, as I said I think I am on to something, pretty sure I nailed it !  :D

A couple days ago I actually burnt the midnight oil, and reverse engineered the entire front panel board, to the tiniest detail. It's all spread over several sheets worth of notes.. but it's all there. Maybe if I feel courageous, I might draw the schematic for it !

So I know at last, exactly how it all works. This sure made trouble-shooting easier than on the first day...

To describe it briefly :

the MCU has 16 I/O pins. It sends data line to the main board just like the main board does : no fuss, the output pin of the MCU goes straight to the cable header ! OK, they added a 100ohm resistor in series, but that's about it...

There are x3 8 bits shift registers, daisy chained, so a total of 24 outputs, that drive all the LEDs on the board :  5 "discrete" LEDs, a couple colour/RGB LED, which are wired in parallel, and then the pair of 3 digit LED displays.

The MCU refreshed theses registers every 4ms or so, and it takes about 0.35ms to clock all 24 bits/LEDs out.

The LED displays are organized this way : each display has its own register which drives its 7 segments and decimal point. Then there are 3 transistors, driven by dedicated individual I/O pins on the MCU. each transistor drives a "column" of digits : first tranny, switches the left -most digit of the upper and lower display, second tranny, switches the middle digits, third tranny the right-most digits of each display.


As for the push-buttons (which work fine, none are stuck), only 4 of them so there is no value in multiplexing them. They are just wired straight to the MCU, one I/O pin for each button.

Then there is the communication with the main board : a little analog circuitry. An LM 2903 dual comparator chip. One comparator processes the Data line, other one the Clock line of course. Circuitry is symmetrical, as you can see from the pictures below. So this makes trouble-shooting easier : I can compare the two and see if there are any differences.


Now, here are the good news : I think I foudn the problem, and it is NOT the MCU !  YEAH, the only potential show-stopper has been ruled out !!!  :phew:
MCU is just fine : It constantly drives the serial clock and data of the LED shift registers, and also drives the 3 transistors needed for the 7 segment displays.

However, it does NOT send ANY data out to the main board !  But I think I know why... I think it might just be normal behavior from its part, because.. it is NOT receiving the clock from the main board ! Data, it receives fine, but no clock !
So that must be it... if the front panel board is unresponsive and displaying nothing but the 6 decimal points of the LED displays, must be its way of saying "hey, I am alive but I can't establish communication with the main board, help !!! "

So my hope is that if I can get the clock to get to the MCU, it will all by magic resurrect  !  8)  Well, that's the plan at this point any way !  ;D


So this brings us back to the analog part of the board, this dual comparator.

To begin with, what do the clock and data signal look like, at the headr pin, after they traveled 6 meters of cable ?? Believe it or not... they look perfect ! OK, my bench is much more comfy than a dish washer, that's for sure, but still...
The signals are still perfect : a full/healthy 5V in amplitude, fast edges, no ringing, undershoot nor overshoot... really they look just as nice as when they left the main board !

Then tehy go through a 10K resistor, in series with the lines. Then they travel a few centimeters on the PCB, and eventually arrive to the comparator circuitry.

I just drew the schematic for this circuitry, so you can better understand. Did that in a hurry with Digikeys on-line schematic capture tool, first time I use this kind of on-line tool.. quite practical and intuitive, did it in no time. Will not hesitate using this tool again, next time I need to draw/show a small bit of circuitry.

I drew only one comparator, since the circuitry is strictly identical for both the Clock and Data signals.

As you can see, the comparator has zero hysteresis, which I found odd but hey, what do I know right !  :-[
The threshold is set once and for good, at 50% of Vcc, by a simple voltage divider, made up of two 8,6K resistors.

There are some losses in the cable, so Vcc is 4,95V at the main board, 4,90 at the front panel board. So divide that by two, and the threshold on the inverting input of the comparator, measures at 2,45 Volts.   So since the incident signal voltage levels are ground and 5V... that makes for a noise immunity of nearly 2.5V for both logic levels... right ? Quite comfortable isn't it... might explain partly why they managed to get this thing to work reliably. The (supposed.. but highly plausible...) error correction mechanism implemented in software at the protocol level, probably does the rest... and the two combined, make up for a reliable link between the two boards.

Anyway, so the problem is that the comparator responsible for shaping the Clock signal, outputs a solid... zero volt, flat line, not a glitch... as boring to look at on the scope, as can be !

Solder joints look nice, but still, I probed right at the IC pins to rule out any potention joint problem : the incoming signal does get to the non-inverting input, and the 2,45 V threshold voltage does also make it to the inverting input. As for the mandatory pull-up resistor at the ouput/open drain, no problems there either.
So, that comparator has everything it needs to work... but doesn't.

Think I found the problem... the amplitude of the incoming clock signal, measured at the comparator INPUT (not at the header), is lower than 5V.... MUCH lower.
When you look at the Data line, handled by the other comparator in that same chip, amplitude it down too, but still on the high side of things : 4,2 Volts.
However on the Clock line, it is down to.... 2,30 Volts !!!   Yes, that's 0.15V LOWER than the threshold !  So of course the comparator never switched its output !

Now we need to figure out WHY is there such a massive voltage drop... so we can figure out how to fix it...
Could it be a defective comparator ? Maybe the non-inverting input is leaking badly and pulling the voltage down ?
Or maybe it is due to some fault in the external circuitry ?...

As you can see on the schematic, there is a feedback resistor in there...

When I probed the board all over the place, I measured all the resistor to figure out their value. I didn't read them by eye, because the packages are so tiny (about 2mm long by one mm wide.. what size is that, 0402 ?? Not too clued on SMD package size I must admit...  )

At that time, I measured the feedback resistor as  being 33K for the Data line, and 16K for the Clock line... though I couldn't figure out why it would need to be different than the other one.. but well, at the time I was not thinking, I was just busy doing brute force reverse engineering.. thinking was to come later....

So this resistor was my first suspect. I just looked at it again, and oh...
Although I don't have microscope, or anything other than my naked eyes (for now at least, hopefully that will chance fairly soon...) for reading the markings on this tiny resistors... I do however have a macro mode on my camera and I took a pic of this area ot the board. Now I can see ! 
that being said... looks like the 8.6K resistors of the voltage divider, as well as teh 1K pull-up, are labeled in a cryptic way, they read " 01B " and " 01C " respectively... eh ?!!  :o  So even if had a micro-scope to see them, I still would have needed to measure them anyway ! LOL
Why did they not used teh standard 3 digit coding like they did for the 33K resistors ?!  Does nay one know how to read this strange code, and the reason behind it ?? I am really very curious to know !  :(

Anyway, as you can see for yourself, this 16K resistor... is supposed to be 33K just like its neighbor ! I prefer that, makes more sense ! So, is the resistor bad ? When I measured it, there  was like a " capacitive " effect to it : at first the DMM would register like 11K, then it would slowly climb up to 16+ K or so, and stabilize there. Of course there is a small MLCC cap on the non-inverting input, as you can see... so could it be causing this issue ?  On the Data line, the 33K resistor registers properly at 33K spot on, and does so instantly, there is zero capacitive effect : you put the probes on the resistor terminals, and instantly bang, the DMM reads 33K, no delay.

So if we have this capacitive effect on the Clock comparator, is it due to a defective comparator input ? to a defective input capacitor ? Or is the resistor itself causing this weird behavior ?  That's what I need to figure out now !

I know for a fact that resistors CAN display such a behavior... at least the old. vintage carbon composition resistors.. but I don't know about modern SMD resistors.

Don't have much gear, but I do have at least, that shitty Atten 858D hot air station that's so popular... should be up to the job here I think, as the packages are so small, and it's only a two layer board wit low thermal mass.

I could just remove that resistor and measure it on its own to see it the symptom persist.. if not, then we know the problem comes from the comparator or capacitor, or.. yeah there is also a diode in there, forgot to mention it !  :P It's in reverse but who knows, maybe it's leaking badly and causing the significant voltage drop we are seeing... 

So I still have a little bit of work to fix this issue, but at least now I know that there is hope for this board : the MCU is not dead !  :D

OK, 2 AM here, time to get some sleep, and back to the bench tomorrow for the final act, can't wait  !  :)




« Last Edit: September 21, 2018, 11:55:13 pm by Vince »
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 17552
  • Country: nz
  • Taupaki Technologies Ltd. NZ Siglent Distributor
    • Taupaki Technologies Ltd.
Re: Board repair : HOBART " hood " type dish washer
« Reply #21 on: September 22, 2018, 01:35:00 am »
You know me Vince, I'll chip in with little bits and bobs anywhere to help anyone. You don't get to my age without discovering a few things along the way.
It is addictive especially when you get an understanding of electronic building blocks and how they're tacked together to make a working circuit which is were you're at and achieving good progress.  :)

Your hot air will easy handle an SOIC8 but tame the flow some and crank up the heat. 5-10s is all it should take before you can pick it off with tweezers. For the passives a HAKKO K style tip is magic as it will reach both SMD pads and then you just wipe the passive off the pads with the iron.
For better magnification a cheap jewelers headset is what you need and 2-5x is plenty enough for all but the tiniest SMD work.

Sounds like you'll crack it soon, keep up the great work.
Avid Rabid Hobbyist
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #22 on: September 22, 2018, 01:55:36 am »
SUCCESS !!!


 :D    :box:

3:30 AM ... couldn't help...had to give it a go before going to bed ! LOL  I am hopeless....


I removed the feedback resistor.... taken on its own, it reads just fine, 33K and no " capacitive " effect.

So I soldered it back to the board... gee this thing is even smaller than I first thought. It's not 2mm long, it's more like 1.5mm long  !  :o

Yet somehow I managed to solder it back in no time and with surprising ease, and good looking solder joints as well... maybe it was just good luck, or maybe watching Louis Rossman YT channel for hundred of hours, finally hammered the basics of SMD soldering into my head, somehow !  LOL


Then I removed the input capacitor... I am sure it must serve some purpose, maybe form a medium frequency low pass filter, along with the in-line 10K resistor, to clean the signal before applying it to the comparator input.. or God knows what else. But, given that I knew the signal was super clean, and this was just for testing purposes, I figured it would probably work just the same without this capacitor. So I removed it from the board... and it looked like it was open circuit ! Eh... Well OK why not. But I could not see how this could cause a voltage drop. After 30 seconds tops, I had lost that cap on the bench any how, so it was not going back in for sure ! LOL

So I was left with just the input diode, and the comparator itself, as remaining suspects. So I pulled the diode... read just fine, both ways. Still, I didn't put it back on the board (again seeing as it was not essential to circuit operation, was just there for input protection I assume), and powered the board back up....... and what do you know, the 2.2V clock signal now is back up to a healthy 5Volts, YES BABY !!!!   So that was it.... what do you think, could I be right ? Could the diode leak in reverse, which would force some current through the feedback resistor and create a voltage drop ? That's my best explanation so far any how. So, signal diodes can test good on the DMM, both ways, yet still leak badly in reverse and cause major disturbances in circuit !  :scared:
So I thought maybe the 4.2V I am getting on the Data line, is just the same problem, and its diode is on the way out too. So I removed it, and yes, this line too rocketed to 5V !  :D

So, with a healthy clock now getting to the MCU... I crossed fingers and pressed the ON/OFF button.... see for yourself !



In case the video doesn't work, here goes :

When I plug the main board to the mains socket, for a brief moment, a second or two tops, I can see, again, the decimal points light up.... but then they go off, and the front panel is completely dark.  It all seems dead.  But in my mind it was more like : " OK, since I assumed that the decimal points meant "can't reach the main board", then the points going off maybe mean that it managed to connect ! ".

So... I pressed the ON/OFF push button on the board, and what do you know, it works ! I hear a relay click on the main board, then all the LEDs light up on the front panel, then the two LED displays, put out some text !  :D  They cycle between 3 different codes, which make sense, from what I recall when I quickly looked at the service manual.

Then I probed the data line going out from the front panel TO the main board, and yeah, I now do get activity there, no worries ! It sends stuff whenever I press any of the push buttons !  :) 

You can't imagine the joy I felt when it came to life the first time, like : "WHAT, I really managed to fix this thing ?! Without any schematics at that ?! Maybe I am not so worthless after all ! LOL ".

Oh boy that was fun and gratifying working on this thing, one of my most enjoyable repairs for sure  :)

Now I guess I ought to buy new diodes and caps (the electrolytic caps on the power rails as well while I am it ), and that will be the end of it  :)

Well, almost : now we must fit these boards back into the dish-washer and see if it works ! It should... but there is still some uncertainty due to the fact that they gave the main board to some "expert" who replaced a couple relays... hoping he didn't goof that up, and didn't do more damage to the board that I am not aware of...   :(  So still a bit anxious as to whether the machine will work or not !  ???

So, glad the Hobart Service Tech did not charge them 1,500 euros to replace the main board... which was NOT faulty !


« Last Edit: September 22, 2018, 02:31:51 am by Vince »
 

Offline Vince

  • Frequent Contributor
  • **
  • Posts: 432
  • Country: fr
Re: Board repair : HOBART " hood " type dish washer
« Reply #23 on: September 22, 2018, 01:58:41 am »
For better magnification a cheap jewelers headset is what you need and 2-5x is plenty enough for all but the tiniest SMD work.

Yeah I need this dearly indeed !

Quote
Sounds like you'll crack it soon, keep up the great work.

Indeed I cracked it !  My previous post collided yours, have a look, it's all there, success !  :D
 

Online xrunner

  • Super Contributor
  • ***
  • Posts: 4626
  • Country: us
  • hp>Agilent>Keysight>?
Re: Board repair : HOBART " hood " type dish washer
« Reply #24 on: September 22, 2018, 02:03:58 am »
Congratulations you are now the official dishwasher repair expert for the EEVBlog forum.  :-/O
I am a Test Equipment Addict (TEA) - by virtue of this forum signature, I have now faced my addiction
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf