Undead Programmable DC Supply - any ideas?

[EHT]

Pin 10 is connected to the Reset output on the Watchdog MAX1232, so this pin does line up with the standard arrangement. Does that help identify the variant?

I have connected up the +/-15V supplies as well as +5V. Now the display stays on 8888 8888, so "Err 15U" does indicate loss of the 15 supply. Shows the MCU is doing something, but its getting stuck somewhere after this basic check.

Since I now get the same results from powering up just these internal lines, it does let me experiment with the high voltage sections off.

I was using the Scopemeter before, but now I've tried with a 'real' scope, I can see the digital signals better and found some of those I said were static do carry signals, for instance the GPIO serial data

[EHT]

OK, MCU is alive!  After probing more with the scope I found there were pulses on GPIO SDA but just a bit of noise on SCL. Turned out this was a bad pin (27, A11) on the MCU socket. I bent and cleaned the pin and now it boots and there is proper communication with the front panel. First I saw "AC Fail" when I had powered up the internal rails from the bench PSU.

I plugged the AC back in and it now almost works. It shows "Fan Pr0" and indeed one of the fans isn't running. Not too surprising as the fan had a lot of dirt and corrosion. When i rotate the control knob it will show settings and I see it is indeed set to 14.00V so that fully explains why the output was 14v!

Despite it seeming to shutoff the main output now, the crowbar's heatsink is still getting very warm, so that is my remaining concern. I'll try to fix the fan and come back to that.

List of faults so far:

• TOP224YN switching regulator dead - loss of internal power
• SMD cap in PFC circuit corroded / dry - PFC not running
• SMD resistor in display corroded - no display
• PLCC socket damaged / dirty - MCU won't boot
• Fan not working - PSU in fail mode
• One 7 segment LED faulty

Ironically the mfr promotes this range as highly reliable, and the tamper sticker seemed to be intact when I got it...

[m k]

Very good.

The thing has been in a hostile environment, at least sort of.
I wouldn't be surprised if that missing segment was also a bad contact.

It seems that you should change to finding disconnected parts instead of faulty ones.

[EHT]

Yes, this is the repair that keeps giving!

There is another problem... in the manual it says Fan Fail should not switch the output off, but it has. When the MCU wasn't running properly I was seeing +14.00V on the output. Now I'm seeing 14.00v in the display and 0v at the output! The PFC is no longer running properly... its sending bursts of pulses resulting in only 280v output. Still, at least I know the regulator IC and can have a stab at finding this (last ) fault.

 

[m k]

Maybe working I2C just put startup so much further that it showed the message and saw the crowbar.

[Swake]

Excellent  progress :-)  Don't give up, you're almost there 

Have you tried USB/serial/ethernet again now you know the MCU is alive again? Maybe there is some info there. You also might try that factory reset (not a full reset that would erase calibration data too if I recall well). As there are so many settings on that thing it might be in a somewhat weird configuration.
Replacing the fan, even with a temporary unit that does not fit would at least clear the error and make sure this is not the cause. Do you recall it was working when you had the 14V output without display?

As there was 14V output before and not anymore it sounds like something 'new' has been broken or is simply miss-installed. Connector loose, bolt missing.

[Swake]

If that crowbar thing is effectively a crowbar, under what conditions would it 'short'?
- On my HP PS it kicks in when the output is turned off so it pulls to 0V as quickly as possible. This does not match the 14V on the output you had previously, but it does match the 0V there is now.

What happens if you configure the output voltage to something very small/high.

[EHT]

Excellent  progress :-)  Don't give up, you're almost there 

Thanks! Yes, I'm pleased the MCU has woken up!

If that crowbar thing is effectively a crowbar, under what conditions would it 'short'?
- On my HP PS it kicks in when the output is turned off so it pulls to 0V as quickly as possible. This does not match the 14V on the output you had previously, but it does match the 0V there is now.

So it has two modes of operation. There are two stages: startup/shutdown and normal run. With reference to the diagram I posted, the comparator (a) outputs a very high level when the output is off at startup/shutdown since the + input is fed from a voltage divider from +15v and the - input from the PSU output. So, crowbar is full-on at startup/shutdown like the HP.
What is unclear is the purpose of the negative feedback in (a) which keeps Q25 in semi-conduction if the input to (b) - is 0v. Strangely I found this same control line alters the fan speed.
As you suggested I'm just going to hack it so that the fan error clears - it has a shunt on each fan.

Replacing the fan, even with a temporary unit that does not fit would at least clear the error and make sure this is not the cause. Do you recall it was working when you had the 14V output without display?

No, this fan had bad internal corrosion and never worked. I found it was partially shorted. I think it might occasionally go open as well, so maybe the fan error may not have been present initially. This fan being shorted might explain the original fault of the switching reg failed (it feeds the unregulated voltage to the fans).

As there was 14V output before and not anymore it sounds like something 'new' has been broken or is simply miss-installed. Connector loose, bolt missing.

Not as simple as a missing connector, there are very few - just one large PCB and a front panel. I think it could be the MCU putting it into shutdown mode though. I'll clear the fan error then experiment with the controls and see if the o/p will come back on.

[EHT]

Close, but not there yet...

I hacked the fan protection. It uses a current shunt for each fan and opamp comparator. I pulled the input of the opamp such that the failed fan appears OK, and unplugged the dead fan.

* Fixed the 7-segment display by cleaning as it was socketed. Output enable button also didn't work at all, after spraying with IPA and agitating seems to work.
* I can navigate through the menus and program values
* I did a soft reset which clears the set values back to default
* Voltage and current program values can be set, as can all the other settings.
* Output can be enabled on the front panel, output LED and CV LED are on BUT the output is still 0.0V 
* PSU voltage readback on the display is also 0.0V

PFC is now running fine: 370V DC output. I think it was put into shutdown before due to control from the MCU. Now I have to look into the control to the M51995AFP switching IC I suppose. Presumably the switcher is OK as it has worked before, but who knows!

This time, the crowbar is not the problem - there's no output across the main rails at all

[m k]

Is the crowbar FET still on, should it be?

The device has so many optos that you can expect feedbacks from practically everywhere.
Means that what ever you do can also affect somewhere else.

Maybe you should once again measure voltages between MCU socket pins and chip pins.
For zero levels use AC area if you can't trust resistances.

Can you go back to that output voltage on situation?
It's probably too far away but then you could check what MCU pins are different.

[EHT]

Is the crowbar FET still on, should it be?

Yes, this has not changed behaviour. I checked that the control input to this circuit (opamp (b) - ) is still being held at 0V all the time which means the gate is driven by opamp (a) comparator.  Q25 MOSFET will be in full conduction (gate ~ 13v) but D-S voltage is 0.0V, so no effect. Judging from the previous behaviour this should sink a bit of current but not enough to trip OCP etc. I've disabled OVP, OCP on the front panel. So, I think the crowbar circuit is not instrumental in the current behaviour.

The device has so many optos that you can expect feedbacks from practically everywhere.
Means that what ever you do can also affect somewhere else.

Yeah indeed! I guess the general setup is there is a control line from MCU and feedback return to it. When the MCU was crashed it wouldn't respond to the feedback, but now it is. I'm sure there are a number of things that could result in it shutting off the output but it is odd that it doesn't show an error condition. Previous behaviours were error conditions (No +15V, No AC, no GPIO).

Maybe you should once again measure voltages between MCU socket pins and chip pins.
For zero levels use AC area if you can't trust resistances.

Can you go back to that output voltage on situation?
It's probably too far away but then you could check what MCU pins are different.

Both great ideas, thanks. However, I never was able to measure resistance pin-socket because I can't reach a probe through the very tiny gap in PLCC socket. I've been probing the socket (i.e. external circuit). I did remove the MCU and check resistance of its pins to its ground pin, and same for the circuit.

To go back i'd have to try to put kaptan tape over the MCU SCL pin. That is a really good idea for sure. However, the overall behaviour is quite different and I only traced some of the MCU pins.

For now I'm going to try to trace a bit of the M51995 circuit (switching controller). It is hooked up to 3x 2SK2611 MOSFETs. As expected, they have no signal on the gate - should be PWM. If i can find the control lines to this, then I'd know where to poke around for differences in behaviour.

[m k]

My guess is that everything is fine with the switcher.
Earlier it was putting 14V out as was requested.
Current limit was also so that crowbar didn't matter.
Now situation is different and it is ordered off.
So it's a result.

Same with the crowbar.
It's operation has not changed.
But it's also a result.

Now when you can communicate with the MCU through the front panel you know that both are generally fine.
But you are communicating with the internals of the MCU.
All those I/O pins are still generally unknown.
You need to get to the situation where you can be sure that all MCU pins are really connected and operational.

PLCC socket has a one sided spring effect towards the chip pin.
For larger chips this socket type is not very common for a reason, there were also some generally bad quality issues.
If you can put cleaner drops on the connection point you may have some visual effects how the connection actually is.
You can also do some visual estimations by eyeballing the empty socket.
Everything should obviously be level but they are not, but don't over estimate either.
If you need to bend a pin, too little and it wont last, too much and it will bend under when you put the chip in.

[EHT]

My guess is that everything is fine with the switcher.
Earlier it was putting 14V out as was requested.
Current limit was also so that crowbar didn't matter.
Now situation is different and it is ordered off.
So it's a result.

Thanks, yes, after I traced out all of the crowbar I can tell the problem is not that area but the control line going in to it. I was just going to do enough tracing on the switcher to understand what the incoming and feedback signals to the MCU are. So far I can see it has 2 standard optos going in.

There will be plenty of measurement of the output voltage and quite complex remaining circuit with 8-input ADC and at least 2 more quad opamps. I guess the ADC reading is fine, because (a) the analog readback signal was inline with the old 14v output, (b) the voltage readback of 0.0v now is correct.
Not only are there 3 DACs, 8 input ADC but also 2 quad 'digital potentiometers' and some digital->analog switches. Its hard to see how they can have made a single output PSU so complex!!

All those I/O pins are still generally unknown.
You need to get to the situation where you can be sure that all MCU pins are really connected and operational.

Right

PLCC socket has a one sided spring effect towards the chip pin.
For larger chips this socket type is not very common for a reason, there were also some generally bad quality issues.
If you can put cleaner drops on the connection point you may have some visual effects how the connection actually is.
You can also do some visual estimations by eyeballing the empty socket.
Everything should obviously be level but they are not, but don't over estimate either.
If you need to bend a pin, too little and it wont last, too much and it will bend under when you put the chip in.

Yes, so I could tell that the pin feeding SCL was slightly different in terms of its position to the others. When I fixed this I also inspected the others and tried to clean them all up with IPA. Of course, doesnt mean that there is not another bad one. There are also some PCB layer through-holes that have some corrosion but I haven't yet found any open circuit.

I like your idea of going back to the situation where the output was enabled by re-blocking the GPIO SCL. I'm intending to trace the control of the switcher to see where to look, then break this again if that doesn't directly unearth the problem.

Thanks again for all your suggestions in this ongoing saga!

[m k]

My part is easy, and not very energy intensive, but middle layer corrosion is a bad rap, luckily all power holes are usually full.

Demo circuit for M51995 had 3 optos, on/off was one, current limit was trafo but it can also be an opto.

What AD/DA functions there could be.
Voltage, current and temp are obvious A/D and all kinds of internal voltage adjustments are clear for D/A but it's still pretty much for a single output.

Can you see any connector variations or so around the PCB?
Maybe it's a general module for varieties of models.

Are there any pin headers or other jumper like things?
Maybe a face plate and some jumpers can change the model.

[EHT]

My part is easy, and not very energy intensive, but middle layer corrosion is a bad rap, luckily all power holes are usually full.

Thanks!

What AD/DA functions there could be.
Voltage, current and temp are obvious A/D and all kinds of internal voltage adjustments are clear for D/A but it's still pretty much for a single output.

There is also analog programming and readback so that's part of it. There is overtemp cutout which can be enabled/disabled via the MCU. There are 105C cutout switches on the two main heatsinks, so I think it is just on/off. It means the behaviour now is another "unexpected" fault since it doesn't register error with MCU

Can you see any connector variations or so around the PCB?
Maybe it's a general module for varieties of models.

Are there any pin headers or other jumper like things?
Maybe a face plate and some jumpers can change the model.

I dont think so. See pic. There is this one huge PCB plus small front panel which is just the display and controls. There are little or no "optional parts". There are just two unused connectors. One a 10 pin low voltage one CN10 near the RS485 section. There is optional LAN & GPIB which this unit doesn't have, so pretty sure the connector is for that.
Aside from that a 2-pin connector right by the optos on main switcher - J3. I guess its a test point or shutdown input for testing. Still haven't explored this bit yet. I wont have time on this over the weekend , so the saga continues!

[EHT]

  Result! I cleaned and bent the MCU socket pins again and now there is correct output! Very accurately regulated too -  measures within a few mV

The crowbar is still partially active though, so that is the remaining mystery. I'll post more details of the circuit. It is clearly designed to have this mode where it sinks some current but is not full-on; I don't understand why. It may even be working as intended now. For instance the current sunk into it doesn't register on the current measured. Perhaps it is normal for zero load even, but the heatsink is small and gets pretty hot. Doesn't feel right. Anyway.. progress

[m k]

Dam those sockets!
And it's not even wide.
Back in the day there where contracted cushion shaped aluminum supports to counter those bulked cushion shaped sockets.
And that was for computer motherboards.

Initial load is usually static, can be different here since variations are so wide.

Have you tried communicating with it remotely?
RS-232 is clearly easiest.

Manual says that in case of flash checksum failure an error is pushed to SCPI error queue.
It is also happening during boot up, can be cleared by soft reset and is last in the error priority list.
So being able to do a soft reset all higher priority errors must be off.

From the picture:

So all sections are optically isolated, that explains it.
There seems to be also few pico fuses.

Below left bigger trafo is a connector, is it CN33, are there 32 other connectors.

Is there a USB controller between RS-section and Isolated I/O, below xtal.
Chip type can possibly help with computer connection.

A12 in the middle of RS-section.
Doesn't look like 10baseT magnetics and if so shouldn't have equal letter with all other chips.
My guess is that it's a floating power for external communications, I counted 22 optical connections around.

If Ethernet and GPIB locations are combined can it mean that ENET is actually AUI-connector.
That would be pretty old school.

If GPIB is connected through the red CN10 there must be 8 bidirectional optos somewhere, and some controls.
Some are promising but they are under staffed.

[EHT]

Dam those sockets!
And it's not even wide.

Yeah! Maybe there are still pins that are not connecting properly (crowbar issue). I tried IPA, maybe I should try contact cleaner or find some useful abrasive

Have you tried communicating with it remotely?
RS-232 is clearly easiest.

No, I'll do that next week. It does have USB - that is an FTDI chip.

Manual says that in case of flash checksum failure an error is pushed to SCPI error queue.
It is also happening during boot up, can be cleared by soft reset and is last in the error priority list.
So being able to do a soft reset all higher priority errors must be off.

I think I did this successfully (display said SrS) but I'll do it again.

There seems to be also few pico fuses.
...
Below left bigger trafo is a connector, is it CN33, are there 32 other connectors.

Yes, I've checked these - one tiny fuse, what appear to be a couple of fusible resistors and two black plastic circular fuses (see just above area labelled crowbar). There are a lot of connectors, since they've labelled incoming mains, switches, rear panel etc. There are only 2 unused ones - J3 and CN10. I presume that if GBIP/LAN module is fitted it will have a PCB perhaps with more optos on it. This unit has RS232,485, USB and Analog voltage control.

So far the problems have all been due to the corrosion apart from the TOP224 switching reg, which I expect overheated from the fan being shorted. I may recheck for the PCB through-holes showing a bit of corrosion, clean up the MCU socket again and look for dry SMD joints underneath (many components mounted on underside).

Thanks again!

[m k]

Soft Reset from the manual,
PSU S rS is displayed for starters and
PSU CLr when done but for 1 second only.

I think you can use decent contact cleaner freely, it's just a regular PCB, so no need to be specially cautious.
But you don't need to flood it, just a wet cotton swab will do.
Back in the day we used all means necessary, but that was usually just for the time before the next machine arrives.

PLCC type of contact is self cleaning, by abrasion when inserting.
So you can do that, but there's an intuitive catch, it only matters there where the contact will be.
Now when you have reshaped those socket pins the contact position of a chip pin is also different, maybe a bit higher.
Irritable things are these contact issues, at least you know where to look next time.

[EHT]

Soft reset done; it showed PSU Clr. I think it may be operating properly now... The remaining 'problem' was the 'crowbar' circuit. I wonder this thing is actually operating as intended. It is rather hard to tell. I thought it was not working correctly simply because it was sinking some current and heating up under normal operation, but with no load. All external signs are fine. The voltage is being very accurately regulated and the voltage and current readback is accurate.

After tracing the circuit out, I found it is controlled by a line I labelled 'I Share' on the attached schematic (this is connected to the 'current share output' on the analog I/O). When there is no load, this is at 0v, resulting in no conduction through D61, leaving the crowbar in its 'floating' mode. I found this line is also used to control the voltage going to the fans through a rather crude mechanism of a 5.1v zener (ZD14) going from the reference connection of a 7805 regulator to this line. Further, I found that when there is load on the main PSU output, this line moves up from 0V, perhaps in proportion to the current. This results in the fans going faster and the crowbar easing off. However, I don't have a decent load and the full rated load is 25A so I haven't managed to budge it much, but it seems this could be operating as intended. Note, it can't be tested by shorting the output, because when the output voltage is zero that turns the crowbar on (voltage divider R270, R271, R272).

The crowbar floats given the negative feedback over the two 2.4 Ohm resistors such that it doesn't consume too much power. I wonder it helps regulate the output accurately when there is low load by damping it. I also notice that on the top panel there are more pronounced markings (after cleaning) showing this heatsink has been hotter than the others - see pic. This would certainly happen if the PSU was powered on with no load.

I am intending to replace the broken fan and the working one with quieter and lower power models. This is a 1U unit which has 2 x 42dbA 40mm fans - quite annoying. These are 14CFM. I found 8.9CFM 25.5dbA or 7.7CFM 21dbA

It is more use to me as an accurate digital power supply rather than a very high power one, but I'll have to check this crowbar heatsink doesn't get too hot. Unlike the 2 MOSFET switcher heatsinks, there is no thermal cutout...

[m k]

Yes, classic crowbar it is not and controlled damper it seems to be.

I'd say that congratulation, you have a sophisticated, almost new and working power supply.

Maybe you can setup your new silent fans to speed up and let the crowbar rest.